otelcol.exporter.otlp.tracesservice allow config for write_buffer_size

[radupopa369]

getting this error

ts=2024-09-05T13:36:46.474811899Z level=error msg="Exporting failed. Dropping data." componentpath=/ componentid=otelcol.exporter.otlp.tracesservice error="not retryable error: Permanent error: rpc error: code = ResourceExhausted desc = grpc: received message after decompression larger than max (7050547 vs. 4194304)" droppeditems=6466

Looked through the chart and there is no way I can configure the generated config.alloy

The template file for this section https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/templates/alloy_config/_traces_service.alloy.txt
needs a way to configure write_buffer_size . Refference in the docs here https://grafana.com/docs/alloy/latest/reference/components/otelcol/otelcol.exporter.otlp/

[Raboo]

I ran into the same problem.
It can't be the write_buffer_size as it defaults to 512KiB and the limit that is hit is 4MiB.

I tried increasing grpc_server_max_recv_msg_size and grpc_server_max_send_msg_size in Tempo to 8MiB, and yes setting limits in the client so it doesn't send to big payloads is probably better than just increasing the limits server side. But I don't know which client side/alloy setting it would be.

Now I seem to get less errors, but still getting errors.

However I do feel that the correct setting in the server side to adjust would be https://grafana.com/docs/tempo/latest/troubleshooting/response-too-large/#ingestion.

distributor:
  receivers:
    otlp:
      grpc:
        max_recv_msg_size_mib: <size>

However this setting can't easily be changed using the tempo-distributed helm chart afaik.

[petewall]

Making a PR for this: #742