Headers not set using environment variable when using gRPC

[Andrew-Prendergast]

Hi,

When using gRPC trace exporting it seems like the headers set using the OTEL_EXPORTER_OTLP_TRACES_HEADER environment variable are not being set. Looking in the code, it seems to be set for HTTP (which I can confirm works fine), but not gRPC: https://github.com/grafana/beyla/blob/main/pkg/export/otel/traces.go#L793 its being set in the getHTTPTracesEndpointOptions function but not getGRPCTracesEndpointOptions function. We run Grafana Tempo as multi tenancy and are not able to send traces using our gRPC endpoint.

Env variable config we are setting:

Environment=OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://<tempo_distributor_host>:4317"
Environment=OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="grpc"
Environment=OTEL_EXPORTER_OTLP_TRACES_HEADERS="X-Scope-OrgID=default"

Error message on Tempo Distributor:

Feb 25 15:51:45 tempo-distributor-0 tempo[963892]: ts=2025-02-25T15:51:45.517099262Z caller=middleware.go:68 msg="failed to extract org id from both grpc and HTTP" err="no org id" client=10.106.4.126:59582

[grcevski]

Thanks for reporting this @Andrew-Prendergast !