Inconsistent Kafka behaviour

[3timeslazy]

Hi Beyla Team,

First of all, thank you so much for your work! Beyla is an amazing tool.

I've been playing with Beyla lately and noticed some bugs when working with Kafka.

To put it simply, I have a simple microservice that accepts a key-value pair, stores it in redis, and published it to Kafka. And while redis works fine in all cases, Kafka traces are different depending on the client library.

I have 3 implementations:

• github.com/IBM/sarama
• github.com/segmentio/kafka-go
• github.com/confluentinc/confluent-kafka-go/kafka

All three have differences in the generated traces. More details here: https://github.com/3timeslazy/ebpf-beyla-bug-report/tree/main

tldr;

github.com/IBM/sarama

traceID, spanID and parentSpanID are empty values.

I'm not an expert in Beyla's code, but out of curiosity I''ve checked it and my guess is that client_trace_parent might be missing in go_sarama.h

github.com/segmentio/kafka-go

works fine in general, but sometimes parent span id is missing

Also, I think span kind needs to be producer, not consumer.

github.com/confluentinc/confluent-kafka-go/kafka

Since this is a Cgo wrapper around rdkafka, the client traces are not sent at all. However, there are traces on the Kafka service side with messaging.client_id = rdkafka.

unrelated comment

Beyla's documentation does not say anything (or have I missed it?) about client libraries that the Go code needs to use. However, from the debug logs, I figured out that it actually assumes that the code uses one of the following symbols:

net/http.(*conn).readRequest
    golang.org/x/net/http2.(*responseWriterState).writeHeader
    net/http.(*persistConn).roundTrip
    google.golang.org/grpc.(*Server).handleStream
    github.com/redis/go-redis/v9/internal/pool.(*Conn).WithWriter
    github.com/segmentio/kafka-go.(*Client).roundTrip
    github.com/IBM/sarama.(*Broker).write
    net/http.serverHandler.ServeHTTP
    github.com/Shopify/sarama.(*responsePromise).handle
    golang.org/x/net/http2.(*serverConn).runHandler
    database/sql.(*DB).execDC
    google.golang.org/grpc/internal/transport.(*http2Server).WriteStatus
    google.golang.org/grpc.(*ClientConn).NewStream
    github.com/redis/go-redis/v9.(*baseClient)._process
    github.com/segmentio/kafka-go.(*Writer).WriteMessages
    github.com/segmentio/kafka-go.(*Writer).produce
    net/http.(*http2ClientConn).roundTrip
    github.com/segmentio/kafka-go/protocol.RoundTrip
    net/http.(*http2serverConn).runHandler
    github.com/redis/go-redis/v9.(*baseClient).pipelineProcessCmds
    github.com/segmentio/kafka-go.(*reader).read
    github.com/Shopify/sarama.(*Broker).write
    golang.org/x/net/http2.(*Framer).WriteHeaders
    net/http.(*http2Framer).WriteHeaders
    net/http.(*http2responseWriterState).writeHeader
    database/sql.(*DB).queryDC
    github.com/IBM/sarama.(*Broker).sendInternal
    net/http.(*http2ClientConn).RoundTrip
    google.golang.org/grpc.(*clientStream).RecvMsg
    google.golang.org/grpc/internal/transport.(*http2Client).NewStream
    google.golang.org/grpc.(*ClientConn).Invoke
    runtime.goexit1 net/http.(*Transport).roundTrip
    net/http.(*conn).serve
    google.golang.org/grpc.(*ClientConn).Close
    github.com/segmentio/kafka-go.(*reader).sendMessage
    runtime.newproc1 golang.org/x/net/http2.(*ClientConn).RoundTrip
    net/http.(*response).WriteHeader
    golang.org/x/net/http2.(*serverConn).processHeaders
    net.(*netFD).Read
    google.golang.org/grpc/internal/transport.(*serverHandlerTransport).HandleStreams
    golang.org/x/net/http2.(*ClientConn).roundTrip
    net/http.(*http2serverConn).processHeaders
    google.golang.org/grpc.(*clientStream).CloseSend
    google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders
    github.com/redis/go-redis/v9.(*baseClient).txPipelineProcessCmds
    github.com/IBM/sarama.(*responsePromise).handle
    github.com/Shopify/sarama.(*Broker).sendInternal
    net/http.Header.writeSubset
    net/textproto.(*Reader).readContinuedLineSlice

And I'm wondering if it's worth including this list in the documentation, or at least mentioning that the instrumented code needs to use certain libraries.

P.S.

If you need any help or more info from me, please let me know

P.S.S.

Sorry if the code/compose is a mess