Beyla with OTEL instrumented applications.

[cxk314]

We have OTEL auto-instrumented .NET applications (with OTEL operator) running on Kubernetes and sending telemetry to Alloy receiver. We are also looking at Beyla, but we are struggling to see the difference between OTEL and Beyla. Can someone provide more insight into following:

1. What telemetry does Beyla provide that OTEL instrumentation doesn't? With OTEL we get metrics, logs and traces.
2. Is Beyla telemetry complementary to what OTEL instrumentation provides or they are mutually exclusive?
3. Can Beyla run together with OTEL instrumented applications? Are we going to get duplicated telemetry for the same applications?
4. What is the recommended approach with Beyla? When to have Beyla and when to do OTEL auto-instrumented applications and when to use both?

I would really appreciate some insights on the above.

[marctc]

Hey @cxk314, thanks for your questions. I'm curious to hear if you couldn't answer these questions yourself looking
at the docs. Asking because maybe we have some gaps.

Beyla was initially born to auto instrument Go applications. With time we evolved to support any programming language.
Beyla wasn't created to replaced native OTEL auto instrumentations (like .NET) and honestly, it's hard to compete with native
SDK. However, it's a good choice when for example you don't wanna mess up with those SDKs (also don't know how hard is work with them) or you have a multiple services running in different programming languages. Also, in some cases using Beyla would be more performant than adding auto SDK to many running services.

1. What telemetry does Beyla provide that OTEL instrumentation doesn't? With OTEL we get metrics, logs and traces.

metrics and traces

2. Is Beyla telemetry complementary to what OTEL instrumentation provides or they are mutually exclusive?

Complementary. You can instrument your .NET apps with the SDK and use Beyla for Go or PHP apps (to name some).

3. Can Beyla run together with OTEL instrumented applications? Are we going to get duplicated telemetry for the same applications?

Beyla has a flag (enabled by default) that detects that an app is already instrumented with OpenTelemetry. If that's the case, it won't instrument it.

4. What is the recommended approach with Beyla? When to have Beyla and when to do OTEL auto-instrumented applications and when to use both?

I think already answered before

[cxk314]

Thank you @marctc. That is really helpful. Beyla documentation is good it was just not clear how different Beyla is compared to OTEL auto-instrumentation and when should one be used vs the other. The documentation states that Beyla provides OTEL telemetry and can auto-instrument .Net and other applications, but it wasn't clear how this will work with already instrumented applications and what we will get that OTEL doesn't give us. I was also trying to see what set of metrics Beyla can provide and whether they are different (we get more metrics or less) to what OTEL provides. It would be specifically helpful for teams who are looking at auto-instrumentation options and trying to understand which approach to use.

Also, when reading it again, it is there that Beyla supports metrics and traces, but it could be more explicit that logs are not supported, so if we want trace to log correlation then we will not get it. Since it says Beyla provides OTEL telemetry the impression was it will give full set of OTEL (metrics, logs and traces). But it is much clearer now. Thank you.

[marctc]

Also, when reading it again, it is there that Beyla supports metrics and traces, but it could be more explicit that logs are not supported, so if we want trace to log correlation then we will not get it. Since it says Beyla provides OTEL telemetry the impression was it will give full set of OTEL (metrics, logs and traces). But it is much clearer now. Thank you.

Beyla provides a way to generate logs from the captured traces using BEYLA_TRACE_PRINTER. This is not a structured OTEL log but it's essentially a request log. I'm curious to hear which logs would you expect Beyla to generate (I'm not familiar with .NET instrumnetation).

[cxk314]

We are interested in consistent OTEL telemetry so that correlation between traces, logs and metrics can be easily visualized for troubleshooting. We are using self-hosted Grafana stack (Grafana, Loki, Mimir, Tempo) on AKS Kubernetes. This requirement is not specific to .Net. We are just following OTEL standard. .Net implements OTEL standard. We used OpenTelemetry Operator to inject SDKs and auto-instrument our applications. It works fine but it requires certain changes to the bootstrap code and the Nuget packages we include during the build that will make sure it does not conflict with the OpenTelemetry Operator injected SDKs. It also requires putting annotation in deployment.yaml. We were just looking if Beyla can do what OTEL does but without requiring us to keep our builds aware of the injected SDKs and without us making sure our Nuget configuration will not conflict with injected SDKs during runtime while it is fine during build time.

[grcevski]

Hi @cxk314,

The main reason we are building Beyla is to ease these kinds of problems, and we hope this project will soon become mainstream OpenTelemetry project because of the same reasons, it's much easier to get started and it doesn't matter if you are already instrumenting/using other agents etc.

We currently don't generate OTel logs, but it's not out of the realm of the possibility, we just haven't done it yet. Is it possible to have your logs converted to OTel logs in the collector? Or is it the lack of TraceID that's a blocker here if you are shipping the regular logs to Loki?