Does Beyla 2.0 support sending traces directly to Tempo on port 4318?

[trungtien]

Hi Team,

I deployed the Beyla 2.0 DaemonSet using a Helm chart in Kubernetes, and then sent traces to my local Tempo. I checked my Tempo, and it looks good, but Beyla is not sending anything to Tempo

k get all -n beyla
NAME READY STATUS RESTARTS AGE
pod/beyla-25292 1/1 Running 0 137m
pod/beyla-52csf 1/1 Running 0 137m
pod/beyla-82fls 1/1 Running 0 137m
pod/beyla-8hh6v 1/1 Running 0 138m
pod/beyla-g254s 1/1 Running 0 138m
pod/beyla-hpk6h 1/1 Running 0 138m
pod/beyla-j74gv 1/1 Running 0 137m

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/beyla 7 7 7 7 7 5d21h

k get configmap beyla -n beyla -o yaml
apiVersion: v1
data:
beyla-config.yml: |-
discovery:
services:
- k8s_namespace: bank-of-athnos
exclude_services:
- exe_path: ".alloy.|.otelcol.|.beyla."
attributes:
kubernetes:
enable: true
select:
beyla_network_flow_bytes:
include:
- k8s.src.owner.type
- k8s.dst.owner.type
- direction

filter:
  network:
    k8s_dst_owner_name:
      not_match: '{kube*,*jaeger-agent*,*prometheus*,*promtail*,*grafana-agent*}'
    k8s_src_owner_name:
      not_match: '{kube*,*jaeger-agent*,*prometheus*,*promtail*,*grafana-agent*}'
ebpf:
  wakeup_len: 100
log_level: DEBUG
prometheus_export:
  path: /metrics
  port: 9090
otel_traces_export:
  endpoint: http://tempo.tempo.svc:4318/v1/traces
  insecure_skip_verify: true

kind: ConfigMap
metadata:
annotations:
meta.helm.sh/release-name: beyla
meta.helm.sh/release-namespace: beyla
creationTimestamp: "2025-02-12T09:20:43Z"
labels:
app.kubernetes.io/component: config
app.kubernetes.io/instance: beyla
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: beyla
app.kubernetes.io/part-of: beyla
app.kubernetes.io/version: 2.0.2
helm.sh/chart: beyla-1.7.2
k8slens-edit-resource-version: v1
name: beyla
namespace: beyla
resourceVersion: "217335281"
uid: c94b4c70-808c-4904-b374-98e7ad2c373f

[marctc]

Hey @trungtien, thanks for reporting. Would you mind sharing some logs? Can you see traces if you set the env var BEYLA_TRACE_PRINTER=text?

[trungtien]

hi @marctc
I have added the environment variable BEYLA_TRACE_PRINTER=text to this DaemonSet configuration. I then restarted Beyla after making this change. The log of the Beyla service is provided below.

beyla-zpfxx.log

[marctc]

It looks that there's a bug in the eBPF side. Could you try first to run with image: grafana/beyla:2.0.4?

[grcevski]

Thanks for sending a log, I think we have a bug with kernel 5.10. Can you please share your exact Linux version, so that I can try to replicate with a VM?

[grcevski]

Hi @trungtien, I tried on the following Linux kernel Linux debian11 5.10.0-26-amd64 and I don't see an issue. If you can share your exact kernel and platform it would be very much appreciated.

One other thing to try is to use Beyla 1.9.x which doesn't have the probe we fail on with the verifier error.

[trungtien]

Hi @grcevski and @marctc , can I send tracing data directly from Beyla to Tempo? (No need for OpenTelemetry or Alloy.)?
@marctc image: grafana/beyla:2.0.4 -> yes i will try.
@grcevski Linux Hostname-645-53 5.10.10-051010-generic #202101231639 SMP Sat Jan 23 17:16:37 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

[marctc]

Hi @grcevski and @marctc , can I send tracing data directly from Beyla to Tempo?

Yes, your config is right in this case

[grcevski]

Hi @trungtien, I can't seem to find a distribution with your exact kernel version, did you build your kernel yourself or is this an official Linux distribution of some kind?

[trungtien]

hi @marctc grafana/beyla:2.0.4 is not working -> tempo not show any services.

hi @grcevski
this is information what you need:
root@node-hostname:# dpkg -l | grep linux-image
ii linux-image-5.4.0-126-generic 5.4.0-126.142 amd64 Signed kernel image generic
ii linux-image-generic 5.4.0.126.127 amd64 Generic Linux kernel image
ii linux-image-unsigned-5.10.10-051010-generic 5.10.10-051010.202101231639 amd64 Linux kernel image for version 5.10.10 on 64 bit x86 SMP
ii linux-image-unsigned-5.19.17-051917-generic 5.19.17-051917.202210240939 amd64 Linux kernel image for version 5.19.17 on 64 bit x86 SMP
root@node-hostname:# cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
root@node-hostname:~# uname -a
Linux node-hostname 5.10.10-051010-generic #202101231639 SMP Sat Jan 23 17:16:37 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

[grcevski]

Hi @trungtien,

I dug deeper into this and it appears to be a problem with corrupted BTF symbols, which Beyla uses to find offsets in the kernel. Someone reported it when moving from kernel 5.19 to 6.1 and it was fixed there: https://lore.kernel.org/bpf/Y%[email protected]/T/.

Does the same problem occur if you boot your 5.19 kernel image instead of the 5.10?

[trungtien]

hi @grcevski
I downgraded the Helm version of Beyla from 1.7.2 to 1.6.3. It works; I can see traces on Tempo Distributed :)