[BUG]: 429s in nginx output

[ganey]

Module Version(s)

1.4.0

Bug Report

Getting a lot of 429's in nginx output with a sample rate of 0.01 running to apm via a socket.

Not sure where the 429 is from, the agent socket? or upstream at datadog? How can i figure out where?

Reproduction Code

The config is a bit complex as it's a pod on kubernetes, we have around 30 pods running, traffic can be 2,000 requests a second when these logs appear.

nginx config has:

datadog_sample_rate 0.01;
datadog_agent_url unix:///var/run/datadog/apm.socket;

Error Logs

nginx output log:

2025/01/30 14:59:33 [error] 170#176: nginx-datadog: Unexpected telemetry response status 429 with body (if any, starts on next line):
{}
2025/01/30 14:59:33 [error] 175#178: nginx-datadog: Unexpected telemetry response status 429 with body (if any, starts on next line):
{}
2025/01/30 14:59:33 [error] 160#165: nginx-datadog: Unexpected telemetry response status 429 with body (if any, starts on next line):
{}
2025/01/30 14:59:33 [error] 107#111: nginx-datadog: Unexpected telemetry response status 429 with body (if any, starts on next line):
{}
2025/01/30 14:59:33 [error] 152#157: nginx-datadog: Unexpected telemetry response status 429 with body (if any, starts on next line):
{}
2025/01/30 14:59:33 [error] 131#136: nginx-datadog: Unexpected telemetry response status 429 with body (if any, starts on next line):
{}
2025/01/30 14:59:33 [error] 164#172: nginx-datadog: Unexpected telemetry response status 429 with body (if any, starts on next line):
{}

Operating System

5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 GNU/Linux

[dmehala]

Hi @ganey
This issue went under my radar, my bad.

The output log indicate that NGINX is sending too many telemetry metrics to the agent. This usually happens when too many NGINX child process send telemetry at the same time. We do collect anonymous telemetry on usage of our library to assess the reliability and usage of our product. You can opt-out by setting the environment variable to DD_INSTRUMENTATION_TELEMETRY_ENABLED=false.

[ganey]

@dmehala will disabling telemetry disable the apm request/hit counts in datadog?

edit looks fine thanks!

[ganey]

This has resolved the issue, thanks!