Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unbounded batching span processor #7099

Open
dimitarg opened this issue Feb 11, 2025 · 1 comment
Open

Unbounded batching span processor #7099

dimitarg opened this issue Feb 11, 2025 · 1 comment
Labels
Feature Request Suggest an idea for this project

Comments

@dimitarg
Copy link

Is your feature request related to a problem? Please describe.
The default span processor which is a batch span processor, and the default maximum queue size which is 2048, are very susceptible to span loss for a system under any non-trivial load.

Describe the solution you'd like
It should be possible, with the java opentelemetry SDK, to configure a span processor which is batching but has an unbounded queue size, to ensure no span loss under arbitrary load, at the cost of unbounded memory usage.

Describe alternatives you've considered
With the current jctools-based implementation, one can configure otel.bsp.max.queue.size to a very large value, at most 1073741824. This probably achieves the same in practice, but is a workaround.

Additional context

I'm unsure whether adding the possibility of making the queue unbounded contradicts the specification at https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#batching-processor

If so, perhaps a new batching processor type can be introduced, UnboundedBatchingSpanProcessor.

@dimitarg dimitarg added the Feature Request Suggest an idea for this project label Feb 11, 2025
@trask
Copy link
Member

trask commented Feb 11, 2025

I've generally seen this (span loss due to batch span processor overflowing) when hitting a remote ingestion endpoint.

I personally think #4264 is a better solution to this particular problem at least.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature Request Suggest an idea for this project
Projects
None yet
Development

No branches or pull requests

2 participants