Out-of-Order Sample Errors with Alloy Remote Write to Mimir

[girish-ms]

Description

We are experiencing frequent out-of-order sample rejections when using Alloy to remote_write to Mimir. The issue occurs across multiple metric series, and the affected samples have timestamps that are significantly different from each other.

Alloy Logs:(example)

ts=2025-03-07T13:42:26.330800121Z level=error msg="non-recoverable error" component_path=/ component_id=prometheus.remote_write.default subcomponent=rw remote_name=985df6 url=http://mimir-distributor.mimir.svc.cluster.local:8080/api/v1/push failedSampleCount=38 failedHistogramCount=0 failedExemplarCount=0 err="server returned HTTP status 400 Bad Request: send data to ingesters: failed pushing to ingester mimir-ingester-0: user=anonymous: the sample has been rejected because another sample with a more recent timestamp has already been ingested and out-of-order samples are not allowed (err-mimir-sample-out-of-order). The affected sample has timestamp 2025-03-07T13:04:02.887Z and is from series go_memory_classes_heap_objects_bytes{instance=\"ip-xxxxxxxx.us-west-2.compute.internal\", job=\"integrations/kubernetes/kubelet\"} (sampled 1/10)"

Distributor Logs:(example)

ts=2025-03-07T13:42:07.060631562Z caller=push.go:221 level=error user=anonymous msg="detected an error while ingesting Prometheus remote-write request (the request may have been partially ingested)" httpCode=400 err="send data to ingesters: failed pushing to ingester mimir-ingester-3: user=anonymous: the sample has been rejected because another sample with a more recent timestamp has already been ingested and out-of-order samples are not allowed (err-mimir-sample-out-of-order). The affected sample has timestamp 2025-03-07T13:06:52.418Z and is from series container_cpu_load_average_10s{id=\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe26629_5479_483e_8e29_ec171951e388.slice/cri-containerd-b192c4e181597b618062b91cd71e7ca05ea9298cb7e0773b9ab5d342372841e8.scope\", image=\"xxxxxx.dkr.ecr.us-west-2.amazonaws.com/eks/pause:3.5\", instance=\"ip-xxxx.us-west-2.compute.internal\", job=\"integrations/kubernetes/cadvisor\", name=\"b192c4e181597b618062b91cd71e7ca05ea9298cb7e0773b9ab5d342372841e8\", namespace=\"kube-system\", pod=\"node-xxxxx\"} (sampled 1/10)"

Observations

• The timestamps of rejected samples are several minutes apart, making it unclear why they are being classified as out-of-order.
• Logs show errors across Alloy, Mimir Distributor, and Mimir Ingester, indicating the problem exists throughout the ingestion pipeline.
• The issue is not isolated to a single job or namespace.

Questions:

• What debugging steps can help determine whether the issue originates from Alloy, Mimir Distributor, or Mimir Ingester?
• Are there any best practices or mitigation strategies to prevent out-of-order sample rejections in this setup?

Tagging @pracucci for quick resolution. Any insights would be greatly appreciated!