Unable to Query Data Beyond 8 Days in Thanos Query

[anvar-bidgely]

Description:

We are encountering an issue where querying data beyond 8 days is not possible, whereas, in the past, we were able to retrieve data for up to 20 days. Recently, queries for longer time periods (e.g., 20 days) are returning no data.

Problem Description:
Issue: When querying data for periods longer than 8 days, the data is not returned. The logs show that certain store addresses do not have data for the requested time period.

Previously Working Period: We were able to query data for up to 20 days, but now we are unable to retrieve data beyond 8 days.

Expected Behavior:
When querying data beyond 8 days (e.g., 20 days), the data should be returned, as it was previously working without issues.
Data should be available from the Thanos Store for the configured retention periods, specifically --retention.resolution-raw=7d, --retention.resolution-5m=20d and --retention.resolution-1h=60d.

Logs:
Here is a sample log from when querying for a longer period:

ts=2025-01-14T07:09:48.457343893Z caller=proxy.go:274 level=debug msg="Tenant info in Series()" tenant=default-tenant
ts=2025-01-14T07:09:48.457624482Z caller=proxy.go:320 level=debug component=proxy request="min_time:1736838286201 max_time:1736838586201 matchers:<name:\"resource\" value:\"memory\" > matchers:<name:\"__name__\" value:\"kube_pod_container_resource_limits\" > aggregates:MAX without_replica_labels:\"replica\" " msg="Series: started fanout streams" status="Store Addr: 10.2.99.41:10901 LabelSets: {cluster=\"prodca-eks\", replica=\"prometheus-0\"} MinTime: 1736229600010 MaxTime: 9223372036854775807 queried;Store Addr: 172.20.178.26:10901 LabelSets: {@thanos_compatibility_store_type=\"store\"},{cluster=\"prodca\"},{cluster=\"prodca-eks\", replica=\"prometheus-0\"},{cluster=\"prodca-eks\"} MinTime: 1733356800029 MaxTime: 1736834400000 filtered out due to: does not have data within this time period: [1736838286201,1736838586201]. Store time ranges: [1733356800029,1736834400000]" ```

Thanos Configuration:

Compactor config
`spec:
  containers:
  - args:
    - compact
    - --wait
    - --log.level=debug
    - --log.format=logfmt
    - --http-address=0.0.0.0:10902
    - --data-dir=/data
    - --objstore.config-file=/conf/objstore.yml
    - --retention.resolution-raw=7d
    - --retention.resolution-5m=20d
    - --retention.resolution-1h=60d
    - --delete-delay=48h
    - --compact.concurrency=1`

Store config 
`spec:
  containers:
  - args:
    - store
    - --log.level=info
    - --log.format=logfmt
    - --data-dir=/data
    - --grpc-address=0.0.0.0:10901
    - --http-address=0.0.0.0:10902
    - --objstore.config-file=/conf/objstore.yml
    - --index-cache.config-file=/etc/thanos/index-cache.yml`

Query config
`spec:
  containers:
  - args:
    - query
    - --grpc-address=0.0.0.0:10901
    - --http-address=0.0.0.0:10902
    - --log.level=debug
    - --log.format=logfmt
    - --query.replica-label=replica
    - --endpoint=dnssrv+_grpc._tcp.thanos-storegateway.monitoring.svc.cluster.local
    - --endpoint=dnssrv+_grpc._tcp.prometheus-grpc.monitoring.svc.cluster.local`
    
  Thanos bucket inpect details 
    
    `|            ULID            |         FROM         |        UNTIL         |     RANGE      |  UNTIL-DOWN   |  #SERIES  |   #SAMPLES    |  #CHUNKS   | COMP-LEVEL | COMP-FAILED |                 LABELS                  | RESOLUTION |  SOURCE   |
|----------------------------|----------------------|----------------------|----------------|---------------|-----------|---------------|------------|------------|-------------|-----------------------------------------|------------|-----------|
| 01JG6W7PBYBQE29ZY2SV7BHPK1 | 2024-12-05T00:00:00Z | 2024-12-19T00:00:00Z | 335h59m59.971s | -             | 6,628,008 | 21,200,589    | 6,679,637  | 5          | false       | cluster=prodca                          | 1h0m0s     | compactor |
| 01JH7D6W3DM2W6HB45GJ0XXNCW | 2024-12-23T00:00:00Z | 2024-12-27T00:00:00Z | 95h59m59.978s  | 144h0m0.022s  | 1,914,841 | 54,755,104    | 2,190,150  | 4          | false       | cluster=prodca-eks,replica=prometheus-0 | 5m0s       | compactor |
| 01JG8EEBHWSHNVA03XV9G1BT7H | 2024-12-27T00:00:00Z | 2024-12-29T00:00:00Z | 47h59m59.955s  | 192h0m0.045s  | 624,646   | 26,329,675    | 743,194    | 3          | false       | cluster=prodca-eks                      | 5m0s       | compactor |
| 01JGDK71B8SXH8QBQEDZ9PPA0C | 2024-12-29T00:00:00Z | 2024-12-31T00:00:00Z | 47h59m59.999s  | 192h0m0.001s  | 596,436   | 28,905,058    | 735,419    | 3          | false       | cluster=prodca-eks                      | 5m0s       | compactor |
| 01JGJH60GXTPWPKJXYQNWF31QP | 2024-12-31T00:00:00Z | 2025-01-02T00:00:00Z | 48h0m0s        | 192h0m0s      | 644,618   | 30,100,008    | 800,849    | 3          | false       | cluster=prodca-eks                      | 5m0s       | compactor |
| 01JGQP0SZJQX5SRTGE96YXGB79 | 2025-01-02T00:00:00Z | 2025-01-04T00:00:00Z | 48h0m0s        | 192h0m0s      | 1,056,772 | 33,962,252    | 1,207,725  | 3          | false       | cluster=prodca-eks                      | 5m0s       | compactor |
| 01JGX1RGNS91NBJEC34RQYGW84 | 2025-01-04T08:00:00Z | 2025-01-06T00:00:00Z | 39h59m59.979s  | 21ms          | 321,658   | 456,889,444   | 3,847,327  | 3          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | compactor |
| 01JGVAHAYVP2Q61VJ6AP0QD28P | 2025-01-05T00:00:00Z | 2025-01-05T08:00:00Z | 7h59m59.935s   | 32h0m0.065s   | 172,030   | 104,261,693   | 923,796    | 2          | false       | cluster=prodca-eks                      | 0s         | compactor |
| 01JH26X6QWCRZM1WBY0KYPC5M4 | 2025-01-06T00:00:00Z | 2025-01-08T00:00:00Z | 47h59m59.935s  | -7h59m59.935s | 1,935,561 | 946,347,896   | 8,888,867  | 3          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | compactor |
| 01JH27GNW2HJ5J0KQQZSKZ7T0B | 2025-01-06T00:00:00Z | 2025-01-08T00:00:00Z | 47h59m59.935s  | 192h0m0.065s  | 1,935,561 | 49,570,973    | 2,099,678  | 3          | false       | cluster=prodca-eks,replica=prometheus-0 | 5m0s       | compactor |
| 01JH7BM70S8TRN08YTQ6AFMXSB | 2025-01-08T00:00:00Z | 2025-01-10T00:00:00Z | 47h59m59.99s   | -7h59m59.99s  | 2,126,668 | 1,147,285,134 | 10,692,150 | 3          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | compactor |
| 01JH7C9WTKQS3JJ2859HMT7CXQ | 2025-01-08T00:00:00Z | 2025-01-10T00:00:00Z | 47h59m59.99s   | 192h0m0.01s   | 2,126,666 | 59,951,343    | 2,335,245  | 3          | false       | cluster=prodca-eks,replica=prometheus-0 | 5m0s       | compactor |
| 01JHCG6B2BRF5WMTBA475XCNVR | 2025-01-10T00:00:00Z | 2025-01-12T00:00:00Z | 47h59m59.934s  | -7h59m59.934s | 1,219,486 | 715,545,665   | 6,567,471  | 3          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | compactor |
| 01JHCGJJKBYP6RG8W8R1Q5YS4F | 2025-01-10T00:00:00Z | 2025-01-12T00:00:00Z | 47h59m59.934s  | 192h0m0.066s  | 1,219,486 | 37,235,038    | 1,387,933  | 3          | false       | cluster=prodca-eks,replica=prometheus-0 | 5m0s       | compactor |
| 01JHC78P9YBEXS0PPQC2G5H4G5 | 2025-01-12T00:00:00Z | 2025-01-12T02:00:00Z | 1h59m59.985s   | 38h0m0.015s   | 162,292   | 41,390,053    | 383,180    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHDBHHMW0A16S1S9Y0BJ8HQ6 | 2025-01-12T00:00:00Z | 2025-01-12T08:00:00Z | 7h59m59.985s   | 32h0m0.015s   | 195,786   | 113,704,103   | 1,020,671  | 2          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | compactor |
| 01JHHQMF97NYNY978WWWQN0RX9 | 2025-01-12T00:00:00Z | 2025-01-14T00:00:00Z | 47h59m59.985s  | -7h59m59.985s | 571,376   | 614,706,655   | 5,382,963  | 3          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | compactor |
| 01JHHQVCR6N4ECM5E3AP83JNQ6 | 2025-01-12T00:00:00Z | 2025-01-14T00:00:00Z | 47h59m59.985s  | 192h0m0.015s  | 571,376   | 31,498,242    | 748,108    | 3          | false       | cluster=prodca-eks,replica=prometheus-0 | 5m0s       | compactor |
| 01JHCE4675ED5Z4YYZRY5GC6KS | 2025-01-12T02:00:00Z | 2025-01-12T04:00:00Z | 2h0m0s         | 38h0m0s       | 103,441   | 26,332,744    | 253,988    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHCMZXWAD52CHT858M0DV7WB | 2025-01-12T04:00:00Z | 2025-01-12T06:00:00Z | 1h59m59.934s   | 38h0m0.066s   | 59,136    | 22,936,788    | 194,832    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHCVVMQ5DBV5V8MDF09P4JJV | 2025-01-12T06:00:00Z | 2025-01-12T08:00:00Z | 1h59m59.995s   | 38h0m0.005s   | 49,069    | 23,044,518    | 188,671    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHD2QK9ZR0WPFA68ZFZT141C | 2025-01-12T08:00:00Z | 2025-01-12T10:00:00Z | 1h59m59.99s    | 38h0m0.01s    | 63,640    | 22,408,792    | 192,189    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHE706THK6DWM3HHT84H217P | 2025-01-12T08:00:00Z | 2025-01-12T16:00:00Z | 7h59m59.99s    | 32h0m0.01s    | 84,690    | 91,202,032    | 765,609    | 2          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | compactor |
| 01JHD9K374PXFRWKZYZX1TVG18 | 2025-01-12T10:00:00Z | 2025-01-12T12:00:00Z | 1h59m59.946s   | 38h0m0.054s   | 63,909    | 22,747,575    | 193,971    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHDGETT9WE17XT9GVCFFNTWK | 2025-01-12T12:00:00Z | 2025-01-12T14:00:00Z | 1h59m59.934s   | 38h0m0.066s   | 50,605    | 22,857,154    | 189,234    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHDQAHQ96P6PFZMWH4J2FD8Q | 2025-01-12T14:00:00Z | 2025-01-12T16:00:00Z | 1h59m59.976s   | 38h0m0.024s   | 49,090    | 23,188,511    | 190,215    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHDY68Z54GN40J4C694GWFH1 | 2025-01-12T16:00:00Z | 2025-01-12T18:00:00Z | 1h59m59.938s   | 38h0m0.062s   | 49,136    | 23,324,741    | 191,524    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHF2F5CRDKJKPT74Q4FVWTNM | 2025-01-12T16:00:00Z | 2025-01-13T00:00:00Z | 7h59m59.938s   | 32h0m0.062s   | 94,304    | 96,208,337    | 813,410    | 2          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | compactor |
| 01JHE520J9Y3EQ5X0RKPPAV5JF | 2025-01-12T18:00:00Z | 2025-01-12T20:00:00Z | 1h59m59.934s   | 38h0m0.066s   | 49,265    | 23,376,510    | 191,940    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHEBXQF70Z4QH187QMMREBFE | 2025-01-12T20:00:00Z | 2025-01-12T22:00:00Z | 1h59m59.963s   | 38h0m0.037s   | 49,265    | 23,395,975    | 192,154    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHEJSEQ4Y2AAQACAB6BSP6BA | 2025-01-12T22:00:00Z | 2025-01-13T00:00:00Z | 1h59m59.992s   | 38h0m0.008s   | 94,304    | 26,111,111    | 237,792    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHESN5Z6CS0W2NS21MSYDX0B | 2025-01-13T00:00:00Z | 2025-01-13T02:00:00Z | 1h59m59.963s   | 38h0m0.037s   | 98,176    | 26,259,593    | 239,225    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHHQD5RCPAV640N6B15FNVRA | 2025-01-13T00:00:00Z | 2025-01-13T08:00:00Z | 7h59m59.963s   | 32h0m0.037s   | 101,311   | 97,561,048    | 824,592    | 2          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | compactor |
| 01JHF0GX7VCB0RYKTHD5YW2H5G | 2025-01-13T02:00:00Z | 2025-01-13T04:00:00Z | 1h59m59.985s   | 38h0m0.015s   | 49,387    | 23,260,290    | 191,109    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHF7CMW4KAN7V265MDGBSS22 | 2025-01-13T04:00:00Z | 2025-01-13T06:00:00Z | 1h59m59.934s   | 38h0m0.066s   | 50,794    | 23,754,762    | 195,046    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHFE8BQ9WTV30EDT62Q9R215 | 2025-01-13T06:00:00Z | 2025-01-13T08:00:00Z | 1h59m59.959s   | 38h0m0.041s   | 51,578    | 24,286,403    | 199,212    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHFN42Z66HNJYHNBKVZ5TDVS | 2025-01-13T08:00:00Z | 2025-01-13T10:00:00Z | 1h59m59.991s   | 38h0m0.009s   | 51,658    | 24,512,180    | 201,457    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHHQEJVVPF147CKVCF9S7VG1 | 2025-01-13T08:00:00Z | 2025-01-13T16:00:00Z | 7h59m59.991s   | 32h0m0.009s   | 82,230    | 100,981,255   | 849,631    | 2          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | compactor |
| 01JHFVZT79YHJDKCP6KWEKWV11 | 2025-01-13T10:00:00Z | 2025-01-13T12:00:00Z | 1h59m59.971s   | 38h0m0.029s   | 53,919    | 24,760,957    | 204,162    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHG2VHF5QMM65E3HG6Z3K4GD | 2025-01-13T12:00:00Z | 2025-01-13T14:00:00Z | 1h59m59.976s   | 38h0m0.024s   | 72,730    | 26,166,513    | 227,013    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHG9Q8Q9R1T64MK2M49T7DVB | 2025-01-13T14:00:00Z | 2025-01-13T16:00:00Z | 1h59m59.997s   | 38h0m0.003s   | 63,268    | 25,541,605    | 216,999    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHGGK0C203S9GK1DS57FV81H | 2025-01-13T16:00:00Z | 2025-01-13T18:00:00Z | 1h59m59.934s   | 38h0m0.066s   | 154,383   | 27,867,000    | 291,219    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHHQG9ET0975CD89RN2F87H3 | 2025-01-13T16:00:00Z | 2025-01-14T00:00:00Z | 7h59m59.934s   | 32h0m0.066s   | 293,444   | 115,049,880   | 1,109,050  | 2          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | compactor |
| 01JHGQEQM111YC2TZ06TZA01HW | 2025-01-13T18:00:00Z | 2025-01-13T20:00:00Z | 1h59m59.934s   | 38h0m0.066s   | 97,546    | 26,183,738    | 248,052    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHGYAEFDV039CKHNHD6KHEKG | 2025-01-13T20:00:00Z | 2025-01-13T22:00:00Z | 1h59m59.984s   | 38h0m0.016s   | 88,232    | 26,662,161    | 242,459    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHH565QEWD1G0ME6XZ5B72XY | 2025-01-13T22:00:00Z | 2025-01-14T00:00:00Z | 1h59m59.997s   | 38h0m0.003s   | 139,167   | 34,336,981    | 327,320    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHHC1X5DBKNKA717JCZV8FK3 | 2025-01-14T00:00:00Z | 2025-01-14T02:00:00Z | 1h59m59.934s   | 38h0m0.066s   | 186,164   | 36,857,951    | 369,486    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHHJXV26CEVF6E2T86X7RWJF | 2025-01-14T02:00:00Z | 2025-01-14T04:00:00Z | 1h59m59.988s   | 38h0m0.012s   | 276,029   | 43,967,456    | 430,291    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
| 01JHHSSBF94E7FJ98ZC34R0QBD | 2025-01-14T04:00:00Z | 2025-01-14T06:00:00Z | 2h0m0s         | 38h0m0s       | 207,251   | 44,879,457    | 459,685    | 1          | false       | cluster=prodca-eks,replica=prometheus-0 | 0s         | sidecar   |
ts=2025-01-14T07:29:25.503887965Z caller=main.go:174 level=info msg=exiting`

[dosubot]

Hey @anvar-bidgely! 👋 I'm here to help you with bug fixes, questions, and becoming a contributor.

The issue you're experiencing with querying data beyond 8 days in Thanos Query could be related to the retention settings and the availability of data in the Thanos Store. Here are a few points to consider:

1. Retention Settings: Your current retention settings are --retention.resolution-raw=7d, --retention.resolution-5m=20d, and --retention.resolution-1h=60d. This means that raw data is only retained for 7 days. If you are querying for data beyond 8 days, the raw data will not be available, and you will need to rely on the downsampled data (5m or 1h resolution) ^[1].

2. Data Availability: The logs indicate that certain store addresses do not have data for the requested time period. This could be due to the retention policy deleting raw data after 7 days, and if the data has not been downsampled to 5m or 1h resolution, it will not be available for queries beyond 8 days.

3. Compactor Role: Ensure that the Thanos Compactor is running correctly and performing downsampling as expected. The compactor is responsible for downsampling data, which allows you to query longer time ranges efficiently ^[2].

4. Bucket Inspection: From the bucket inspection details, verify that there are blocks available for the time range you are querying. If blocks for the desired time range are missing, it could indicate an issue with data retention or compaction.

5. Configuration Check: Double-check your Thanos Store and Compactor configurations to ensure they are set up correctly to handle the retention and downsampling processes.

By reviewing these aspects, you should be able to identify why data beyond 8 days is not being returned and make the necessary adjustments to your Thanos setup.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}