422 Unprocessable Entity, body: \"block time range crosses boundary of configured compactor time range (1d)

[mifinn85]

Hi,

We are trying to upload TSDB blocks from Prometheus to Mimir but it keeps failing on time range set to 1d, older block files are not being uploaded.

Where exactly we need to change this setting from 1d to a different value.

Helm Release:

  chart: mimir-distributed

  version: 5.6.0-weekly.316

Error:

/prometheus $ ./mimirtool backfill --address=http://mimir-compactor.monitoring.svc.cluster.local:8080 --id=anonymous /prometheus/01J9HCHX3VV8W28445J8G7DGEC
INFO[0000] Backfilling blocks=/prometheus/01J9HCHX3VV8W28445J8G7DGEC user=anonymous
INFO[0000] making request to start block upload block=01J9HCHX3VV8W28445J8G7DGEC file=meta.json path=/prometheus/01J9HCHX3VV8W28445J8G7DGEC
ERRO[0000] response body="block time range crosses boundary of configured compactor time range (1d)\n" status="422 Unprocessable Entity"
ERRO[0000] failed uploading block error="request to start block upload failed: POST request to http://mimir-compactor.monitoring.svc.cluster.local:8080/api/v1/upload/block
/01J9HCHX3VV8W28445J8G7DGEC/start failed: server returned HTTP status: 422 Unprocessable Entity, body: "block time range crosses boundary of configured compactor time range (1d)\n"" path=/pro
metheus/01J9HCHX3VV8W28445J8G7DGEC
INFO[0000] finished uploading blocks already_exists=0 failed=1 succeeded=0
mimirtool: error: blocks failed to upload 1 block(s), try --help

[mifinn85]

Would really appreciate some help here please.

[andyasp]

Hello,

Where exactly we need to change this setting from 1d to a different value.

I would not suggest changing the -compactor.block-ranges setting. It's best to leave that at the default.

ERRO[0000] response body="block time range crosses boundary of configured compactor time range (1d)\n" status="422 Unprocessable Entity"

The error you are encountering is due to an uploaded block not aligning to the time boundaries that are expected by Mimir. Mimir, by default, expects at most 24-hour blocks aligned by day. For example, if you try to upload a block that spans noon of one day to noon of the following day you would hit this error. This validation was added in #9524. Ensuring this alignment helps avoid encountering performance issues later on in the query path.

The solution is to use a tool in the Mimir repository, splitblocks, to split the original block into multiple blocks that have an expected alignment. Then those blocks can be uploaded instead. It would certainly be nice if the backfill did this for you, but that hasn't been implemented.