Timeouts manifesting as consistency issues (err-mimir-store-consistency-check-failed)

[callumj]

We have this limit: querier.timeout: 1m which we are using as a forcing function to ensure we have good queries written, but this seems to interact with the Ruler consistency check:

Ruler:

{"caller":"spanlogger.go:80","err":null,"level":"warn","method":"blocksStoreQuerier.selectSorted","msg":"failed consistency check","ts":"2023-01-12T19:31:17.774927229Z","user":"lombard"}
{"caller":"manager.go:650","err":"the consistency check failed because some blocks were not queried (err-mimir-store-consistency-check-failed). The non-queried blocks are: 01GPCPB6YJ288WMT6TCJQY7JNQ","file":"/rules/lombard/managed_alerts"

Store Gateway:

{"caller":"grpc_logging.go:64","duration":"1m0.023788627s","err":"rpc error: code = Aborted desc = fetch series for block 01GPCPB6YJ288WMT6TCJQY7JNQ: load chunks: fast forward range reader: context canceled","level":"warn","method":"/gatewaypb.StoreGateway/Series","msg":"gRPC","ts":"2023-01-12T19:54:17.864728929Z"}
{"caller":"grpc_logging.go:64","duration":"1m0.067047097s","err":"rpc error: code = Aborted desc = fetch series for block 01GPCPB6YJ288WMT6TCJQY7JNQ: load chunks: fast forward range reader: context canceled","level":"warn","method":"/gatewaypb.StoreGateway/Series","msg":"gRPC","ts":"2023-01-12T19:57:17.911995753Z"}
{"caller":"grpc_logging.go:64","duration":"1m4.720462939s","err":"rpc error: code = Aborted desc = fetch series for block 01GPCPB6YJ288WMT6TCJQY7JNQ: load chunks: get range reader: context canceled","level":"warn","method":"/gatewaypb.StoreGateway/Series","msg":"gRPC","ts":"2023-01-12T20:02:22.573460033Z"}

The duration seems to be quite close the timeout so I am guessing that this isn't actually a consistency issue but instead the Ruler is aborting the operation after 1m, causing a context canceled on the Store Gateway which manifests itself back as a consistency issue on the Ruler side?

[56quarters]

Your analysis seems correct to me. There was a fix for this just recently merged which should be in the next Mimir release: #3837

[callumj]

My mistake, I should have checked the issues - thanks for getting back so quick!