MimirBucketIndexNotUpdated with multiple tenants

[TA3]

I have a self-hosted Mimir instances installed on a Kubernetes cluster with S3. Mimir is setup with mutil-tenancy enabled with around 12 tenants each pushing around 6000 samples per second.

For the past two months we've been seeing MimirBucketIndexNotUpdated alert, we've had a bunch of tweaking of the config based on some of the issues on Github, some of which has helped, but it's still happening at times. I did follow the steps provided in the runbook, that unfortunately didn't seem to help.

The only error logs that we see from the Mimir components are the following:

• query-frontend - A bunch of these errors from various users (tenants)
  caller=retry.go:89 user=_local trace_id=4a50cf8a3ae421c7 level=error user=_local traceID=4a50cf8a3ae421c7 msg="error processing request" try=4 err="expanding series: the bucket index is too old. It was last updated at 2025-02-14T13:36:44Z, which exceeds the maximum allowed staleness period of 2h0m0s (err-mimir-bucket-index-too-old)"
• querier - A bunch of these errors from various users (tenants)
  caller=scheduler_processor.go:249 level=error user=_local traceID=4a50cf8a3ae421c7 msg="error notifying scheduler about finished query after the scheduler stream previously failed and returned error" err=EOF addr=10.244.76.246:9095 previousErr="rpc error: code = Canceled desc = context canceled: frontend cancelled query"
• store-gateway - A bunch of these errors from various users (tenants)
  caller=bucket.go:450 level=error user=_local msg="loading block failed" elapsed=2.356089ms id=01JM0JHB7PJ658NBN75DBW5R7A err="create index header reader: write index header: new index reader: get object attributes of 01JM0JHB7PJ658NBN75DBW5R7A/index: The specified key does not exist."

From the compactor we get loads of these Warning messages again for pretty much all tenants:
caller=updater.go:131 level=warn component=cleaner run_id=1740402538 task=clean_up_users user=_local msg="skipped partial block when updating bucket index" block=01JHK9WTYWBZR00WKT8KB9QMWB

Mimir Config:

multitenancy_enabled: true
    usage_stats:
      enabled: false
    server:
      grpc_server_max_concurrent_streams: 1000
    common:
      storage:
        backend: s3
        s3:
          endpoint: minio.storage.svc:9000
          access_key_id: user
          secret_access_key: pass
          insecure: true
          trace:
            enabled: true
          http:
            idle_conn_timeout: 5m
            response_header_timeout: 5m
            expect_continue_timeout: 5s
    distributor:
      instance_limits:
        max_ingestion_rate: 30000
        max_inflight_push_requests: 150
    frontend:
      log_queries_longer_than: 1m
    querier:
      query_store_after: 12h
    limits:
      ingestion_rate: 30000
      max_global_series_per_user: 1000000 
      max_global_series_per_metric: 100000
      max_label_names_per_series: 100
      cardinality_analysis_enabled: true
      compactor_blocks_retention_period: 14d 
      query_ingesters_within: 13h
      out_of_order_time_window: 15m
      max_cache_freshness: 15m
      compactor_tenant_shard_size: 2
    blocks_storage:
      bucket_store:
        ignore_blocks_within: 10h
        bucket_index:
          max_stale_period: 2h
      tsdb:
        retention_period: 24h
      s3:
        bucket_name: mimir-data
    ruler_storage:
      s3:
        bucket_name: mimir-ruler
    alertmanager_storage:
      s3:
        bucket_name: mimir-alerts
    compactor: 
      max_compaction_time: 12h
      compaction_concurrency: 4
      meta_sync_concurrency: 200
      block_sync_concurrency: 100
      cleanup_interval: 5m

We have investigated our S3 (Minio) but there are no errors on there, Loki and Tempo also uses the same Minio instance with no issues at all.