mimir Program cannot be started

[rayn316]

Describe the bug

mimir Program cannot be started
I'm planning to migrate to the new service from elsewhere, but I'm having this problem
Very strange, it starts up fine on other servers, but something seems to be missing on this new vps
I don't know how to solve this problem

To Reproduce

https://grafana.com/docs/mimir/latest/operators-guide/get-started/
Follow the tutorial to download the binaries, fill in the examples and start

Expected behavior

Normal start

Environment

os: ubuntu 20.04
Service provider: contabo

Additional Context

error text:

level=error ts=2022-11-08T08:24:38.795613501Z caller=mimir.go:774 msg="module failed" module=memberlist-kv err="invalid service state: Stopping, expected: Running"
level=error ts=2022-11-08T08:24:38.795685666Z caller=mimir.go:774 msg="module failed" module=store-gateway err="failed to start store-gateway, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running"
level=error ts=2022-11-08T08:24:38.795727164Z caller=mimir.go:774 msg="module failed" module=ingester-service err="failed to start ingester-service, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running"
level=error ts=2022-11-08T08:24:38.795740268Z caller=mimir.go:774 msg="module failed" module=distributor-service err="failed to start distributor-service, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running"
level=error ts=2022-11-08T08:24:38.795751039Z caller=mimir.go:774 msg="module failed" module=compactor err="failed to start compactor, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running"
level=error ts=2022-11-08T08:24:38.795760306Z caller=mimir.go:774 msg="module failed" module=store-queryable err="failed to start store-queryable, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running"
level=error ts=2022-11-08T08:24:38.795769944Z caller=mimir.go:774 msg="module failed" module=ring err="failed to start ring, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running"
level=error ts=2022-11-08T08:24:38.795782588Z caller=mimir.go:774 msg="module failed" module=query-frontend err="failed to start query-frontend, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running"
level=error ts=2022-11-08T08:24:38.795799459Z caller=mimir.go:774 msg="module failed" module=querier err="failed to start querier, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running"
level=error ts=2022-11-08T08:24:38.795808747Z caller=mimir.go:774 msg="module failed" module=ruler err="failed to start ruler, because it depends on module ring, which has failed: context canceled"
level=error ts=2022-11-08T08:24:38.797185786Z caller=log.go:60 msg="error running application" err="failed services\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/go/src/github.com/grafana/mimir/pkg/mimir/mimir.go:809\nmain.main\n\t/go/src/github.com/grafana/mimir/cmd/mimir/main.go:212\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"

[dimitarvdimitrov]

Hi, are there any other log lines before the ones you posted? Can you please post the whole stdout/stderr output of mimir?

[rayn316]

@dimitarvdimitrov This log is started without a configuration file

level=info ts=2022-11-08T09:33:23.82610506Z caller=main.go:210 msg="Starting application" version="(version=2.4.0, branch=release-2.4, revision=32137ee2c)"
level=info ts=2022-11-08T09:33:23.828560288Z caller=server.go:306 http=[::]:8080 grpc=[::]:9095 msg="server listening on addresses"
level=warn ts=2022-11-08T09:33:23.838431612Z caller=ingester.go:254 msg="-blocks-storage.backend=filesystem is for development and testing only; you should switch to an external object store for production use or use a shared filesystem"
level=info ts=2022-11-08T09:33:23.842459405Z caller=ingester.go:336 msg="TSDB idle compaction timeout set" timeout=1h11m23.530738682s
level=info ts=2022-11-08T09:33:23.84322567Z caller=memberlist_client.go:436 msg="Using memberlist cluster label and node name" cluster_label= node=m-grafana.blockpi.io-f10d6abe
level=info ts=2022-11-08T09:33:23.844350193Z caller=inmemory.go:173 msg="created in-memory index cache" maxItemSizeBytes=134217728 maxSizeBytes=1073741824 maxItems=maxInt
level=info ts=2022-11-08T09:33:23.848239067Z caller=modules.go:563 msg="The ruler is not being started because you need to configure the ruler storage."
level=info ts=2022-11-08T09:33:23.84964847Z caller=modules.go:416 msg="The querier worker has not been configured with either the query-frontend or query-scheduler address. Because Mimir is running in monolithic mode, it's attempting an automatic worker configuration. If queries are unresponsive, consider explicitly configuring the query-frontend or query-scheduler address for querier worker." address=127.0.0.1:9095
level=info ts=2022-11-08T09:33:23.849701238Z caller=worker.go:130 msg="Starting querier worker connected to query-frontend" frontend=127.0.0.1:9095
level=info ts=2022-11-08T09:33:23.85007864Z caller=modules.go:573 msg="RulerStorage is nil.  Not starting the ruler."
level=info ts=2022-11-08T09:33:23.850385661Z caller=module_service.go:82 msg=initialising module=sanity-check
level=info ts=2022-11-08T09:33:23.850538435Z caller=sanity_check.go:32 msg="Checking directories read/write access"
level=info ts=2022-11-08T09:33:23.850858511Z caller=module_service.go:82 msg=initialising module=activity-tracker
level=info ts=2022-11-08T09:33:23.850866936Z caller=module_service.go:82 msg=initialising module=usage-stats
level=info ts=2022-11-08T09:33:23.850912461Z caller=sanity_check.go:37 msg="Directories read/write access successfully checked"
level=info ts=2022-11-08T09:33:23.850948819Z caller=sanity_check.go:39 msg="Checking object storage config"
level=info ts=2022-11-08T09:33:23.851402212Z caller=sanity_check.go:44 msg="Object storage config successfully checked"
level=info ts=2022-11-08T09:33:23.851466161Z caller=module_service.go:82 msg=initialising module=server
level=info ts=2022-11-08T09:33:23.851539899Z caller=module_service.go:82 msg=initialising module=memberlist-kv
level=error ts=2022-11-08T09:33:23.851607144Z caller=mimir.go:774 msg="module failed" module=memberlist-kv err="invalid service state: Stopping, expected: Running"
level=error ts=2022-11-08T09:33:23.851661645Z caller=mimir.go:774 msg="module failed" module=querier err="failed to start querier, because it depends on module store-queryable, which has failed: context canceled"
level=error ts=2022-11-08T09:33:23.851679397Z caller=mimir.go:774 msg="module failed" module=store-queryable err="failed to start store-queryable, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running"
level=error ts=2022-11-08T09:33:23.851762152Z caller=mimir.go:774 msg="module failed" module=ingester-service err="failed to start ingester-service, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running"
level=error ts=2022-11-08T09:33:23.851785004Z caller=mimir.go:774 msg="module failed" module=compactor err="failed to start compactor, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running"
level=error ts=2022-11-08T09:33:23.851796616Z caller=mimir.go:774 msg="module failed" module=ring err="failed to start ring, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running"
level=error ts=2022-11-08T09:33:23.851819809Z caller=mimir.go:774 msg="module failed" module=query-frontend err="failed to start query-frontend, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running"
level=error ts=2022-11-08T09:33:23.851852049Z caller=mimir.go:774 msg="module failed" module=distributor-service err="failed to start distributor-service, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running"
level=error ts=2022-11-08T09:33:23.851926939Z caller=mimir.go:774 msg="module failed" module=store-gateway err="failed to start store-gateway, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running"
level=info ts=2022-11-08T09:33:23.852189006Z caller=server_service.go:55 msg="server stopped"
level=info ts=2022-11-08T09:33:23.852237576Z caller=module_service.go:114 msg="module stopped" module=server
level=info ts=2022-11-08T09:33:23.85229352Z caller=module_service.go:114 msg="module stopped" module=usage-stats
level=info ts=2022-11-08T09:33:23.852372177Z caller=module_service.go:114 msg="module stopped" module=sanity-check
level=info ts=2022-11-08T09:33:23.855506197Z caller=module_service.go:114 msg="module stopped" module=activity-tracker
level=info ts=2022-11-08T09:33:23.855586396Z caller=mimir.go:763 msg="Application stopped"
level=error ts=2022-11-08T09:33:23.855667357Z caller=log.go:60 msg="error running application" err="failed services\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/go/src/github.com/grafana/mimir/pkg/mimir/mimir.go:809\nmain.main\n\t/go/src/github.com/grafana/mimir/cmd/mimir/main.go:212\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"

[dimitarvdimitrov]

I can't see anything obvious in these logs. Can you also enable debug logging and share the logs? It's enabled via YAML or CLI flag

server:
  log_level: debug

or

-log.level=debug

[rayn316]

cmd: mimir -log.level=debug
error.txt

[aknuds1]

From the latest attached logs (error.txt), this looks like the originating error:

failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided

So basically looks as if a private IP address can't be detected on your machine. I'm not familiar enough with this component to tell if the problem lies elsewhere.

[pracucci]

Mimir tried to auto-detect a private IP from the available host interfaces to expose memberlist. If can't find any private IP it fails. However you can manually set the IP of the interface on which open the memberlist port and then advertise to other memberlist members using the following configuration option:

# IP address to listen on for gossip messages. Multiple addresses may be
# specified. Defaults to 0.0.0.0
# CLI flag: -memberlist.bind-addr
[bind_addr: <list of strings> | default = []]

[aknuds1]

Should we move this to a discussion, since it doesn't quite look like a Mimir bug @dimitarvdimitrov?

[ctml91]

Hitting this error on an OpenShift Deployment.

1. Start with values-small.yaml

2. Adjust settings for deploying on OCP based on PR helm: added OpenShift support #2219 (rbac/scc)

3. Still fails to deploy as per Docs: Write documentation for deploying on OpenShift Container Platform (OCP) #4040

4. Adjust values to set seccompProfile: null for every component deployed in mimir

5. Some pods are now created, but they crash with the error message in this discussion. There are still services failing due to seccompProfile being set.

$ oc get pods
NAME                                        READY   STATUS             RESTARTS        AGE
mimir-alertmanager-0                        0/1     CrashLoopBackOff   6 (3m3s ago)    9m5s
mimir-compactor-0                           0/1     CrashLoopBackOff   6 (2m59s ago)   9m4s
mimir-distributor-8596c9d64f-hrrmv          0/1     CrashLoopBackOff   6 (3m8s ago)    9m5s
mimir-distributor-8596c9d64f-l75m5          0/1     CrashLoopBackOff   6 (3m22s ago)   9m5s
mimir-ingester-zone-a-0                     0/1     CrashLoopBackOff   6 (2m57s ago)   9m4s
mimir-ingester-zone-b-0                     0/1     CrashLoopBackOff   6 (2m59s ago)   9m4s
mimir-ingester-zone-c-0                     0/1     CrashLoopBackOff   6 (3m16s ago)   9m5s
mimir-overrides-exporter-789db77b46-g76pn   1/1     Running            0               4m11s
mimir-querier-6d95f8d66-wtf9f               0/1     CrashLoopBackOff   6 (3m13s ago)   9m5s
mimir-query-frontend-7565579cbc-ttmt4       0/1     CrashLoopBackOff   5 (63s ago)     4m11s
mimir-rollout-operator-7f987c5f5b-5clwr     1/1     Running            0               9m9s
mimir-ruler-76f99f864c-qld5b                0/1     CrashLoopBackOff   6 (3m14s ago)   9m5s
mimir-store-gateway-zone-a-0                0/1     CrashLoopBackOff   6 (3m11s ago)   9m4s
mimir-store-gateway-zone-b-0                0/1     CrashLoopBackOff   6 (3m4s ago)    9m5s
mimir-store-gateway-zone-c-0                0/1     CrashLoopBackOff   6 (2m59s ago)   9m4s

minio:
  enabled: false
rbac:
  create: true
  type: scc
serviceAccount:
  create: true
common:
  storage:
    backend: s3
    s3:
      endpoint: minio-service.minio.svc
      region: minio
      secret_access_key: "XXXX"
      access_key_id: "XXXX"

alertmanager:
  persistentVolume:
    enabled: true
  replicas: 2
  resources:
    limits:
      memory: 2Gi
    requests:
      cpu: 1
      memory: 1Gi
  statefulSet:
    enabled: true
  securityContext:
    fsGroup: 10001
    runAsGroup: 10001
    runAsNonRoot: true
    runAsUser: 10001
    seccompProfile: null

compactor:
  persistentVolume:
    size: 20Gi
  resources:
    limits:
      memory: 2Gi
    requests:
      cpu: 1
      memory: 1Gi
  securityContext:
    fsGroup: 10001
    runAsGroup: 10001
    runAsNonRoot: true
    runAsUser: 10001
    seccompProfile: null

distributor:
  replicas: 2
  resources:
    limits:
      memory: 6Gi
    requests:
      cpu: 2
      memory: 4Gi
  securityContext:
    fsGroup: 10001
    runAsGroup: 10001
    runAsNonRoot: true
    runAsUser: 10001
    seccompProfile: null

ingester:
  persistentVolume:
    size: 50Gi
  replicas: 3
  resources:
    limits:
      memory: 12Gi
    requests:
      cpu: 1
      memory: 8Gi
  topologySpreadConstraints: {}
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: target # support for enterprise.legacyLabels
                operator: In
                values:
                  - ingester
          topologyKey: 'kubernetes.io/hostname'

        - labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/component
                operator: In
                values:
                  - ingester
          topologyKey: 'kubernetes.io/hostname'

  zoneAwareReplication:
    topologyKey: 'kubernetes.io/hostname'
  securityContext:
    fsGroup: 10001
    runAsGroup: 10001
    runAsNonRoot: true
    runAsUser: 10001
    seccompProfile: null

chunks-cache:
  enabled: true
  replicas: 2
  securityContext:
    fsGroup: 10001
    runAsGroup: 10001
    runAsNonRoot: true
    runAsUser: 10001
    seccompProfile: null

index-cache:
  enabled: true
  replicas: 3
  securityContext:
    fsGroup: 10001
    runAsGroup: 10001
    runAsNonRoot: true
    runAsUser: 10001
    seccompProfile: null

metadata-cache:
  enabled: true
  securityContext:
    fsGroup: 10001
    runAsGroup: 10001
    runAsNonRoot: true
    runAsUser: 10001
    seccompProfile: null

results-cache:
  enabled: true
  replicas: 2
  securityContext:
    fsGroup: 10001
    runAsGroup: 10001
    runAsNonRoot: true
    runAsUser: 10001
    seccompProfile: null

minio:
  enabled: false

overrides_exporter:
  replicas: 1
  resources:
    limits:
      memory: 128Mi
    requests:
      cpu: 100m
      memory: 128Mi
  securityContext:
    fsGroup: 10001
    runAsGroup: 10001
    runAsNonRoot: true
    runAsUser: 10001
    seccompProfile: null

querier:
  replicas: 1
  resources:
    limits:
      memory: 6Gi
    requests:
      cpu: 2
      memory: 4Gi
  securityContext:
    fsGroup: 10001
    runAsGroup: 10001
    runAsNonRoot: true
    runAsUser: 10001
    seccompProfile: null

query_frontend:
  replicas: 1
  resources:
    limits:
      memory: 3Gi
    requests:
      cpu: 2
      memory: 2Gi
  securityContext:
    fsGroup: 10001
    runAsGroup: 10001
    runAsNonRoot: true
    runAsUser: 10001
    seccompProfile: null

ruler:
  replicas: 1
  podAnnotations: null
  annotations: null
  resources:
    limits:
      memory: 3Gi
    requests:
      cpu: 1
      memory: 2Gi
  securityContext:
    fsGroup: 10002
    runAsGroup: 10001
    runAsNonRoot: true
    runAsUser: 10001
    seccompProfile: null

store_gateway:
  persistentVolume:
    size: 10Gi
  replicas: 3
  resources:
    limits:
      memory: 2Gi
    requests:
      cpu: 1
      memory: 1Gi
  topologySpreadConstraints: {}
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: target # support for enterprise.legacyLabels
                operator: In
                values:
                  - store-gateway
          topologyKey: 'kubernetes.io/hostname'

        - labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/component
                operator: In
                values:
                  - store-gateway
          topologyKey: 'kubernetes.io/hostname'
  zoneAwareReplication:
    topologyKey: 'kubernetes.io/hostname'
  securityContext:
    fsGroup: 10001
    runAsGroup: 10001
    runAsNonRoot: true
    runAsUser: 10001
    seccompProfile: null

nginx:
  replicas: 1
  resources:
    limits:
      memory: 1Gi
    requests:
      cpu: 1
      memory: 512Mi
  securityContext:
    fsGroup: 10001
    runAsGroup: 10001
    runAsNonRoot: true
    runAsUser: 10001
    seccompProfile: null

[pracucci]

Some pods are now created, but they crash with the error message in this discussion

Can you clarify what is the "error message in this discussion", please? If you mean "failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided" then I would need more details about the network interfaces reported inside the container (e.g. could you run ifconfig and show its output), to better understand why it can't detect any private IP address.

[ctml91]

@pracucci yes specifically the failed to start due to memberlist-kv being failed.

Alert Manager

level=info ts=2023-02-08T15:19:50.606689377Z caller=main.go:210 msg="Starting application" version="(version=2.5.0, branch=HEAD, revision=25533fd)"
level=info ts=2023-02-08T15:19:50.610212894Z caller=server.go:323 http=[::]:8080 grpc=[::]:9095 msg="server listening on addresses"
level=warn ts=2023-02-08T15:19:50.614870728Z caller=store.go:63 msg="-alertmanager-storage.backend=filesystem is for development and testing only; you should switch to an external object store for production use or use a shared filesystem"
level=info ts=2023-02-08T15:19:50.61548846Z caller=memberlist_client.go:436 msg="Using memberlist cluster label and node name" cluster_label= node=mimir-alertmanager-0-c079cc58
level=info ts=2023-02-08T15:19:50.61647388Z caller=module_service.go:82 msg=initialising module=activity-tracker
level=info ts=2023-02-08T15:19:50.616532433Z caller=module_service.go:82 msg=initialising module=sanity-check
level=info ts=2023-02-08T15:19:50.616586587Z caller=sanity_check.go:32 msg="Checking directories read/write access"
level=info ts=2023-02-08T15:19:50.616724143Z caller=sanity_check.go:37 msg="Directories read/write access successfully checked"
level=info ts=2023-02-08T15:19:50.61673848Z caller=sanity_check.go:39 msg="Checking object storage config"
level=info ts=2023-02-08T15:19:50.61684351Z caller=sanity_check.go:44 msg="Object storage config successfully checked"
level=info ts=2023-02-08T15:19:50.61686662Z caller=module_service.go:82 msg=initialising module=server
level=info ts=2023-02-08T15:19:50.616891431Z caller=module_service.go:82 msg=initialising module=memberlist-kv
level=info ts=2023-02-08T15:19:50.61688744Z caller=module_service.go:82 msg=initialising module=runtime-config
level=error ts=2023-02-08T15:19:50.61693635Z caller=mimir.go:791 msg="module failed" module=memberlist-kv err="invalid service state: Stopping, expected: Running"
level=error ts=2023-02-08T15:19:50.616960371Z caller=mimir.go:791 msg="module failed" module=alertmanager err="failed to start alertmanager, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running"
level=error ts=2023-02-08T15:19:50.616972382Z caller=mimir.go:791 msg="module failed" module=runtime-config err="context canceled"
level=info ts=2023-02-08T15:19:50.617039827Z caller=server_service.go:55 msg="server stopped"
level=info ts=2023-02-08T15:19:50.61706879Z caller=module_service.go:114 msg="module stopped" module=server
level=info ts=2023-02-08T15:19:50.617085781Z caller=module_service.go:114 msg="module stopped" module=sanity-check
level=info ts=2023-02-08T15:19:50.618582896Z caller=module_service.go:114 msg="module stopped" module=activity-tracker
level=info ts=2023-02-08T15:19:50.618623923Z caller=mimir.go:780 msg="Application stopped"
level=error ts=2023-02-08T15:19:50.618665335Z caller=log.go:60 msg="error running application" err="failed services\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:834\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:212\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"

Distributor

level=error ts=2023-02-08T15:33:35.126316452Z caller=mimir.go:791 msg="module failed" module=memberlist-kv err="invalid service state: Stopping, expected: Running"
level=error ts=2023-02-08T15:33:35.126352567Z caller=mimir.go:791 msg="module failed" module=distributor-service err="failed to start distributor-service, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running"
level=error ts=2023-02-08T15:33:35.126361704Z caller=mimir.go:791 msg="module failed" module=ring err="failed to start ring, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running"
level=error ts=2023-02-08T15:33:35.126370535Z caller=mimir.go:791 msg="module failed" module=runtime-config err="context canceled"
level=info ts=2023-02-08T15:33:35.126444805Z caller=server_service.go:55 msg="server stopped"
level=info ts=2023-02-08T15:33:35.126472045Z caller=module_service.go:114 msg="module stopped" module=server
level=info ts=2023-02-08T15:33:35.126500443Z caller=module_service.go:114 msg="module stopped" module=sanity-check
level=info ts=2023-02-08T15:33:35.127917212Z caller=module_service.go:114 msg="module stopped" module=activity-tracker
level=info ts=2023-02-08T15:33:35.12793598Z caller=mimir.go:780 msg="Application stopped"
level=error ts=2023-02-08T15:33:35.127977894Z caller=log.go:60 msg="error running application" err="failed services\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:834\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:212\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"

They are all the same errors. This is mostly a vanilla deployment of OpenShift, with the default OpenShift SDN as the CNI.

$ oc get pods
NAME                                        READY   STATUS             RESTARTS           AGE
mimir-alertmanager-0                        0/1     CrashLoopBackOff   3588 (36s ago)     12d
mimir-compactor-0                           0/1     CrashLoopBackOff   3589 (3m58s ago)   12d
mimir-distributor-8596c9d64f-hrrmv          0/1     CrashLoopBackOff   3589 (5m1s ago)    12d
mimir-distributor-8596c9d64f-l75m5          0/1     CrashLoopBackOff   3588 (3m15s ago)   12d
mimir-ingester-zone-a-0                     0/1     CrashLoopBackOff   3588 (4m28s ago)   12d
mimir-ingester-zone-b-0                     0/1     CrashLoopBackOff   3589 (95s ago)     12d
mimir-ingester-zone-c-0                     0/1     CrashLoopBackOff   3587 (111s ago)    12d
mimir-overrides-exporter-789db77b46-g76pn   1/1     Running            0                  12d
mimir-querier-6d95f8d66-wtf9f               0/1     CrashLoopBackOff   3588 (4m10s ago)   12d
mimir-query-frontend-7565579cbc-ttmt4       0/1     CrashLoopBackOff   3587 (51s ago)     12d
mimir-rollout-operator-7f987c5f5b-5clwr     1/1     Running            0                  12d
mimir-ruler-76f99f864c-qld5b                0/1     CrashLoopBackOff   3588 (3m45s ago)   12d
mimir-store-gateway-zone-a-0                0/1     Error              3588 (5m10s ago)   12d
mimir-store-gateway-zone-b-0                0/1     CrashLoopBackOff   3588 (2m41s ago)   12d
mimir-store-gateway-zone-c-0                0/1     CrashLoopBackOff   3589 (52s ago)     12d

$ oc debug mimir-alertmanager-0
Starting pod/mimir-alertmanager-0-debug ...
Pod IP: 254.23.5.71
If you don't see a command prompt, try pressing enter.
~ $ ifconfig
eth0      Link encap:Ethernet  HWaddr 0A:58:FE:17:05:47
          inet addr:254.23.5.71  Bcast:254.23.5.255  Mask:255.255.254.0
          inet6 addr: fe80::2ca3:4ff:fe8d:81df/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:816 (816.0 B)  TX bytes:516 (516.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

$ oc get pods mimir-alertmanager-0 -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    checksum/alertmanager-fallback-config: 3044cc5561ab9f55e548777f0999c79b723c3566dfd3c73327de679c0589a4df
    checksum/config: 87bad3a617e6500823a5acc604382b8a796b2fa5db1fe07da5772525bff35a51
    k8s.v1.cni.cncf.io/network-status: |-
      [{
          "name": "openshift-sdn",
          "interface": "eth0",
          "ips": [
              "254.23.5.122"
          ],
          "default": true,
          "dns": {}
      }]
    k8s.v1.cni.cncf.io/networks-status: |-
      [{
          "name": "openshift-sdn",
          "interface": "eth0",
          "ips": [
              "254.23.5.122"
          ],
          "default": true,
          "dns": {}
      }]
    openshift.io/scc: mimir

[pracucci]

The IP auto-detection isn't able to detect any "valid" IP address for Mimir because 127.0.0.1 is the loopback and 254.23.5.71 (eth0) is not a private IP (at least it doesn't belong to a private IP class).

If you want to move forward using eth0 the only option we have today is to explicitly configure the IP address to Mimir through the various options. Unfortunately it's tedious because it should be specified to multiple config options:

• -overrides-exporter.ring.instance-addr
• -distributor.ring.instance-addr
• -ingester.ring.instance-addr
• -query-frontend.instance-addr
• -query-scheduler.ring.instance-addr
• -ruler.ring.instance-addr
• -alertmanager.sharding-ring.instance-addr
• -memberlist.advertise-addr
• -compactor.ring.instance-addr
• -store-gateway.sharding-ring.instance-addr

How to get the actual IP address of the pod? I have no experience with OpenShift, but in Kubernetes there's a variable available, so I hope there's something similar on OpenShift too.

[ctml91]

How to get the actual IP address of the pod?

Ah, nice catch! 254.X is under a class E which is reserved and cannot be used on the public internet. We use it for the internal pod network because it's non-routable both internal and external, so that is the real IP address of the pod. Normal private ranges for pod network is not an option in our org as it conflicts with internal IP addressing assignment and also because we don't need routable IP's (outside the cluster CNI) assigned to the internal pod network which can require hundreds of thousands of IPs, hence why the range we select is important.

We've seen this probably 2-3 times over 4 years across hundreds of Kubernetes apps/operators being used where certain products are hardcoded to look for specific class of subnets. In those cases there were minor changes implemented to not restrict to well known private ranges. While we cannot change the networking configuration now, we will be using privately used public IP (PUPI) ranges for new clusters which still may not be considered private by mimirs standard.

If there is no option to get mimir to run on non-standard private IP ranges that's too bad for us I guess, but hopefully with some minor changes to that auto-detection routine this could still work as we would like to use mimir over thanos.