Access to s3 bucket

[rrusso1982]

Describe the bug

I am attempting to deploy grafana mimir on an amazon EKS cluster with s3 as it's backend storage. It have the necessary buckets created, EKS cluster running, IRSA enabled on the cluster, IAM roles deployed, and service account setup with the role annotation attached.

When I start up mimir using helm I am getting this error from any pod attempting to use the s3 storage

level=warn ts=2022-08-25T21:48:07.218854521Z caller=sanity_check.go:116 msg="Unable to successfully connect to configured object storage (will retry)" err="2 errors: blocks storage: unable to successfully send a request to object storage: Access Denied; ruler storage: unable to successfully send a request to object storage: Access Denied"

Hoping into the pod shows that the pod is showing the correct aws role but the software is still unable to connect even though the permissions to the bucket are currently set to s3:*.

Any assistance with this would be appreciated

Expected behavior

I expect the pods to be able to connect to their object storage correctly.

Environment

• Infrastructure: AWS EKS and s3 object storage
• Deployment tool: helm

Additional Context

Here is a snippet of my configuration

mimir:
  structuredConfig:
    blocks_storage:
      backend: s3
      s3:
        bucket_name: mimir-us-east-2
        endpoint: s3.us-east-2.amazonaws.com
        region: us-east-2
        sse:
          type: "SSE-S3"

[pracucci]

A quick double check, please: if you install the aws CLI client in the same container where Mimir is running, can the aws tool access the S3 bucket?

[...] ruler storage: unable to successfully send a request to object storage: Access Denied" [...]

The error log both mentions the blocks_storage (which you configured) and ruler_storage. Have you also configured the ruler_storage? (not the root cause of your issue because it fails the check on blocks_storage too, but ruler_storage should be configured too if you want to run the ruler)

[pracucci]

Also, would this solution work to you too? (even if in that case you shouldn't get an "Access Denied" error)
#2613 (comment)

[pracucci]

I don't have an EKS at hand with which experiment IAM-based privileges. Could you tell me which of the following environment variables are set in the Mimir container (you can run printenv to check them)? I'm just looking for which are set, do not share the value (which is sensitive):

• AWS_WEB_IDENTITY_TOKEN_FILE
• AWS_REGION
• AWS_ROLE_ARN
• AWS_ROLE_SESSION_NAME
• AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
• AWS_CONTAINER_CREDENTIALS_FULL_URI

[rrusso1982]

I figured it all out late last night. There was an error in the IRSA config for the service account as it relates to the namespace mimir was running in. I fixed it and it started working right away.

[pracucci]

There was an error in the IRSA config for the service account as it relates to the namespace mimir was running in.

Could you share any non-sensitive detail about the issue, in case other people will have a similar one, please?

[rrusso1982]

Sure thing.

So when you create an IAM role that is going to be connected to a service account within an EKS cluster inside the configuration for the IAM role you have to specify the namespace and the name of the service account that will be authorized to assume that role. That config block looks like this:

system:serviceaccount:<namespace>:<serviceaccount>

The error on my side was that the namespace being created by the mimir deployment was not aligned with the namespace in the IAM role that had been created to support the service account. I aligned those values and things worked right away.

[ManojSuyal]

getting below error while configuring azure blob for mimir.

ts=2023-06-13T07:31:22.543051476Z caller=main.go:213 level=info msg="Starting application" version="(version=2.8.0, branch=HEAD, revision=f917e08)"
ts=2023-06-13T07:31:22.545614638Z caller=log.go:65 level=error msg="error running application" err="no s3 endpoint in config file\ngithub.com/thanos-io/objstore/providers/s3.validate\n\t/__w/mimir/mimir/vendor/github.com/thanos-io/objstore/providers/s3/s3.go:349\ngithub.com/thanos-io/objstore/providers/s3.NewBucketWithConfig\n\t/__w/mimir/mimir/vendor/github.com/thanos-io/objstore/providers/s3/s3.go:219\ngithub.com/grafana/mimir/pkg/storage/bucket/s3.NewBucketClient\n\t/__w/mimir/mimir/pkg/storage/bucket/s3/bucket_client.go:27\ngithub.com/grafana/mimir/pkg/storage/bucket.NewClient\n\t/__w/mimir/mimir/pkg/storage/bucket/client.go:163\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).initUsageStats\n\t/__w/mimir/mimir/pkg/mimir/modules.go:871\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:136\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:761\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:215\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598\ncreate usage-stats bucket client\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).initUsageStats\n\t/__w/mimir/mimir/pkg/mimir/modules.go:873\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:136\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:761\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:215\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598\nerror initialising module: usage-stats\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:138\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:761\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:215\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598"

[ManojSuyal]

this is what my values.yaml for storage looks like.

common:
storage:
backend: azure
azure:
account_key: "${SWIFT_ACCOUNT_KEY}"
account_name:
endpoint_suffix: "blob.core.windows.net"

blocks_storage:
azure:
container_name: mimir-blocks

alertmanager_storage:
azure:
container_name: mimir-alertmanager

ruler_storage:
azure:
container_name: mimir-ruler

[premkottu]

I had a similar error when I use instance IAM role and it was working fine with access keys.

This was due to the limit on number of hops for ec2 instance using IMDSv2. By default, HttpPutResponseHopLimit is 1 and I had to increase it to 3 to make it work.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html#imds-considerations