Is it possible to use AWS ECR as repository for K3S running on AWS EC2

[urupaud]

We have a k3s cluster which is running on AWS ec2, we want to deploy containers into this cluster using images in our AWS ECR, is this possible ?

[HuJake]

We also want to ask this question.

[brandond]

It appears that containerd does not support external credential helpers like docker does. See: https://github.com/containerd/cri/issues/1131. This would prevent you from being able to authenticate to ECR when using the default containerd backend.

[HuJake]

@brandond thanks your reply my question.

We have changed it to docker but after setting the following flag

k3s server --docker --kubelet-arg = "cloud-provider = external" --kubelet-arg = "provider-id = aws: /// $ (curl -s http://169.254.169.254/latest/meta-data/placement/ availability-zone) / $ (curl -s http://169.254.169.254/latest/meta-data/instance-id) "

Seems unable to solve..
It seems there is no information to refer to

[bigbohne]

not nice but you can always generate the /etc/rancher/k3s/registries.yaml using crontab ...

[brandond]

@HuJake did you actually install the out-of-tree aws cloud provider, set up the aws credential helper, and configure docker registry auth to use the helper?

@bigbohne will it actually re-read it, or is it just loaded on startup? I couldn't tell.

[HuJake]

@brandond

I simply set the relevant flag,

k3s server 
--docker
--kubelet-arg = "cloud-provider = external" \
--kubelet-arg = "provider-id = aws: /// $ (curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone)/$(curl -s http: / /169.254.169.254/latest/meta-data/instance-id) "

I'm want to use [aws cloud provider] but not sure how to apply related yaml?
Are there references to successful cases?
thanks

[brandond]

@HuJake All that does is set the ID. If you want a full cloud provider you can find deployment documentation at https://github.com/kubernetes/cloud-provider-aws. For authenticating Docker to ECR for image pulls you would want https://github.com/awslabs/amazon-ecr-credential-helper.

[bigbohne]

With docker as container engine this is easily possible using the ecr-credential-helper. But what about containerd? Or is it a design decision to not include plugable authenticators?

[brandond]

The containerd folks seem to have philosophical issues that preclude them from adding support for pluggable authentication.

[bigbohne]

Than it is what it is ...

[brandond]

@bigbohne see https://github.com/containerd/cri/issues/1131#issuecomment-631633816

[canadiannomad]

will it actually re-read it, or is it just loaded on startup? I couldn't tell.

I'm wondering the same... I can easily write a cronjob to grab new creds, but restarting k3s every time....?

[studiocredo]

We are running k3s with docker as container engine here. I installed and configured the amazon-ecr-credential-helper.
I can successfully docker pull images from our private ECR repo as a regular user, but k3s itself keeps complaining with ErrImagePull.

A kubectl describe pod shows: Failed to pull image "***": rpc error: code = Unknown desc = Error response from daemon: Get https://***: no basic auth credentials

So my understanding of this is that the kubelet service (running inside a docker container) cannot pull the image because it does not know about the credential helper. Any pointers on how to overcome this?

[brandond]

Probably related: awslabs/amazon-ecr-credential-helper#210

[studiocredo]

I was able to solve/work around this by deploying registry-creds in my cluster (this is basically what the registry-creds addon in Minikube is doing)

[KennyReeldata]

The "registry creds" project didn't work for me, I ran into upmc-enterprises/registry-creds#97. However, I was able to get auth working with K3s by slightly editing this script for my needs: https://stackoverflow.com/a/55658863
Requires the aws cli to be installed on your servers

[johnlane]

Just in case it's useful.... I'm running k3s locally with the docker backend and I was able to pull from ecr using information from here and here.

[mandrean]

@urupaud I got it working by:

1. $ git clone github.com/upmc-enterprises/registry-cred
2. Create a new RBAC manifest, k8s/rbac.yaml:

kind: ServiceAccount
metadata:
  name: registry-creds
  namespace: kube-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: registry-creds
subjects:
- kind: ServiceAccount
  name: registry-creds
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: edit
  apiGroup: rbac.authorization.k8s.io

3. Edit k8s/deployment.yaml and secret.yaml to suit your needs. For me, I removed everything ACR/GCR/DPR and only kept the ECR stuff. I also added the reference to the new service account and added a missing env for AWS_SESSION_TOKEN:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: registry-creds
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      name: registry-creds
  template:
    metadata:
      labels:
        name: registry-creds
    spec:
      serviceAccountName: registry-creds
      containers:
      - image: upmcenterprises/registry-creds:1.10
        name: registry-creds
        imagePullPolicy: Always
        env:
          - name: AWS_ACCESS_KEY_ID
            valueFrom:
              secretKeyRef:
                name: registry-creds-ecr
                key: AWS_ACCESS_KEY_ID
          - name: AWS_SECRET_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: registry-creds-ecr
                key: AWS_SECRET_ACCESS_KEY
          - name: AWS_SESSION_TOKEN
            valueFrom:
              secretKeyRef:
                name: registry-creds-ecr
                key: AWS_SESSION_TOKEN
          - name: awsaccount
            valueFrom:
              secretKeyRef:
                name: registry-creds-ecr
                key: aws-account
          - name: awsregion
            valueFrom:
              secretKeyRef:
                name: registry-creds-ecr
                key: aws-region
          # - name: aws_assume_role
          #   valueFrom:
          #     secretKeyRef:
          #       name: registry-creds-ecr
          #       key: aws-assume-role

3. $ kubectl apply -f k8s/
4. Add awsecr-cred as an image pull secret to any deployment using images in ECR:

...
imagePullSecrets:
  - name: awsecr-cred
...

5. $ kubectl logs -l name=registry-creds -n kube-system --tail 1000 | grep "Updated secret awsecr-cred" should return something like:

time="2021-02-18T12:58:12Z" level=info msg="Updated secret awsecr-cred in namespace default"

[moonape1226]

@mandrean
Your solution above works like a charm on my on-premises K3S cluster. Thanks a lot.