Assuming IAM role from within a EKS Pod Identity-enabled container does not work using named profile

[rkubik-hostersi]

Describe the bug

When working on a pod in EKS with Pod Identity assigned, it is not possible to assume another role using ~/.aws/config and profiles.

When specifying role_arn in ~/.aws/config, it is required to provide source_profile or credential_source. Since we are in the pod, source_profile is not an option. Unfortunately credential_source is pretty limited:

• Environment value does not work as there is no env variables
• Ec2InstanceMetadata points to the IAM role attached to the EC2, Pod Identity is not being used
• EcsContainer is for ECS

Expected Behavior

It should be possible to instruct aws-cli to use EKS Pod Identity as a credential_source.

Current Behavior

It is not possible to utilize aws-cli with Assume Role mechanism using named profiles within ~/.aws/config when working on a EKS Pod Identity-enabled pods.

Reproduction Steps

1. Create EKS with Pod Identity agent
2. Assign sts:assumeRole permission to the pod
3. Prepare IAM role to be assumed
4. Create the pod with the Pod Identity assigned, prepare ~/.aws/config
5. Try to assume a different IAM role using aws --profile

Possible Solution

No response

Additional Information/Context

No response

CLI version used

2.15.57

Environment details (OS name and version, etc.)

aws-cli/2.15.57 Python/3.12.6 Linux/6.8.0-41-generic source/x86_64.alpine.3

[drunkensway]

experiencing this as well using hashicorp/terraform:1.5.6.

after installing the aws cli and running aws configure set role_arn <role-arn> getting:

Error relocating /usr/lib/python3.11/lib-dynload/pyexpat.cpython-311-x86_64-linux-musl.so: XML_SetReparseDeferralEnabled: symbol not found

[tim-finnigan]

experiencing this as well using hashicorp/terraform:1.5.6.

after installing the aws cli and running aws configure set role_arn <role-arn> getting:

Error relocating /usr/lib/python3.11/lib-dynload/pyexpat.cpython-311-x86_64-linux-musl.so: XML_SetReparseDeferralEnabled: symbol not found

Same error as #8913, replied there:

Looks like this is the same as hashicorp/terraform#35715, where a member of Terraform replied:

The Dockerfile for the build wasn't changed during that time, so any differences would be solely from the upstream image. Your above example works correctly if the package is updated, and I also confirmed that newer images have already updated the problematic packages.
Closing since there's nothing the Terraform CLI can do to fix the old docker image.

Can you confirm that this is fixed in newer images?

[tim-finnigan]

But the original issue here looks related to #3875 and aws/aws-sdk#350.

[jcary741]

I am encountering this as well, which is breaking our gitlab CI that uses apk add aws-cli.

Here is the relevant section from a working run from yesterday:

$ apk add --no-cache aws-cli
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz
(1/59) Installing libbz2 (1.0.8-r5)
(2/59) Installing libffi (3.4.4-r2)
(3/59) Installing gdbm (1.23-r1)
(4/59) Installing xz-libs (5.4.3-r0)
(5/59) Installing libgcc (12.2.1_git20220924-r10)
(6/59) Installing libstdc++ (12.2.1_git20220924-r10)
(7/59) Installing mpdecimal (2.5.1-r2)
(8/59) Installing libpanelw (6.4_p20230506-r0)
(9/59) Installing readline (8.2.1-r1)
(10/59) Installing sqlite-libs (3.41.2-r3)
(11/59) Installing python3 (3.11.8-r1)
(12/59) Installing python3-pycache-pyc0 (3.11.8-r1)
(13/59) Installing pyc (0.1-r0)
(14/59) Installing py3-certifi (2024.2.2-r0)
(15/59) Installing py3-certifi-pyc (2024.2.2-r0)
(16/59) Installing py3-cparser (2.21-r2)
(17/59) Installing py3-cparser-pyc (2.21-r2)
(18/59) Installing py3-cffi (1.15.1-r3)
(19/59) Installing py3-cffi-pyc (1.15.1-r3)
(20/59) Installing py3-cryptography (41.0.3-r0)
(21/59) Installing py3-cryptography-pyc (41.0.3-r0)
(22/59) Installing py3-six (1.16.0-r6)
(23/59) Installing py3-six-pyc (1.16.0-r6)
(24/59) Installing py3-dateutil (2.8.2-r3)
(25/59) Installing py3-dateutil-pyc (2.8.2-r3)
(26/59) Installing py3-distro (1.8.0-r2)
(27/59) Installing py3-distro-pyc (1.8.0-r2)
(28/59) Installing py3-colorama (0.4.6-r3)
(29/59) Installing py3-colorama-pyc (0.4.6-r3)
(30/59) Installing py3-docutils (0.19-r4)
(31/59) Installing py3-docutils-pyc (0.19-r4)
(32/59) Installing py3-jmespath (1.0.1-r1)
(33/59) Installing py3-jmespath-pyc (1.0.1-r1)
(34/59) Installing py3-urllib3 (1.26.18-r0)
(35/59) Installing py3-urllib3-pyc (1.26.18-r0)
(36/59) Installing py3-wcwidth (0.2.6-r2)
(37/59) Installing py3-wcwidth-pyc (0.2.6-r2)
(38/59) Installing py3-prompt_toolkit (3.0.38-r1)
(39/59) Installing py3-prompt_toolkit-pyc (3.0.38-r1)
(40/59) Installing py3-ruamel.yaml.clib (0.2.7-r1)
(41/59) Installing py3-ruamel.yaml (0.17.28-r0)
(42/59) Installing py3-ruamel.yaml-pyc (0.17.28-r0)
(43/59) Installing aws-cli-pyc (2.15.14-r0)
(44/59) Installing py3-awscrt-pyc (0.20.2-r0)
(45/59) Installing python3-pyc (3.11.8-r1)
(46/59) Installing aws-c-common (0.9.12-r0)
(47/59) Installing aws-c-cal (0.6.9-r0)
(48/59) Installing aws-c-compression (0.2.17-r0)
(49/59) Installing s2n-tls (1.3.47-r0)
(50/59) Installing aws-c-io (0.14.2-r0)
(51/59) Installing aws-c-http (0.8.0-r0)
(52/59) Installing aws-c-sdkutils (0.1.14-r0)
(53/59) Installing aws-c-auth (0.7.14-r0)
(54/59) Installing aws-checksums (0.1.17-r0)
(55/59) Installing aws-c-event-stream (0.4.1-r0)
(56/59) Installing aws-c-mqtt (0.10.1-r0)
(57/59) Installing aws-c-s3 (0.4.10-r0)
(58/59) Installing py3-awscrt (0.20.2-r0)
(59/59) Installing aws-cli (2.15.14-r0)
Executing busybox-1.36.1-r2.trigger
OK: 200 MiB in 100 packages

The CI job then goes on to use the AWS CLI successfully.

And here is a broken one today:

$ apk add --no-cache aws-cli
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz
(1/59) Installing libbz2 (1.0.8-r5)
(2/59) Installing libffi (3.4.4-r2)
(3/59) Installing gdbm (1.23-r1)
(4/59) Installing xz-libs (5.4.3-r0)
(5/59) Installing libgcc (12.2.1_git20220924-r10)
(6/59) Installing libstdc++ (12.2.1_git20220924-r10)
(7/59) Installing mpdecimal (2.5.1-r2)
(8/59) Installing libpanelw (6.4_p20230506-r0)
(9/59) Installing readline (8.2.1-r1)
(10/59) Installing sqlite-libs (3.41.2-r3)
(11/59) Installing python3 (3.11.10-r0)
(12/59) Installing python3-pycache-pyc0 (3.11.10-r0)
(13/59) Installing pyc (0.1-r0)
(14/59) Installing py3-certifi (2024.2.2-r0)
(15/59) Installing py3-certifi-pyc (2024.2.2-r0)
(16/59) Installing py3-cparser (2.21-r2)
(17/59) Installing py3-cparser-pyc (2.21-r2)
(18/59) Installing py3-cffi (1.15.1-r3)
(19/59) Installing py3-cffi-pyc (1.15.1-r3)
(20/59) Installing py3-cryptography (41.0.3-r0)
(21/59) Installing py3-cryptography-pyc (41.0.3-r0)
(22/59) Installing py3-six (1.16.0-r6)
(23/59) Installing py3-six-pyc (1.16.0-r6)
(24/59) Installing py3-dateutil (2.8.2-r3)
(25/59) Installing py3-dateutil-pyc (2.8.2-r3)
(26/59) Installing py3-distro (1.8.0-r2)
(27/59) Installing py3-distro-pyc (1.8.0-r2)
(28/59) Installing py3-colorama (0.4.6-r3)
(29/59) Installing py3-colorama-pyc (0.4.6-r3)
(30/59) Installing py3-docutils (0.19-r4)
(31/59) Installing py3-docutils-pyc (0.19-r4)
(32/59) Installing py3-jmespath (1.0.1-r1)
(33/59) Installing py3-jmespath-pyc (1.0.1-r1)
(34/59) Installing py3-urllib3 (1.26.18-r0)
(35/59) Installing py3-urllib3-pyc (1.26.18-r0)
(36/59) Installing py3-wcwidth (0.2.6-r2)
(37/59) Installing py3-wcwidth-pyc (0.2.6-r2)
(38/59) Installing py3-prompt_toolkit (3.0.38-r1)
(39/59) Installing py3-prompt_toolkit-pyc (3.0.38-r1)
(40/59) Installing py3-ruamel.yaml.clib (0.2.7-r1)
(41/59) Installing py3-ruamel.yaml (0.17.28-r0)
(42/59) Installing py3-ruamel.yaml-pyc (0.17.28-r0)
(43/59) Installing aws-cli-pyc (2.15.14-r0)
(44/59) Installing py3-awscrt-pyc (0.20.2-r0)
(45/59) Installing python3-pyc (3.11.10-r0)
(46/59) Installing aws-c-common (0.9.12-r0)
(47/59) Installing aws-c-cal (0.6.9-r0)
(48/59) Installing aws-c-compression (0.2.17-r0)
(49/59) Installing s2n-tls (1.3.47-r0)
(50/59) Installing aws-c-io (0.14.2-r0)
(51/59) Installing aws-c-http (0.8.0-r0)
(52/59) Installing aws-c-sdkutils (0.1.14-r0)
(53/59) Installing aws-c-auth (0.7.14-r0)
(54/59) Installing aws-checksums (0.1.17-r0)
(55/59) Installing aws-c-event-stream (0.4.1-r0)
(56/59) Installing aws-c-mqtt (0.10.1-r0)
(57/59) Installing aws-c-s3 (0.4.10-r0)
(58/59) Installing py3-awscrt (0.20.2-r0)
(59/59) Installing aws-cli (2.15.14-r0)
Executing busybox-1.36.1-r2.trigger
OK: 200 MiB in 100 packages

Which then fails with Error relocating /usr/lib/python3.11/lib-dynload/pyexpat.cpython-311-x86_64-linux-musl.so: XML_SetReparseDeferralEnabled: symbol not found

The difference I'm seeing is python 3.11.10-r0 is used now, instead of 3.11.8, so maybe this is a new issue there?

[joerawr]

We are seeing this across our CICD. All versions of 1.5.x are impacted. So far in our brief testing 1.6 through 1.9 are not impacted. We're scrambling to test newer versions and update our shared templates.

Likely that Python 3 version from Alpine is the issue. The timestamp is 9/11:

https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/
python3-3.11.10-r0.apk 11-Sep-2024 10:14 9M

[joerawr]

Here is a similar issue with Alpine 3.18 via Terraform 1.5.7:
https://gitlab.alpinelinux.org/alpine/aports/-/issues/16441

[tim-finnigan]

For those using Terraform have you referred to: hashicorp/terraform#35715?

[rkubik-hostersi]

Guys, this is not about terraform or any other library, or even python versions. This is about the missing configuration parameter for credential_source when running aws in EKS Pod Identity enabled container. AWS CLI version also does not matter as there is no "legit" parameter to be used in EKS on PI containers and credential_source.

The scenario has been described in the first post. We need to be able to use aws --profile from within a pod to assume some external role with Pod Identity. This is not possible for now officially. :)

[jcary741]

My bad @rkubik-hostersi, the timing of when you submitted this issue and the environment you described, then followed by what drunkensway said made me think we were encountering different versions of the same problem. I see now that your submission is actually different. Just to update anyone who happens upon this issue who makes the same mistake, the issue we were encountering appears to have been resolved in Python build 3.11.10-r1.