Velero policy does not contain necessary S3 PutObjectTagging permissions

[chrisRedwine]

Description

The velero-plugin-for-aws supports tagging the objects in S3, as advertised here and implemented here. This requires the s3:PutObjectTagging permission for the bucket, which the current velero policy here does not contain.

• ✋ I have searched the open/closed issues and my issue is not listed.

Versions

• Module version [Required]: 5.44.0

• Terraform version: OpenTofu 1.7.3

• Provider version(s): AWS 5.66.0

Reproduction Code [Required]

Probably a bit overkill, but: https://github.com/chrisRedwine/velero-mre

The following steps show not only how to reproduce the behavior, but also how the fix in this PR resolves the issue:

• Clone the repo above, which contains the minimal reproducible example for the issue.
• Run tofu init and tofu apply to create and configure the required resources (VPC, EKS, EBS CSI driver, Snapshot Controller, Velero, Pod w/ PVC and data, etc.).
• Wait until everything is set up and the ebs-pvc-pod has run and saved data to the volume.
• Run velero backup create ebs-test-broken --include-namespaces=default --snapshot-move-data.
• See the error in the logs described below, and note that velero backup describe ebs-test-broken shows the backup failed.
• Switch the use_fixed_velero_policy variable in terraform.tfvars to true in the chrisRedwine/velero-mre repo.
• Run tofu apply
• Wait for the 3 velero pods to restart so that they use the fixed IRSA.
• Run velero backup create ebs-test-fixed --include-namespaces=default --snapshot-move-data.
• Notice no errors in the logs, and that velero backup describe ebs-test-fixed shows the backup succeeded.
• (Make sure to clean up with tofu destroy afterwards)

Expected behavior

The command should succeed without any errors.

Actual behavior

The command fails with the following permissions issue (taken from the velero pod logs):

level=error msg="Error uploading log file" backup=ebs-test bucket=velero-backups-velero-mre error="rpc error: code = Unknown desc = error putting object backups/backups/ebs-test/ebs-test-logs.gz: operation error S3: PutObject, https response error StatusCode: 403, RequestID: <redacted>, HostID: <redacted>, api error AccessDenied: User: arn:aws:sts::<redacted>:assumed-role/velero-irsa-velero-mre/<redacted> is not authorized to perform: s3:PutObjectTagging on resource: \"arn:aws:s3:::velero-backups-velero-mre/backups/backups/ebs-test/ebs-test-logs.gz\" because no identity-based policy allows the s3:PutObjectTagging action" error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:279" error.function="main.(*ObjectStore).PutObject" logSource="pkg/persistence/object_store.go:256" prefix=backups

level=error msg="backup failed" backuprequest=velero/ebs-test controller=backup error="rpc error: code = Unknown desc = error putting object backups/backups/ebs-test/velero-backup.json: operation error S3: PutObject, https response error StatusCode: 403, RequestID: <redacted>, HostID: <redacted>, api error AccessDenied: User: arn:aws:sts::<redacted>:assumed-role/velero-irsa-velero-mre/<redacted> is not authorized to perform: s3:PutObjectTagging on resource: \"arn:aws:s3:::velero-backups-velero-mre/backups/backups/ebs-test/velero-backup.json\" because no identity-based policy allows the s3:PutObjectTagging action" logSource="pkg/controller/backup_controller.go:287"

Notice that the IRSA role lacks the s3:PutObjectTagging permission, which it needs because of this code.

Additional context

• I submitted a PR that will fix this issue.
• I also have an issue/PR to fix the docs in the velero-plugin-for-aws repo.

[bryantbiggs]

if their policy contains this, we'll support it - but not before then

[chrisRedwine]

Looks like they've approved the PR to add it to their docs - I'll post back here once it's merged.

Thanks, @bryantbiggs

[chrisRedwine]

@bryantbiggs FYI, the upstream PR to include this in the docs has been merged.

[antonbabenko]

This issue has been resolved in version 5.44.2 🎉

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.