ACK S3 bucket - MalformedXML

[itaiatu]

Describe the bug
We have upgraded the ACK S3 controller from v1.0.18 to v1.0.23.

When trying to create a simple s3 bucket, we get the following error from AWS:

"errorCode": "MalformedXML",
"errorMessage": "The XML you provided was not well-formed or did not validate against our published schema",

Steps to reproduce

Create and apply simple s3 bucket ack object

apiVersion: s3.services.k8s.aws/v1alpha1
kind: Bucket
metadata:
  name: test-bucket-v1.0.23
  namespace: ci-clusters
spec:
  name: test-bucket-v1.0.23

The object looks like this after it's being reconciled by the ACK controller:

kubectl -n ci-clusters get buckets test-bucket-v1.0.23 -o yaml

apiVersion: s3.services.k8s.aws/v1alpha1
kind: Bucket
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"s3.services.k8s.aws/v1alpha1","kind":"Bucket","metadata":{"annotations":{},"name":"test-bucket-v1.0.23","namespace":"ci-clusters"},"spec":{"name":"test-bucket-v1.0.23"}}
  creationTimestamp: "2025-02-19T12:38:38Z"
  finalizers:
  - finalizers.s3.services.k8s.aws/Bucket
  generation: 1
  name: test-bucket-v1.0.23
  namespace: ci-clusters
  resourceVersion: "622738"
  uid: 3fd706ad-8861-43bd-b0f8-a0edf13a7cd4
spec:
  name: test-bucket-v1.0.23
status:
  ackResourceMetadata:
    ownerAccountID: "381491899637"
    region: us-east-1
  conditions:
  - message: 'api error MalformedXML: The XML you provided was not well-formed or
      did not validate against our published schema'
    status: "True"
    type: ACK.Recoverable
  - lastTransitionTime: "2025-02-19T12:38:52Z"
    message: Unable to determine if desired resource state matches latest observed
      state
    reason: 'operation error S3: CreateBucket, https response error StatusCode: 400,
      RequestID: T9GYJBB2GQH1JP1Y, HostID: EndEVpYGVQyqDLIm/a86iYgoF7wZZIL+9j06remtm9PdquEoIUR9p5UdcAamnpbc4jiM9ikBsSE=,
      api error MalformedXML: The XML you provided was not well-formed or did not
      validate against our published schema'
    status: Unknown
    type: ACK.ResourceSynced

When looking in CloudTrail for the test-bucket-v1.0.23 resource, we have the CreateBucket event:

...
    "userAgent": "[aws-controllers-k8s/s3.services.k8s.aws-1.0.23 (GitCommit/3dd19c5c81c5d27ae5be965852b99b500cf6d64d; BuildDate/2025-02-18T01:57; CRDKind/Bucket; CRDVersion/v1alpha1) aws-sdk-go-v2/1.34.0 ua/2.1 os/linux lang/go#1.23.6 md/GOOS#linux md/GOARCH#amd64 api/s3#1.74.1 m/E]",
    "errorCode": "MalformedXML",
    "errorMessage": "The XML you provided was not well-formed or did not validate against our published schema",
    "requestParameters": {
        "CreateBucketConfiguration": {
            "xmlns": "http://s3.amazonaws.com/doc/2006-03-01/"
        },
        "bucketName": "test-bucket-v1.0.23",
        "Host": "s3.us-east-1.amazonaws.com"
    },
...

Then, we used the old v1.0.18 ACK S3 controller version and it correctly reconciled the bucket and created it in the cloud.

Another thing is that, in the CloudTrail, the event for the same bucket (but with name test-bucket-v1.0.18), when created with the v1.0.18 ACK S3 controller version, looks like this:

...
    "userAgent": "[aws-controllers-k8s/s3.services.k8s.aws-1.0.18 (GitCommit/7d0c69df1645cc146bfc78863cf0ce0db6796b6b; BuildDate/2024-12-03T18:23; CRDKind/Bucket; CRDVersion/v1alpha1) aws-sdk-go/1.49.0 (go1.23.3; linux; amd64)]",
    "requestParameters": {
        "bucketName": "test-bucket-v1.0.18",
        "Host": "s3.amazonaws.com"
    },
...

We can see that the CreateBucketConfiguration is missing.

The problem could be from the migration to aws-sdk-go-v2.

Expected outcome
Create the s3 bucket in the cloud.

Environment

• Kubernetes version: 1.29
• Using EKS (yes/no), if so version?: 1.29
• AWS service targeted (S3, RDS, etc.): S3

[eqe-aws]

We will take a look and see if it's reproducible on our end

[eqe-aws]

We have found the issue and will post back with next steps.

[rushmash91]

Hi @itaiatu, the MalformedXML issue for us-east-1 is fixed in the release v1.0.26 published today.