Yes this is reproducible.

class IssueTriagePyStack(Stack):
  def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
    super().__init__(scope, construct_id, **kwargs)
  
    my_lambda = _lambda.Function(
          self, 'HelloHandler',
            runtime=_lambda.Runtime.PYTHON_3_8,
            code=_lambda.Code.from_asset('lambda'),
            handler='hello.handler',
    )    
  
    bucket = s3.Bucket.from_bucket_name(self, "Bucket", "my-existing-bucket-name")
    notification = s3_notify.LambdaDestination(my_lambda)

    notificationfilter1 = s3.NotificationKeyFilter(prefix="foo/", suffix="bar/")
    bucket.add_event_notification(s3.EventType.OBJECT_CREATED, notification, notificationfilter1)

    notificationfilter2 = s3.NotificationKeyFilter(prefix="fo1/",suffix="ba1/",)
    bucket.add_event_notification(s3.EventType.OBJECT_CREATED, notification, notificationfilter2)

On cdk deploy to update the stack:

issue-triage: creating CloudFormation changeset...
10:35:14 PM | UPDATE_FAILED        | Custom::S3BucketNotifications | BucketNotifications8F2E257D
Received response status [FAILED] from custom resource. Message returned: Error: An error occurred (InvalidArgument) when calling the PutBucketNotificati
onConfiguration operation: Configuration is ambiguously defined. Cannot have overlapping suffixes in two rules if the prefixes are overlapping for the sa
me event type.. See the details in CloudWatch Log Stream: 2024/09/04/[$LATEST]ff56425363ce4d8db0b594f6d560320a (RequestId: 9b4cb9c7-2141-4e7b-ba02-b4a880
70d580)

10:35:18 PM | UPDATE_FAILED        | Custom::S3BucketNotifications | BucketNotifications8F2E257D
Received response status [FAILED] from custom resource. Message returned: Error: An error occurred (InvalidArgument) when calling the PutBucketNotificati
onConfiguration operation: Configuration is ambiguously defined. Cannot have overlapping suffixes in two rules if the prefixes are overlapping for the sa
me event type.. See the details in CloudWatch Log Stream: 2024/09/04/[$LATEST]ff56425363ce4d8db0b594f6d560320a (RequestId: dc73fb2a-5abc-48d6-b0dd-da0625
2e55bc)

cdk synth

  BucketNotifications8F2E257D:
    Type: Custom::S3BucketNotifications
    Properties:
      ServiceToken:
        Fn::GetAtt:
          - BucketNotificationsHandler050a0587b7544547bf325f094a3db8347ECC3691
          - Arn
      BucketName: pahud-foo-bar
      NotificationConfiguration:
        LambdaFunctionConfigurations:
          - Events:
              - s3:ObjectCreated:*
            Filter:
              Key:
                FilterRules:
                  - Name: suffix
                    Value: bar/
                  - Name: prefix
                    Value: foo/
            LambdaFunctionArn:
              Fn::GetAtt:
                - HelloHandler2E4FBA4D
                - Arn
          - Events:
              - s3:ObjectCreated:*
            Filter:
              Key:
                FilterRules:
                  - Name: suffix
                    Value: ba1/
                  - Name: prefix
                    Value: fo1/
            LambdaFunctionArn:
              Fn::GetAtt:
                - HelloHandler2E4FBA4D
                - Arn
      Managed: false
      SkipDestinationValidation: false

As I can't figure out any workaround. Making this a p1 bug.

Facing the same issue: I previously defined two notifications, one with
prefix: person/upload
suffix: .csv

another with
prefix: car/upload
suffix: .csv

This configuration worked with 2.146.0, results in same issue by simply updating cdk to 2.155.0

I’ve been tracking down this same issue, and can confirm that I see an identical error when attempting to update to a new Lambda function ARN on an existing notification configured for an externally-defined bucket.

Abbreviated version of my reproduction, first deployment succeeds:

log_function = aws_lambda.Function(self, "Function1", …)
bucket.add_event_notification(
    aws_s3.EventType.OBJECT_CREATED,
    aws_s3_notifications.LambdaDestination(log_function),
    aws_s3.NotificationKeyFilter(prefix="logs/resource-cdn/", suffix=".gz"),
)

Second deployment fails:

log_function = aws_lambda.Function(self, "Function2", …)
bucket.add_event_notification(
    aws_s3.EventType.OBJECT_CREATED,
    aws_s3_notifications.LambdaDestination(log_function),
    aws_s3.NotificationKeyFilter(prefix="logs/resource-cdn/", suffix=".gz"),
)

Looks like the the same issue #28915.

I was able to reproduce in v2.147+ and it works fine in v2.146.0. However I don't see any related change in the diff. Need to look further into it.

@xazhao Related issue #31413. It might give some clue about the root cause.

Hello everyone, here are my 2 cents. THIS IS NOT A SORTING PROBLEM.

In CDK v2.3 the method used to verify if a notification configuration was external was based on the following check (this is from the Custom Resource behind aws_s3_notifications.LambdaDestination resource):
external_notifications[t] = [n for n in existing_notifications.get(t, []) if not n['Id'].startswith(f"{stack_id}-")]

In CDK v2.136 the method is this one (not checked when this change was introduced):

  ids = [with_id(n) for n in old.get(t, [])]
  old_incoming_ids = [n['Id'] for n in ids]
  external_notifications[t] = [n for n in existing_notifications.get(t, []) if not n['Id'] in old_incoming_ids]

The function with_id(n) is this one:

def with_id(n):
    n['Id'] = f"{stack_id}-{hash(json.dumps(n, sort_keys=True))}"
    return n

The built-in function hash does not return the same value across different instances of python even if the input is the same.

The Custom Resource will treat as external_notifications even the ones that are not, leading to duplicates

EDIT: for versions older than Python 3.2 (included) the hash function generates the same value for the same input across different python instances. Starting from Python 3.3, the hash function starts with a random seed for each python instance.

@xenonlcs Thanks for adding more context here. From the description, it looks like a different issue from the original issue. Do you mean hash() here return different values? Is there a way for me to reproduce the issue?

Hi @xazhao, I will attach this screenshot to explain better what I mean:

As you can see, stopping and restarting a new process of Python leads to the re-initialization of the built-in hash function with another seed. This behavior is introduced starting from version 3.3 of Python. I think that this is not a different issue cause as far as I can see in the code of the custom resource, no code could break due to some sorting problems (the checks are performed with not in). EDIT: I checked again the issue description and it seems that it's also a sorting problem of the returned information by the s3.get_bucket_notification_configuration, or a sorting problem about how the cdk synth is performed, but due to the behaviour of the built-in hash function I think that just sorting the properties will not solve the issue (I have a situation in which there is the same error but only with a prefix).

Since the fact that s3.put_bucket_notification_configuration wants the final configuration for the bucket and not only the items to add, the lambda function tries to understand which notification configurations are not managed by the CloudFormation stack that is being deployed.

So, in the Lambda function we can see that there are 3 different kinds of notification configurations:

• existing_notifications: these are the notification configurations that are currently attached to the s3 bucket (retrieved with s3.get_bucket_notification_configuration)
• incoming: these are the notification configurations that comes from the custom resource (both new and previously deployed)
• external_notifications: these are the notification configurations that could be created manually or by other CloudFormation stacks. These ones must be preserved too.

The check to identify external_notifications is performed comparing the hashes of existing_notifications and the hashes calculated by the function get_id on the old list.

The old list is populated through the custom resource event that details the configuration of the custom resource before the CloudFormation Update event.

Since the get_id function uses the built-in hash function, the hashes calculated on the old list will never match the ones on the existing_notifications list. This leads to treating all the existing_notifications as external_notifications.

The final notification configuration is created merging the external with the incoming ones. All the old notification configurations will be present in both the external and incoming lists.

You can also verify the different hashes on the CloudTrail event for the event named PutBucketNotification (ye, this is one of the rare cases where the SDK function name does not match the API name) originated by the Lambda.

Let me know if I can further explain my point,
Best Regards

Thanks for the explanation. The sorting I added is not for old list but for a property in notification object. For old list, it will calculate hash value for each notification object. Then it calculated each notifications in existing_notifications list to check if hash values match.

This is why sorting matters -- objects will have different hash values:

  "FilterRules": [
      {
          "Name": "Suffix",
          "Value": "foo/"
      },
      {
          "Name": "Prefix",
          "Value": "bar/"
      }
  ]

and

  "FilterRules": [
      {
          "Name": "Prefix",
          "Value": "bar/"
      },
      {
          "Name": "Suffix",
          "Value": "foo/"
      }
  ]

When the lambda function is invoked, it will re-run hashing on those 2 lists (because it will not save the hashing value). No matter what seed it's using, the seed should be the same for calculating hashes for those 2 lists within one single lambda execution.

Yep I agree, i got misled by a previous version of cdk (v2.136.0) in which there was no get_id function.
Sorry for the misunderstanding

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.