cdk deploy intermittently fails with ENOMEM

[bkchy]

Hi,

Recently, a number of people in our development team have been seeing cdk deploy fail with the following error:

[sdk error] CloudFormation.DescribeChangeSet({"StackName":"*******","ChangeSetName":"cdk-deploy-change-set"}) -> Error: connect ENOMEM *******:443 - Local (0.0.0.0:0)

If we enable debug logging for Node.js we see that the connection attempt to cloudformation.us-east-1.amazonaws.com is terminating almost immediately:

NET 31088: connect: attempting to connect to *******:443 (addressType: 4)
NET 31088: destroy
NET 31088: close
NET 31088: close handle

Interestingly, prior connections to CloudFormation to same address have succeeded in the same execution of the deploy command and the error does not always occur in the same spot. It seems every now and then polling the stack status during the deploy causes this error. We have tried a number things without any success:

• tried setting --concurrency 1
• changed between wired and wireless connections.
• changed between home and office network
• tried with or without VPN
• changed TCP keep-alive settings at Mac OS level
• disabled keep-alive entirely by editing the bundled index.js of CDK
• although we thought ENOMEM (no memory) error was a bit misleading we tried to change the Node.js heap settings like --max_old_space_size=... --max_semi_space_size=...
• tried different versions of Node.js 20,x and 22.x.
• upgraded CDK to latest (2.174.1).

Has anyone else run into an issue like this? We have been using AWS CDK on Node.js for almost 6 years now and this is the first time we are seeing this. All our developers are using Macbook with M1 or M2 chip.

Any suggestion for resolution or further diagnostics will be much appreciated.

[advdv]

This started happing to us as well, the error message looks like @bkchy metions, but also sometimes we get an "extended" error. This happens intermittently, we're on MacOS, Node v22.13, CDK is version: 2.175.1 (build afe6e87):

Error occurred while monitoring stack: Error: connect ENOMEM 3.78.206.123:443 - Local (0.0.0.0:0)
    at internalConnect (node:net:1115:16)
    at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18)
    at GetAddrInfoReqWrap.emitLookup [as callback] (node:net:1520:9)
    at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:132:8) {
  errno: -12,
  code: 'ENOMEM',
  syscall: 'connect',
  address: '3.78.206.123',
  port: 443,
  '$metadata': [Object]
}

[bkchy]

We just discovered that disabling realtime internet scanning feature of our Sophos Endpoint software (endpoint security solution) seems to make this problem go away. We will monitor it for next couple of days to see if it's sticky but if you have any network filter/proxy enabled in your Mac (Settings -> Network -> VPN & Filters -> Filters & Proxies) then try disabling them to see if the situation improves.

[advdv]

Right, this could indeed be our AdGuard proxy messing things up. I'll see if things improve, thank you

[yuv-c-netapp]

Right, this could indeed be our AdGuard proxy messing things up. I'll see if things improve, thank you

@advdv Did you manage to isolate the issue to your AdGuard proxy? We are also experiencing a similar error and suspect the firewall.

[advdv]

Unfortunately, no. I spend about an hour going through the settings of AdGuard but i couldn't get it to work and are now simply living with it... If you have any more specific ideas please let me know.

[sudoplatform-engineering]

In our case, we had to add website exclusions in Sophos admin console for some of the AWS URLs that the cdk deploy was using.

[advdv]

@sudoplatform-engineering can you publish these urls here? It would help me (and others) out big time i think. Or link to the resource where they are described? thank you