NodeNotReady: Failed to allocate 1 IP addresses on an ENI - despite ample addresses available

[neilharris123]

We recently started using custom networking to allocate IPs to pods from secondary subnets (we allocated a secondary CIDR to our VPC, and create new subnets in the CIDR, and ENIconfigs etc)

We're getting occasional new nodes stuck in a NotReady state.

Having checked the ipamd.log on those nodes, I am seeing that the addon cannot add the secondary ENI due to unavailable prefix. This is despite the fact that at the time of the reported issue, there are over 650 available addresses in the secondary subnet:

{"level":"info","ts":"2024-11-06T10:09:36.238Z","caller":"ipamd/ipamd.go:424","msg":"Got network card index 0 for ENI eni-0a36e415a44a325f2"} {"level":"info","ts":"2024-11-06T10:09:36.238Z","caller":"ipamd/ipamd.go:424","msg":"eni-0a36e415a44a325f2 is of type: interface"} {"level":"debug","ts":"2024-11-06T10:09:36.238Z","caller":"ipamd/ipamd.go:385","msg":"DescribeAllENIs success: ENIs: 1, tagged: 1"} {"level":"debug","ts":"2024-11-06T10:09:36.238Z","caller":"ipamd/ipamd.go:385","msg":"Discovered ENI eni-0a36e415a44a325f2, trying to set it up"} {"level":"debug","ts":"2024-11-06T10:09:36.238Z","caller":"ipamd/ipamd.go:1065","msg":"DataStore add an ENI eni-0a36e415a44a325f2"} {"level":"info","ts":"2024-11-06T10:09:36.238Z","caller":"ipamd/ipamd.go:448","msg":"Found ENIs having 1 secondary IPs and 0 Prefixes"} {"level":"debug","ts":"2024-11-06T10:09:36.238Z","caller":"ipamd/ipamd.go:1128","msg":"Prefix pool stats: Total IPs/Prefixes = 0/0, AssignedIPs/CooldownIPs: 0/0, c.maxIPsPerENI = 224"} {"level":"debug","ts":"2024-11-06T10:09:36.238Z","caller":"ipamd/ipamd.go:1149","msg":"Prefix pool stats: Total IPs/Prefixes = 0/0, AssignedIPs/CooldownIPs: 0/0, c.maxIPsPerENI = 224"} {"level":"info","ts":"2024-11-06T10:09:36.238Z","caller":"ipamd/ipamd.go:385","msg":"ENI eni-0a36e415a44a325f2 set up."} {"level":"info","ts":"2024-11-06T10:09:36.238Z","caller":"ipamd/ipamd.go:470","msg":"Begin ipam state recovery from backing store"} {"level":"debug","ts":"2024-11-06T10:09:36.238Z","caller":"ipamd/ipamd.go:470","msg":"backing store doesn't exists, assuming bootstrap on a new node"} {"level":"debug","ts":"2024-11-06T10:09:36.238Z","caller":"ipamd/ipamd.go:492","msg":"tryUnassignIPsFromENIs"} {"level":"debug","ts":"2024-11-06T10:09:36.238Z","caller":"ipamd/ipamd.go:2044","msg":"No freeable IPs"} {"level":"debug","ts":"2024-11-06T10:09:36.238Z","caller":"ipamd/ipamd.go:607","msg":"Update Rule List with set []"} {"level":"info","ts":"2024-11-06T10:09:36.240Z","caller":"ipamd/ipamd.go:511","msg":"Found sg-00086dbf070996711, added to ipamd cache"} {"level":"info","ts":"2024-11-06T10:09:36.240Z","caller":"ipamd/ipamd.go:511","msg":"Found sg-03dc50cc024105874, added to ipamd cache"} {"level":"info","ts":"2024-11-06T10:09:36.240Z","caller":"ipamd/ipamd.go:511","msg":"Found sg-04340bfc696910a93, added to ipamd cache"} {"level":"info","ts":"2024-11-06T10:09:36.240Z","caller":"ipamd/ipamd.go:511","msg":"Found sg-079e566bf052cb399, added to ipamd cache"} {"level":"info","ts":"2024-11-06T10:09:36.240Z","caller":"ipamd/ipamd.go:511","msg":"Found sg-0daa1755e7520b5ac, added to ipamd cache"} {"level":"info","ts":"2024-11-06T10:09:36.240Z","caller":"ipamd/ipamd.go:523","msg":"Get Node Info for: ip-172-21-75-2.eu-west-1.compute.internal"} {"level":"debug","ts":"2024-11-06T10:09:36.340Z","caller":"eniconfig/eniconfig.go:132","msg":"Using ENI_CONFIG_ANNOTATION_DEF k8s.amazonaws.com/eniConfig"} {"level":"debug","ts":"2024-11-06T10:09:36.340Z","caller":"eniconfig/eniconfig.go:134","msg":"Using ENI_CONFIG_LABEL_DEF topology.kubernetes.io/zone"} {"level":"debug","ts":"2024-11-06T10:09:36.340Z","caller":"ipamd/ipamd.go:565","msg":"IP pool is too low: available (0) < ENI target (1) * addrsPerENI (16)"} {"level":"debug","ts":"2024-11-06T10:09:36.340Z","caller":"ipamd/ipamd.go:2182","msg":"Prefix pool stats: Total IPs/Prefixes = 0/0, AssignedIPs/CooldownIPs: 0/0, c.maxIPsPerENI = 224"} {"level":"debug","ts":"2024-11-06T10:09:36.340Z","caller":"ipamd/ipamd.go:567","msg":"Starting to increase pool size"} {"level":"debug","ts":"2024-11-06T10:09:36.340Z","caller":"ipamd/ipamd.go:799","msg":"Node found \"ip-172-21-75-2.eu-west-1.compute.internal\" - no of taints - 2"} {"level":"debug","ts":"2024-11-06T10:09:36.340Z","caller":"ipamd/ipamd.go:2249","msg":"Prefix target is 1, short of 1 prefixes, free 0 prefixes"} {"level":"debug","ts":"2024-11-06T10:09:36.340Z","caller":"ipamd/ipamd.go:1014","msg":"ToAllocate: 1"} {"level":"debug","ts":"2024-11-06T10:09:36.340Z","caller":"ipamd/ipamd.go:1017","msg":"Skip needs IP check for trunk ENI of primary ENI when Custom Networking is enabled"} {"level":"info","ts":"2024-11-06T10:09:36.340Z","caller":"eniconfig/eniconfig.go:73","msg":"Get Node Info for: ip-172-21-75-2.eu-west-1.compute.internal"} {"level":"debug","ts":"2024-11-06T10:09:36.340Z","caller":"eniconfig/eniconfig.go:132","msg":"Using ENI_CONFIG_ANNOTATION_DEF k8s.amazonaws.com/eniConfig"} {"level":"debug","ts":"2024-11-06T10:09:36.340Z","caller":"eniconfig/eniconfig.go:134","msg":"Using ENI_CONFIG_LABEL_DEF topology.kubernetes.io/zone"} {"level":"info","ts":"2024-11-06T10:09:36.340Z","caller":"ipamd/ipamd.go:849","msg":"Found ENI Config Name: eu-west-1b"} {"level":"info","ts":"2024-11-06T10:09:36.442Z","caller":"ipamd/ipamd.go:825","msg":"ipamd: using custom network config: [sg-04340bfc696910a93 sg-0daa1755e7520b5ac sg-03dc50cc024105874 sg-079e566bf052cb399], subnet-0402b2f2d20212916"} {"level":"debug","ts":"2024-11-06T10:09:36.442Z","caller":"ipamd/ipamd.go:825","msg":"Found security-group id: sg-04340bfc696910a93"} {"level":"debug","ts":"2024-11-06T10:09:36.442Z","caller":"ipamd/ipamd.go:825","msg":"Found security-group id: sg-0daa1755e7520b5ac"} {"level":"debug","ts":"2024-11-06T10:09:36.442Z","caller":"ipamd/ipamd.go:825","msg":"Found security-group id: sg-03dc50cc024105874"} {"level":"debug","ts":"2024-11-06T10:09:36.442Z","caller":"ipamd/ipamd.go:825","msg":"Found security-group id: sg-079e566bf052cb399"} {"level":"debug","ts":"2024-11-06T10:09:36.442Z","caller":"ipamd/ipamd.go:2249","msg":"Prefix target is 1, short of 1 prefixes, free 0 prefixes"} {"level":"debug","ts":"2024-11-06T10:09:36.442Z","caller":"ipamd/ipamd.go:2117","msg":"ToAllocate: 1"} {"level":"info","ts":"2024-11-06T10:09:36.442Z","caller":"awsutils/awsutils.go:795","msg":"Trying to allocate 1 IP addresses on new ENI"} {"level":"debug","ts":"2024-11-06T10:09:36.442Z","caller":"awsutils/awsutils.go:795","msg":"PD enabled - true"} {"level":"info","ts":"2024-11-06T10:09:36.442Z","caller":"awsutils/awsutils.go:914","msg":"Using a custom network config for the new ENI"} {"level":"info","ts":"2024-11-06T10:09:36.442Z","caller":"awsutils/awsutils.go:795","msg":"Creating ENI with security groups: [sg-04340bfc696910a93 sg-0daa1755e7520b5ac sg-03dc50cc024105874 sg-079e566bf052cb399] in subnet: subnet-0402b2f2d20212916"} {"level":"error","ts":"2024-11-06T10:09:36.646Z","caller":"awsutils/awsutils.go:917","msg":"Failed to CreateNetworkInterface InvalidParameterValue: There aren't sufficient free Ipv4 addresses or prefixes\n\tstatus code: 400, request id: 4e11902d-a38a-44d0-8a43-f0b3a9a3592d for subnet subnet-0402b2f2d20212916"} {"level":"error","ts":"2024-11-06T10:09:36.646Z","caller":"ipamd/ipamd.go:825","msg":"Failed to increase pool size due to not able to allocate ENI AllocENI: failed to create ENI: failed to create network interface: InvalidParameterValue: There aren't sufficient free Ipv4 addresses or prefixes\n\tstatus code: 400, request id: 4e11902d-a38a-44d0-8a43-f0b3a9a3592d"} {"level":"warn","ts":"2024-11-06T10:09:36.646Z","caller":"ipamd/ipamd.go:825","msg":"Failed to allocate 1 IP addresses on an ENI: AllocENI: failed to create ENI: failed to create network interface: InvalidParameterValue: There aren't sufficient free Ipv4 addresses or prefixes\n\tstatus code: 400, request id: 4e11902d-a38a-44d0-8a43-f0b3a9a3592d"} {"level":"debug","ts":"2024-11-06T10:09:36.646Z","caller":"ipamd/ipamd.go:870","msg":"Insufficient IP Addresses due to: InvalidParameterValue\n"} {"level":"debug","ts":"2024-11-06T10:09:36.646Z","caller":"ipamd/ipamd.go:567","msg":"Error trying to allocate ENI: AllocENI: failed to create ENI: failed to create network interface: InvalidParameterValue: There aren't sufficient free Ipv4 addresses or prefixes\n\tstatus code: 400, request id: 4e11902d-a38a-44d0-8a43-f0b3a9a3592d"} {"level":"error","ts":"2024-11-06T10:09:36.646Z","caller":"aws-k8s-agent/main.go:42","msg":"Initialization failure: Failed to attach any ENIs for custom networking"}

the config I add to the vpc-cni addon is this:

  ENI_CONFIG_ANNOTATION_DEF: k8s.amazonaws.com/eniConfig
  ENI_CONFIG_LABEL_DEF: topology.kubernetes.io/zone
  AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG: "true"
  ENABLE_PREFIX_DELEGATION: "true"

My understanding is that with ENABLE_PREFIX_DELEGATION: true the addon will try to allocate a /28 range to the new ENI? In which case, considering there are over 650 available IPs in the subnet, why would it not be able to add the ENI?

• Kubernetes version): 1.29
• CNI Version: v1.18.5-eksbuild.1

[neilharris123]

Closing this as I believe this is likely due to subnet fragmentation.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.