Accessing Private EKS Cluster from CodePipeline within the Same VPC

[qcserestipy]

Context

I have a setup where:

1. An EKS Cluster is deployed in Pipeline A. This pipeline creates and manages the EKS cluster.
2. In a separate pipeline (Pipeline B), I import the EKS cluster using the eks.Cluster.from_cluster_attributes method. This pipeline is responsible for deploying application resources such as Helm charts and Kubernetes manifests.

Both pipelines are within the same AWS account and region. The VPC configuration and the networking setup in both pipelines ensure that the CodeBuild steps are run within the same VPC as the EKS cluster.

Problem

• When the EKS cluster endpoint is configured as public_and_private, everything works as expected, and the resources are successfully deployed to the cluster from Pipeline B.
• However, I want to keep the EKS cluster endpoint private for security reasons. When I change the endpoint to private, the connection from the kubectl lambda function in Pipeline B to the EKS cluster times out.
• The timeout occurs during attempts to interact with the EKS API (e.g., deploying Helm charts or applying Kubernetes manifests).

What I’ve Tried

• I've ensured that CodeBuild is running inside the same VPC as the EKS cluster by specifying the VPC configuration in the pipeline.
• The security groups associated with both the EKS cluster and the CodeBuild have been configured to allow traffic between them. It would have to be evaluated whether I do that correctly.

Expected Behavior

The kubectl lambda in Pipeline B should be able to access the private EKS cluster endpoint without timing out when the cluster is configured as private. This should allow the pipeline to deploy resources (Helm charts, Kubernetes manifests) without the need for a public endpoint.

Actual Behavior

The kubectl lambda is unable to reach the private EKS cluster endpoint, resulting in a timeout during Kubernetes-related operations (such as Helm installations or kubectl commands).

Request for Assistance

• How can I configure the pipeline (specifically, CodeBuild) to successfully communicate with a private EKS cluster endpoint?
• Do I have to change settings in which vpc the lambda function is located or what security groups are there?
• Are there additional networking or security group rules I need to apply for a fully private cluster to be accessed by a pipeline within the same VPC?

Code Snippets

Cluster Import Stack in Pipeline B

from aws_cdk import (
    aws_eks as eks,
    aws_ssm as ssm,
    aws_iam as iam,
    aws_dynamodb as dynamodb,
    aws_ec2 as ec2,
    Stack
)
from constructs import Construct

class EksClusterImportStack(Stack):
    def __init__(
            self, 
            scope: Construct, 
            id: str,
            table: dynamodb.Table,
            account: str,
            region: str,
            **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        self.cluster_name = ssm.StringParameter.value_for_string_parameter(
            self, "/eks/clusterName"
        )

        self.cluster_arn = ssm.StringParameter.value_for_string_parameter(
            self, "/eks/clusterArn"
        )

        self.cluster_endpoint = ssm.StringParameter.value_for_string_parameter(
            self, "/eks/clusterEndpoint"
        )

        self.security_group_ids = ssm.StringParameter.value_for_string_parameter(
            self, "/eks/clusterSecurityGroup"
        )

        self.open_id_connect_provider_arn = ssm.StringParameter.value_for_string_parameter(
            self, "/eks/oidc/provider_arn"
        )

        self.kubectl_lambda_role_arn = ssm.StringParameter.value_for_string_parameter(
            self, "/eks/kubectl/lambda/role_arn"
        )

        self.kubectl_role_arn = ssm.StringParameter.value_for_string_parameter(
            self, "/eks/kubectl/role_arn"
        )

        self.kubectl_security_group_id = ssm.StringParameter.value_for_string_parameter(
            self, "/eks/kubectl/sg_id"
        )

        self.vpc_id = ssm.StringParameter.value_from_lookup(
            self, parameter_name="/eks/vpc_id"
        )

        self.kubectl_lambda_role=iam.Role.from_role_arn(
            self, 
            "ImportedKubectlLambdaRole", 
            self.kubectl_lambda_role_arn
        )

        self.vpc=ec2.Vpc.from_lookup(self, 'ImportedVpc', vpc_id=self.vpc_id)

        pipeline_sg = ec2.SecurityGroup(
            self, 
            "CodePipelineSG",
            vpc=self.vpc,
            allow_all_outbound=True,
            description="Security group for CodePipeline to access EKS cluster",
        )

        eks_cluster_sg = ec2.SecurityGroup.from_security_group_id(
            self, 'EKSClusterSG', self.kubectl_security_group_id
        )

        eks_cluster_sg.add_ingress_rule(
            peer=pipeline_sg,
            connection=ec2.Port.tcp(443),
            description="Allow CodePipeline to access the EKS cluster",
        )

        self.cluster = eks.Cluster.from_cluster_attributes(
            self, 
            'ImportedCluster',
            cluster_name=self.cluster_name,
            cluster_endpoint=self.cluster_endpoint,
            kubectl_role_arn=self.kubectl_role_arn,
            kubectl_lambda_role=self.kubectl_lambda_role,
            kubectl_security_group_id=self.kubectl_security_group_id,
            open_id_connect_provider=eks.OpenIdConnectProvider.from_open_id_connect_provider_arn(
                self, 
                "ImportedOidcProvider",
                self.open_id_connect_provider_arn 
            ),
            security_group_ids=[
                self.security_group_ids,
                pipeline_sg.security_group_id,
            ],
            vpc=self.vpc
        )

        # Kubernetes Service Account
        _ = self.cluster.add_service_account(
            "TestServiceAccount",
            name="test-sa",
            namespace="kube-system"
        )

Relevant Pipeline B

class PipelineStack(Stack):
    def __init__(
        self, 
        scope: Construct, 
        id: str, 
        config: dict, 
        **kwargs) -> None:
        super().__init__(scope, id, **kwargs)
        env = kwargs.get("env")
        account = env.account
        # Define a policy statement for DescribeCluster
        describe_cluster_policy = iam.PolicyStatement(
            actions=["eks:DescribeCluster"],
            resources=[f"arn:aws:eks:{config['eks']['target_region']}:{account}:cluster/{config['eks']['cluster_name']}"],
            effect=iam.Effect.ALLOW
        )
        access_eksparams_policy = iam.PolicyStatement(
            actions=[
                "ssm:GetParameter",
                "ssm:GetParameters",
                "ssm:GetParameterHistory",
                "ssm:DescribeParameters",
                "ssm:PutParameter",
                "ssm:DeleteParameter",
                "ssm:AddTagsToResource"
            ],
            resources=[f"arn:aws:ssm:{config['eks']['target_region']}:{account}:parameter/eks/*"],
            effect=iam.Effect.ALLOW
        )

        self.vpc_id = ssm.StringParameter.value_from_lookup(
            self, parameter_name="/eks/vpc_id"
        )
        vpc = ec2.Vpc.from_lookup(self, 'Vpc', vpc_id=self.vpc_id)



        source_repo = codecommit.Repository.from_repository_name(
            self, 
            f"{config['pipeline']['repositoryname']}-repo", 
            repository_name=config['pipeline']['repositoryname']
        )
        pipeline_source = CodePipelineSource.code_commit(source_repo, config['pipeline']['branchname'], code_build_clone_output=True)
        pipeline = CodePipeline(self, f"{config['pipeline']['repositoryname']}",
            pipeline_name=f"{config['pipeline']['repositoryname']}",
            cross_account_keys=True,
            enable_key_rotation=True,
            synth=CodeBuildStep(
                f"{config['pipeline']['repositoryname']}-buildStep",
                input=pipeline_source,
                build_environment=codebuild.BuildEnvironment(
                    build_image=codebuild.LinuxBuildImage.STANDARD_7_0,
                    compute_type=codebuild.ComputeType.SMALL,
                ),
                install_commands=[
                    "npm install -g aws-cdk",
                    "pip install -r requirements.txt",
                ],
                commands=[
                   "cdk synth -q"
                ],
                role_policy_statements=[
                    describe_cluster_policy,
                    access_eksparams_policy
                ],
                vpc=vpc,
                subnet_selection=ec2.SubnetSelection(
                    subnets=vpc.private_subnets
                ),
            )   
        )

Cluster Creation Stack in Pipeline A

class EksClusterStack(Stack):
    def __init__(
        self,
        scope: Construct, 
        id: str,
        policy_kms_cross_account_usage: iam.ManagedPolicy,
        network_stack: Stack,
        config: dict, 
        phase: str,
        **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        self.config = config
        self.phase = phase
        env = kwargs.get("env")
        self._region = env.region
        self._account = env.account

        self.cluster = eks.Cluster(
            self,
            id=config['eks']['cluster_name'],
            cluster_name=config['eks']['cluster_name'],
            version=eks.KubernetesVersion.V1_30,
            vpc=network_stack.vpc,
            vpc_subnets=[SubnetSelection(subnets=network_stack.vpc.private_subnets)],
            role=self.create_controller_role(),
            kubectl_layer=KubectlV30Layer(self, "kubectl"),
            default_capacity=0,
            endpoint_access=eks.EndpointAccess.PUBLIC_AND_PRIVATE
        )