Step 3.1: Deploy the eksctl bastion host to access and manage a private EKS cluster

• Create an IAM role for the bastion host, with the following policies:

  • Policies for Session Manager:
    • AmazonSSMManagedInstanceCore
  • Policies for eksctl: https://eksctl.io/usage/minimum-iam-policies/
    • AmazonEC2FullAccess
    • AWSCloudFormationFullAccess
    • EKSAllAccess (from the guide above)
    • IAMLimitedAccess (from the guide above))

  

• Create a security group for the bastion host:

  • Inbound rule: default none
  • Outbound rule: HTTPS (port 443) - Source: Everywhere IPv4 (0.0.0.0/0)

• Create the EC2 access instance:

  • AMI & Architecture: Amazon Linux 2023 (should have SSM agent preinstalled) & 64-bit Arm (arm64)
  • Instance type: t4g.nano
  • Create a key pair, and save to local machine
  • Network settings:
    • VPC & Subnet: VPC and the Bastion subnet on step 2
    • Security group: the security group just created
  • EBS Storage: 10 GiB gp3
  • Advanced details:
    • IAM Instance profile: the IAM role just created

  

• Before create a session from the local machine to the bastion host:

  • Install SSM plugin for the AWS CLI on local machine: https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html

  • Enable SSH connections over SSM, following the guide here: https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-getting-started-enable-ssh-connections.html

    # Add this to SSH config file
    # Linux / macOS: ~/.ssh/config
    host i-* mi-*
        ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"
    
    # Windows: C:\\Users\\<username>\\.ssh\\config
    host i-* mi-*
        ProxyCommand C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters portNumber=%p"
    

  • Also make sure to make the downloaded EC2 access key not publicly viewable:

    # Linux / macOS
    chmod 400 <path_to_access_key>
    
    # Windows
    icacls.exe <path_to_access_key> /reset
    icacls.exe <path_to_access_key> /GRANT:R "$($env:USERNAME):(R)"
    icacls.exe <path_to_access_key> /inheritance:r
    

• Then using the EC2 access key, SSH over SSM to the bastion host using instance ID:

  ssh -i <path_to_access_key> ec2-user@<instance_id>
  

  

  

  • If the SSM agent is not detected, try rebooting the bastion host

• In the bastion host, install kubectl and eksctl:

  • kubectl: https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html
  • eksctl: https://eksctl.io/installation/
  • (Optional) After installation, create an EC2 image/AMI, in case the bastion host might be needed to be recreated after cleanup

  # Create a binary directory in home, then register it in PATH
  mkdir -p $HOME/bin
  export PATH=$HOME/bin:$PATH && echo 'export PATH=$HOME/bin:$PATH' >> ~/.bashrc
  
  ## Installing **kubectl** (Linux arm64 version) ##
  # Download the binary
  curl -O <https://s3.us-west-2.amazonaws.com/amazon-eks/1.31.0/2024-09-12/bin/linux/arm64/kubectl>
  # Apply execute permission, and move the binary to PATH
  chmod +x ./kubectl
  # Move the binary to a folder in PATH
  mv ./kubectl $HOME/bin/kubectl
  # Verify kubectl
  kubectl version --client
  
  ## Installing **eksctl** (Linux arm64 version) ##
  # Download the gzip
  curl -LO <https://github.com/eksctl-io/eksctl/releases/latest/download/eksctl_Linux_arm64.tar.gz>
  # Extract the gzip, and move the binary to PATH
  tar -xzf ./eksctl_Linux_arm64.tar.gz
  mv ./eksctl $HOME/bin/eksctl
  rm ./eksctl_Linux_arm64.tar.gz
  # Verify eksctl and IAM role
  eksctl info
  aws sts get-caller-identity
  

Step 3.2: Create the private EKS cluster

• Create a cluster config file for eksctl, here is an example:

  # cluster.yaml
  apiVersion: eksctl.io/v1alpha5
  kind: ClusterConfig
  
  metadata:
    name: eks-demo-cluster
    region: us-east-1
    version: "1.31"
  
  privateCluster:
    enabled: true
  
  vpc:
    subnets:
      private:
        us-east-1a:
          id: subnet-080267c9513e12135
        us-east-1b:
          id: subnet-067af50edb4e14159
  
  managedNodeGroups:
    - name: general
      instanceType: t4g.medium
      minSize: 2
      maxSize: 6
      desiredCapacity: 2
      privateNetworking: true
      volumeSize: 20
  

  • version: "1.31" - Make sure the cluster has same version as the kubectl installed on the bastion host
  • privateCluster: & enabled: true - Make the cluster to be fully-private, by doing these things:
    • disable the cluster API server’s public (default) endpoint
    • enable the private endpoint
    • create the necessary VPC endpoints so the cluster can communicate with AWS
    • add necessary security groups for the cluster’s control plane, worker nodes, and VPC endpoints inside it
  • vpc & subnets - Use existing VPC and subnets configured on step 2
    • Make sure specify the correct subnet’s AZ and ID in the format above
  • nodeGroups ****- Create node groups, specify the instance type and desired capacity (minimum 2 for having 2 AZs)
    • privateNetworking - Must be set to true (nodes are in private subnets)
    • volumeSize: 20 - Size of the EBS volume attached to the each node = 20 GB

• Use eksctl to create the EKS cluster:

  # From local machine, copy the file through SCP to the bastion host
  scp -i <path_to_access_key> ./cluster.yaml ec2-user@<instance_id>:/home/ec2-user
  
  # From bastion host
  	eksctl create cluster -f ./cluster.yaml
  

  • eksctl will start creating the cluster by running two CloudFormation stacks: the cluster stack and the node group stack. This process should take around 20 minutes

  <aside> ☝

  Just right after the cluster stack starts running, do this!

  This stack will create two security groups: ControlPlaneSecurityGroup and ClusterSharedNodeSecurityGroup. When the two security groups are just created, add this inbound rule to each of the security group before the cluster stack finishes.

  HTTPS (port 443) - Source: The eksctl bastion host security group created in step 3.1

  </aside>

  <aside> ❓

  Why adding this inbound rule immediately?

  This is because eksctl will add VPC endpoints in the ClusterSharedNodeSecurityGroup, causing the bastion host to ignore the NAT gateway and use the VPC endpoints instead → This makes the rest of the cluster creation process of eksctl to fail, as the old security group doesn’t allow traffic from the bastion host.

  Currently, in eksctl version 0.194.0, there are no ways to assign more inbound rules to ControlPlaneSecurityGroup and ClusterSharedNodeSecurityGroup through the cluster config file. There are ways to completely replace these two security groups, but I think it is better to use the default rules created by eksctl, then add new rules later.

  What adding the rule to the two security groups does:

  • ControlPlaneSecurityGroup: the rule allows the bastion host to communicate with the control plane through kubectl

  • ClusterSharedNodeSecurityGroup: the rule allows the bastion host to communicate with the newly created VPC Endpoints through eksctl </aside>

  • The cluster stack mainly includes:

    • The EKS cluster
    • 3 Security groups:
      • EKS cluster security group: the default one created by EKS, used for managed worker nodes
      • ControlPlaneSecurityGroup: created by CloudFormation, used for private communication with the cluster’s control plane
      • ClusterSharedNodeSecurityGroup: created by CloudFormation, used for private communication between all managed and unmanaged worker nodes, as well as the security group for VPC Endpoints
    • 5 VPC Endpoints so the private cluster within the VPC can communicate with necessary AWS services

  • The node group stack mainly includes:

    • EC2 Launch template that EKS immediately starts creating 2 EC2 worker nodes

  • During the process, eksctl will also add EKS add-ons and update the kubeconfig so we can manage the cluster through kubectl

    

    

    

  • Test connectivity to the private cluster after creation:

• After the cluster has been created, you might want to give the account that you are using the console with an admin cluster role, with the Access tab on the cluster’s console