Step 7: Add the secrets as environment variables in the cluster

In AWS Secrets Manager, there should already be a secret for each RDS instance.
- If your project requires more key/value secrets, create them here before deploying them to EKS. (In this example, my backend applications require JWT secrets)

Step 7.1: Install Secrets Store CSI Driver to fetch the secrets from Secrets Manager

Create a VPC Endpoint to connect the AWS Secrets Manager service to the private cluster:
- Service: com.amazonaws.us-east-1.secretsmanager
- VPC & Subnet: EKS subnet 1 & EKS subnet 2
- Security group: ClusterSharedNodeSecurityGroup

Install the Secrets Store CSI Driver + AWS Secrets Manager provider :

Download the manifest files:
Create a private ECR repository called:
- sig-storage/csi-node-driver-registrar (secrets-store-csi-driver.yaml)
- sig-storage/livenessprobe (secrets-store-csi-driver.yaml)
- csi-secrets-store/driver (secrets-store-csi-driver.yaml)
- aws-secrets-manager/secrets-store-csi-driver-provider-aws (aws-provider-installer.yaml)

From the local machine pull the images from what the manifest file indicated, then push to your own repository:

docker pull --platform linux/arm64 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.11.1

docker tag registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.11.1 <aws_id>.dkr.ecr.us-east-1.amazonaws.com/sig-storage/csi-node-driver-registrar:v2.11.1

docker push <aws_id>.dkr.ecr.us-east-1.amazonaws.com/sig-storage/csi-node-driver-registrar:v2.11.1

# Do the same for the other images

Modify the manifest files:

# secrets-store-csi-driver.yaml
image: <aws_id>.dkr.ecr.us-east-1.amazonaws.com/sig-storage/csi-node-driver-registrar:v2.11.1
image: <aws_id>.dkr.ecr.us-east-1.amazonaws.com/csi-secrets-store/driver:v1.4.6
image: <aws_id>.dkr.ecr.us-east-1.amazonaws.com/sig-storage/livenessprobe:v2.13.1

# aws-provider-installer.yaml
image: <aws_id>.dkr.ecr.us-east-1.amazonaws.com/aws-secrets-manager/secrets-store-csi-driver-provider-aws:1.0.r2-75-g1f97be0-2024.10.17.19.45

Then apply the Secrets Store CSI driver to the cluster:

# From local machine, copy the file through SCP to the bastion host
scp -i <path_to_access_key> \\
	./rbac-secretproviderclass.yaml \\
	./csidriver.yaml \\
	./secrets-store.csi.x-k8s.io_secretproviderclasses.yaml \\
	./secrets-store.csi.x-k8s.io_secretproviderclasspodstatuses.yaml \\
	./secrets-store-csi-driver.yaml \\
	./rbac-secretprovidersyncing.yaml \\
	./aws-provider-installer.yaml \\
	ec2-user@<instance_id>:/home/ec2-user/helpers

# From bastion host
kubectl apply -f helpers/rbac-secretproviderclass.yaml
kubectl apply -f helpers/csidriver.yaml
kubectl apply -f helpers/secrets-store.csi.x-k8s.io_secretproviderclasses.yaml
kubectl apply -f helpers/secrets-store.csi.x-k8s.io_secretproviderclasspodstatuses.yaml
kubectl apply -f helpers/secrets-store-csi-driver.yaml
kubectl apply -f helpers/rbac-secretprovidersyncing.yaml
kubectl apply -f helpers/aws-provider-installer.yaml

# Verify the driver
kubectl get ds -n kube-system

Configure IAM role and k8s Service Account for the controller:
- We have already set up an IAM OpenID Connect provider for the cluster on step 6.1. Now, we only have to create IAM policies:
  
  <aside> ⚠️
  
  Depending on your application, you might have to create multiple IAM policies.
  
  For this project, I have 2 backend applications, each uses its own RDS database and has its own secret. I want to setup IAM policy for each of the backend app, so it only has access to the corresponding database.
  
  Two IAM policies will be created:
  
  *AWSSecretsManagerSSCSIPolicy-EKSDemoCustomer: access to Customer RDS DB secret and the access token secret*
  
  *AWSSecretsManagerSSCSIPolicy-EKSDemoShopping: access to Shopping RDS DB secret and the access token secret*
  
  </aside>
```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret"
            ],
            "Resource": [
                "arn:aws:secretsmanager:<region>:<account-id>:secret:<rds-secret>",
                "arn:aws:secretsmanager:<region>:<account-id>:secret:<token-secret>"
            ]
        }
    ]
}
```
- Copy the ARN of the policies, then create an IAM role and a k8s service account for each:
```
eksctl create iamserviceaccount \\
	--cluster=eks-demo-cluster \\
	--name=aws-secrets-manager-sscsi-<backend-name> \\
	--attach-policy-arn=arn:aws:iam::<AWS_ACCOUNT_ID>:policy/AWSSecretsManagerSSCSIPolicy-EKSDemo<backend-name> \\
	--override-existing-serviceaccounts \\
	--region us-east-1 \\
	--approve
```

Step 7.2: Using SSCSI, create Secrets and ConfigMaps for the deployments

Create the ConfigMaps for non-sensitive values such as URLs:

# config-frontend.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: coffeeshop-frontend-config
data:
  MY_APP_API_CUSTOMER_URL: <https://eks-demo.bachhv.com/api/customer>
  MY_APP_API_SHOPPING_URL: <https://eks-demo.bachhv.com/api/shopping>
---
# config-customer.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: coffeeshop-customer-config
data:
  DB_HOST: <db_customer_endpoint> # ...us-east-1.rds.amazonaws.com
  DB_PORT: "5432"
  DB_NAME: <db_customer_name>
---
# config-shopping.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: coffeeshop-shopping-config
data:
  DB_HOST: <db_shopping_endpoint> # ...us-east-1.rds.amazonaws.com
  DB_PORT: "5432"
  DB_NAME: <db_shopping_name>
  API_CUSTOMER_URL: http://<customer-svc-name>.default.svc.cluster.local:4000/api/customer

Using SecretProviderClass, create the secrets from AWS

parameters are used to select the secrets from Secrets Manager
secretsObjects are used to create Secret objects once SCI mount volumes to the pods
(Also in a microservice architecture with multiple backends, might want to create a provider for each backend, as each of them tends to have its own database)

# secretprovider-shopping.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: coffeeshop-shopping-spc
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "arn:aws:secretsmanager:<region>:<account-id>:secret:<rds-secret>"
        jmesPath:
          - path: username
            objectAlias: username
          - path: password
            objectAlias: password
      - objectName: "arn:aws:secretsmanager:<region>:<account-id>:secret:<token-secret>"
        jmesPath:
          - path: JWT_ACCESS_SECRET
            objectAlias: JWT_ACCESS_SECRET
          - path: JWT_REFRESH_SECRET
            objectAlias: JWT_REFRESH_SECRET
  secretObjects:
    - secretName: coffeeshop-shopping-secret
      type: Opaque
      data:
        - objectName: username
          key: DB_USERNAME
        - objectName: password
          key: DB_PASSWORD
    - secretName: coffeeshop-token-secret
      type: Opaque
      data:
        - objectName: JWT_ACCESS_SECRET
          key: JWT_ACCESS_SECRET
        - objectName: JWT_REFRESH_SECRET
          key: JWT_REFRESH_SECRET
          
# Do the same for secretprovider-customer.yaml

Applying the manifests:
```
# From local machine, copy all deployment and HPA manifests
scp -i <path_to_access_key> ./config-*.yaml ./secretprovider-*.yaml ec2-user@<instance_id>:/home/ec2-user/manifests

# From bastion host
kubectl apply -f './manifests/config-*.yaml'
kubectl apply -f './manifests/secretprovider-*.yaml'
```
<aside> ⚠️

The secrets will only sync once you start a pod mounting the secrets as volume. Next step: Editing the deployment to do 2 things:
- Mount the secrets volume from SSCSI to the pods
- Apply the Secrets (and ConfigMaps) as environment variables
→ The syncing feature will do the rest

</aside>

Then edit the deployment manifest of each app, then reapply them. This is an example on an edited backend deployment manifest (The frontend deployment manifest do not have any secrets, so no need to mount anything):

# deploy-shopping.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: coffeeshop-shopping-deploy
  labels:
    app: coffeeshop-shopping
spec:
  replicas: 2
  selector:
    matchLabels:
      app: coffeeshop-shopping
  template:
    metadata:
      labels:
        app: coffeeshop-shopping
    spec:
	    **# Make sure to select the correct service account
      serviceAccountName:** aws-secrets-manager-sscsi-shopping
	    **# Select the correct provider to create the voulme
      volumes:
	      - name: coffeeshop-shopping-secretvolume
	        csi:
	          driver: secrets-store.csi.k8s.io
	          readOnly: true
	          volumeAttributes:
	            secretProviderClass: coffeeshop-shopping-spc**
      containers:
        - name: eks-demo-coffeeshop-shopping
          image: mortredn/eks-demo-coffeeshop-shopping:latest
          resources:
            limits:
              cpu: "250m"
              memory: "512Mi"
          ports:
            - containerPort: 4000
          **# Mount the created secrets volumes**
	        **volumeMounts:
		        - name: coffeeshop-shopping-secretvolume
		          mountPath: "/mnt/secrets-store"
		          readOnly: true
          # Adding the Secrets and ConfigMaps as env**
          **env:
            - name: API_CUSTOMER_URL
              valueFrom:
                configMapKeyRef:
                  name: coffeeshop-shopping-config
                  key: API_CUSTOMER_URL
            - name: DB_HOST
              valueFrom:
                configMapKeyRef:
                  name: coffeeshop-shopping-config
                  key: DB_HOST
            - name: DB_PORT
              valueFrom:
                configMapKeyRef:
                  name: coffeeshop-shopping-config
                  key: DB_PORT
            - name: DB_DBNAME
              valueFrom:
                configMapKeyRef:
                  name: coffeeshop-shopping-config
                  key: DB_DBNAME
            - name: DB_USERNAME
              valueFrom:
                secretKeyRef:
                  name: coffeeshop-shopping-secret
                  key: DB_USERNAME
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: coffeeshop-shopping-secret
                  key: DB_PASSWORD
            - name: JWT_ACCESS_SECRET
              valueFrom:
                secretKeyRef:
                  name: coffeeshop-token-secret
                  key: JWT_ACCESS_SECRET
            - name: JWT_REFRESH_SECRET
              valueFrom:
                secretKeyRef:
                  name: coffeeshop-token-secret
                  key: JWT_REFRESH_SECRET**

Applying the manifests:

# When deployment manifests are updated, deployments auto redeploy
kubectl apply -f './manifests/deploy-*.yaml'

# If the deployments do not automatically redeploy, manually restart
kubectl rollout restart deployment coffeeshop-frontend-deploy
kubectl rollout restart deployment coffeeshop-customer-deploy
kubectl rollout restart deployment coffeeshop-shopping-deploy

(Optional) You might have to create an invalidation on CloudFront to refresh the caches on edge locations. Because we just add environment variables to the frontend app, there are configurations needed to be changed.
- Also, if your domain registrar also comes with a caching solution like CloudFront, also invalidate from there (For example, my domain uses Cloudflare)