Step 7: Add environment variables as secrets in the cluster

<aside> 🤔

In AWS Secrets Manager, there should already be a secret for each RDS instance.

If your apps require more key/value secrets, create more secrets in AWS Secrets Manager, before going forward to step 7.1. (In this example, my backend apps require one more secret to store JWT secret values)

</aside>

Step 7.1: Create more secrets in AWS Secrets Manager

Create the necessary secrets, then copy the ARNs of the secrets for later use

<aside> 🤔

For this project, my 2 backend apps need to access JWT access & refresh secret values → I want to create one AWS Secrets Manager secret resource to store those values.

</aside>

Step 7.2: Install Secrets Store CSI Driver to fetch the secrets from AWS Secrets Manager

Create a VPC Endpoint to connect the AWS Secrets Manager service to the private cluster:
- Service: com.amazonaws.us-east-1.secretsmanager
- VPC & Subnet: EKS subnet 1 & EKS subnet 2
- Security group: ClusterSharedNodeSecurityGroup
Install the Secrets Store CSI Driver + AWS Secrets Manager provider :

<aside> ℹ️

The official installation guide: https://secrets-store-csi-driver.sigs.k8s.io/getting-started/installation
- https://github.com/kubernetes-sigs/secrets-store-csi-driver
- https://github.com/aws/secrets-store-csi-driver-provider-aws </aside>
- Download the manifest files:
  
  <aside> 📝
  
  We will need:
  - 5 main manifest files for SSCSI Driver
  - Optional rbac-secretprovidersyncing to sync secrets-store as k8s Secrets (used as environment variables for Deployments)
  - The AWS Provider for the SSCSI Driver </aside>
  - https://raw.githubusercontent.com/kubernetes-sigs/secrets-store-csi-driver/refs/heads/release-1.4/deploy/rbac-secretproviderclass.yaml
  - https://raw.githubusercontent.com/kubernetes-sigs/secrets-store-csi-driver/refs/heads/release-1.4/deploy/csidriver.yaml
  - https://raw.githubusercontent.com/kubernetes-sigs/secrets-store-csi-driver/refs/heads/release-1.4/deploy/secrets-store.csi.x-k8s.io_secretproviderclasses.yaml
  - https://raw.githubusercontent.com/kubernetes-sigs/secrets-store-csi-driver/refs/heads/release-1.4/deploy/secrets-store.csi.x-k8s.io_secretproviderclasspodstatuses.yaml
  - https://raw.githubusercontent.com/kubernetes-sigs/secrets-store-csi-driver/refs/heads/release-1.4/deploy/secrets-store-csi-driver.yaml
  - https://raw.githubusercontent.com/kubernetes-sigs/secrets-store-csi-driver/refs/heads/release-1.4/deploy/rbac-secretprovidersyncing.yaml
  - https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml
- Create a private ECR repository called:
  - sig-storage/csi-node-driver-registrar (in secrets-store-csi-driver.yaml)
  - csi-secrets-store/driver (in secrets-store-csi-driver.yaml)
  - sig-storage/livenessprobe (in secrets-store-csi-driver.yaml)
  - aws-secrets-manager/secrets-store-csi-driver-provider-aws (in aws-provider-installer.yaml)
- From the local machine pull the images from what the manifest file indicated, then push to your own repository:
```
docker pull --platform linux/arm64 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.11.1

docker tag registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.11.1 <aws_id>.dkr.ecr.us-east-1.amazonaws.com/sig-storage/csi-node-driver-registrar:v2.11.1

docker push <aws_id>.dkr.ecr.us-east-1.amazonaws.com/sig-storage/csi-node-driver-registrar:v2.11.1

# Do the same for the other images
```
- Modify the manifest files:
```
# secrets-store-csi-driver.yaml
image: <aws_id>.dkr.ecr.us-east-1.amazonaws.com/sig-storage/csi-node-driver-registrar:v2.11.1
image: <aws_id>.dkr.ecr.us-east-1.amazonaws.com/csi-secrets-store/driver:v1.4.6
image: <aws_id>.dkr.ecr.us-east-1.amazonaws.com/sig-storage/livenessprobe:v2.13.1

# aws-provider-installer.yaml
image: <aws_id>.dkr.ecr.us-east-1.amazonaws.com/aws-secrets-manager/secrets-store-csi-driver-provider-aws:1.0.r2-75-g1f97be0-2024.10.17.19.45
```
- Then apply the Secrets Store CSI driver to the cluster:
```
# From local machine, copy the file through SCP to the bastion host
scp -i <path_to_access_key> \\
	./rbac-secretproviderclass.yaml \\
	./csidriver.yaml \\
	./secrets-store.csi.x-k8s.io_secretproviderclasses.yaml \\
	./secrets-store.csi.x-k8s.io_secretproviderclasspodstatuses.yaml \\
	./secrets-store-csi-driver.yaml \\
	./rbac-secretprovidersyncing.yaml \\
	./aws-provider-installer.yaml \\
	ec2-user@<instance_id>:/home/ec2-user/helpers

# From bastion host
kubectl apply -f helpers/rbac-secretproviderclass.yaml
kubectl apply -f helpers/csidriver.yaml
kubectl apply -f helpers/secrets-store.csi.x-k8s.io_secretproviderclasses.yaml
kubectl apply -f helpers/secrets-store.csi.x-k8s.io_secretproviderclasspodstatuses.yaml
kubectl apply -f helpers/secrets-store-csi-driver.yaml
kubectl apply -f helpers/rbac-secretprovidersyncing.yaml
kubectl apply -f helpers/aws-provider-installer.yaml

# Verify the driver
kubectl get ds -n kube-system
```
Configure IAM role and k8s Service Account for the controller:

<aside> ℹ️

We have already set up an IAM OpenID Connect provider for the cluster on step 6.1. Now, we only have to create IAM policies

</aside>

<aside> 🤔

Depending on your stack, you might have to create an IAM policy & role for each app (backend).

For this project, I have 2 backend apps, each uses its own RDS database (separate DB secret) and the same custom token secret. I want to setup IAM policy & role for each, only providing the necessary access to the correct secrets, to each backend.

Two IAM policies will be created:

*AWSSecretsManagerSSCSIPolicy-EKSDemoCustomer: access to Customer RDS DB secret and the access token secret*

*AWSSecretsManagerSSCSIPolicy-EKSDemoShopping: access to Shopping RDS DB secret and the access token secret*

</aside>
- Create IAM policies:
```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret"
            ],
            "Resource": [
                "arn:aws:secretsmanager:<region>:<account-id>:secret:<rds-secret>",
                "arn:aws:secretsmanager:<region>:<account-id>:secret:<token-secret>" // Manually created if needed
            ]
        }
    ]
}
```
- Copy the ARN of the policies, then create an IAM role and a k8s service account for each:
```
eksctl create iamserviceaccount \\
	--cluster=eks-demo-cluster \\
	--name=aws-secrets-manager-sscsi-<backend-name> \\
	--attach-policy-arn=arn:aws:iam::<AWS_ACCOUNT_ID>:policy/AWSSecretsManagerSSCSIPolicy-EKSDemo<backend-name> \\
	--override-existing-serviceaccounts \\
	--region us-east-1 \\
	--approve
```

Step 7.3: Using SSCSI, create Secrets and ConfigMaps for the deployments

Create the ConfigMaps for non-sensitive values such as URLs:

# config-frontend.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: coffeeshop-frontend-config
data:
  MY_APP_API_CUSTOMER_URL: <https://eks-demo.bachhv.com/api/customer>
  MY_APP_API_SHOPPING_URL: <https://eks-demo.bachhv.com/api/shopping>
---
# config-customer.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: coffeeshop-customer-config
data:
  DB_HOST: <db_customer_endpoint> # ...us-east-1.rds.amazonaws.com
  DB_PORT: "5432"
  DB_NAME: <db_customer_name>
---
# config-shopping.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: coffeeshop-shopping-config
data:
  DB_HOST: <db_shopping_endpoint> # ...us-east-1.rds.amazonaws.com
  DB_PORT: "5432"
  DB_NAME: <db_shopping_name>
  API_CUSTOMER_URL: http://<customer-svc-name>.default.svc.cluster.local:4000/api/customer

Using SecretProviderClass, create the secrets from AWS

parameters are used to select the secrets from Secrets Manager
secretsObjects are used to create Secret objects once SCI mount volumes to the pods
(Also in a microservice architecture with multiple backends, might want to create a provider for each backend, as each of them tends to have its own database)

# secretprovider-shopping.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: coffeeshop-shopping-spc
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "arn:aws:secretsmanager:<region>:<account-id>:secret:<rds-secret>"
        jmesPath:
          - path: username
            objectAlias: username
          - path: password
            objectAlias: password
      - objectName: "arn:aws:secretsmanager:<region>:<account-id>:secret:<token-secret>"
        jmesPath:
          - path: JWT_ACCESS_SECRET
            objectAlias: JWT_ACCESS_SECRET
          - path: JWT_REFRESH_SECRET
            objectAlias: JWT_REFRESH_SECRET
  secretObjects:
    - secretName: coffeeshop-shopping-secret
      type: Opaque
      data:
        - objectName: username
          key: DB_USERNAME
        - objectName: password
          key: DB_PASSWORD
    - secretName: coffeeshop-token-secret
      type: Opaque
      data:
        - objectName: JWT_ACCESS_SECRET
          key: JWT_ACCESS_SECRET
        - objectName: JWT_REFRESH_SECRET
          key: JWT_REFRESH_SECRET
          
# Do the same for secretprovider-customer.yaml

Applying the manifests:
```
# From local machine, copy all deployment and HPA manifests
scp -i <path_to_access_key> ./config-*.yaml ./secretprovider-*.yaml ec2-user@<instance_id>:/home/ec2-user/manifests

# From bastion host
kubectl apply -f './manifests/config-*.yaml'
kubectl apply -f './manifests/secretprovider-*.yaml'
```
<aside> 🤔

The secrets will only sync only when you start a pod, which mounts the secrets as volume.

Next step - Edit the deployment to do 2 things:
- Mount the secrets volume from SSCSI to the pods
- Apply the Secrets (and ConfigMaps) as environment variables
→ The syncing feature will do the rest

</aside>

Then edit the deployment manifest of each app, then reapply them. This is an example on an edited backend deployment manifest (The frontend deployment manifest do not have any secrets, so no need to mount anything):

# deploy-shopping.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: coffeeshop-shopping-deploy
  labels:
    app: coffeeshop-shopping
spec:
  replicas: 2
  selector:
    matchLabels:
      app: coffeeshop-shopping
  template:
    metadata:
      labels:
        app: coffeeshop-shopping
    spec:
	    **# Make sure to select the correct service account
      serviceAccountName:** aws-secrets-manager-sscsi-shopping
	    **# Select the correct provider to create the voulme
      volumes:
	      - name: coffeeshop-shopping-secretvolume
	        csi:
	          driver: secrets-store.csi.k8s.io
	          readOnly: true
	          volumeAttributes:
	            secretProviderClass: coffeeshop-shopping-spc**
      containers:
        - name: eks-demo-coffeeshop-shopping
          image: mortredn/eks-demo-coffeeshop-shopping:latest
          resources:
            limits:
              cpu: "250m"
              memory: "512Mi"
          ports:
            - containerPort: 4000
          **# Mount the created secrets volumes**
	        **volumeMounts:
		        - name: coffeeshop-shopping-secretvolume
		          mountPath: "/mnt/secrets-store"
		          readOnly: true
          # Adding the Secrets and ConfigMaps as env**
          **env:
            - name: API_CUSTOMER_URL
              valueFrom:
                configMapKeyRef:
                  name: coffeeshop-shopping-config
                  key: API_CUSTOMER_URL
            - name: DB_HOST
              valueFrom:
                configMapKeyRef:
                  name: coffeeshop-shopping-config
                  key: DB_HOST
            - name: DB_PORT
              valueFrom:
                configMapKeyRef:
                  name: coffeeshop-shopping-config
                  key: DB_PORT
            - name: DB_DBNAME
              valueFrom:
                configMapKeyRef:
                  name: coffeeshop-shopping-config
                  key: DB_DBNAME
            - name: DB_USERNAME
              valueFrom:
                secretKeyRef:
                  name: coffeeshop-shopping-secret
                  key: DB_USERNAME
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: coffeeshop-shopping-secret
                  key: DB_PASSWORD
            - name: JWT_ACCESS_SECRET
              valueFrom:
                secretKeyRef:
                  name: coffeeshop-token-secret
                  key: JWT_ACCESS_SECRET
            - name: JWT_REFRESH_SECRET
              valueFrom:
                secretKeyRef:
                  name: coffeeshop-token-secret
                  key: JWT_REFRESH_SECRET**

Applying the manifests:

# Reapply the updated deployment manifests
kubectl apply -f './manifests/deploy-*.yaml'

# If the deployments do not automatically redeploy, manually restart
kubectl rollout restart deployment coffeeshop-frontend-deploy
kubectl rollout restart deployment coffeeshop-customer-deploy
kubectl rollout restart deployment coffeeshop-shopping-deploy