In Part 1, we deployed a simple application on Kubernetes. Now, let’s introduce Vault to manage secrets dynamically. Our goal is to populate an environment variable in the app with a secret stored in Vault. We will manually unseal Vault and use the Vault Agent Injector to handle secret injection.
Assumptions
- Part 1 is completed link.
- Helm is installed.
jq
is installed.
Verifying the Cluster
Ensure the cluster is operational by running:
curl app.local
If the app responds correctly, proceed.
Step 1: Add Vault Helm Chart
Add the HashiCorp Helm repository:
helm repo add hashicorp https://helm.releases.hashicorp.com
To explore and modify Vault’s configuration, save the default values:
mkdir helm && cd helm
helm show values hashicorp/vault > default-values.yaml
Install Vault using Helm:
helm install vault hashicorp/vault -f default-values.yaml -n vault --create-namespace
Note: Use
--create-namespace
if the namespace does not exist.
Verify Pods
Check the Vault pods:
kubectl get pods -n vault
Expected output:
vault-0 0/1 Running 0 70s
vault-agent-injector-6c5fcb994-nxhdr 1/1 Running 0 71s
Check Vault Logs
View the logs of the Vault server pod:
kubectl logs pods/vault-0 -n vault
Look for initialization messages.
Check Vault status:
kubectl exec -i -t vault-0 -n vault -- vault status
Key points:
- Seal Type:
shamir
(default manual seal). - Initialized:
false
(fresh server). - Sealed:
true
(no access permitted yet).
Step 2: Initialize and Unseal Vault
Initialize Vault:
kubectl exec -i -t vault-0 -n vault -- vault operator init -key-shares=1 -key-threshold=1 -format=json > vault-keys.json
Extract the unseal key:
jq -r ".unseal_keys_b64[]" vault-keys.json
Set the unseal key:
VAULT_UNSEAL_KEY=$(jq -r ".unseal_keys_b64[]" vault-keys.json)
Unseal Vault:
kubectl exec vault-0 -n vault -- vault operator unseal $VAULT_UNSEAL_KEY
Confirm Vault is initialized and unsealed:
kubectl exec -i -t vault-0 -n vault -- vault status
Step 3: Vault Secret Configuration
Log into Vault with the root token:
jq -r ".root_token" vault-keys.json
kubectl exec --stdin=true --tty=true vault-0 -n vault -- /bin/sh
vault login
Enable the kv-v2
secrets engine:
vault secrets enable kv-v2
Enable Kubernetes authentication:
vault auth enable kubernetes
Configure Kubernetes host:
vault write auth/kubernetes/config kubernetes_host=https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT
Store a secret in Vault:
vault kv put kv-v2/vault-local/local-k8s secret="this is a secret stored in vault and exported with vault injector"
Verify the secret:
vault kv get kv-v2/vault-local/local-k8s
Create a Policy
Create a policy file:
cat <<EOF > /tmp/vault-local-policy.hcl
path "kv-v2/data/vault-local/local-k8s" {
capabilities = ["read"]
}
EOF
Apply the policy:
vault policy write vault-local /tmp/vault-local-policy.hcl
Create a role to bind the policy to a service account:
vault write auth/kubernetes/role/vault-local bound_service_account_names=vault-local bound_service_account_namespaces=default policies=vault-local ttl=1h
Exit vault pod
exit
Step 4: Configure Kubernetes Service Account For Vault
Create a service account, role, and role binding:
mkdir vault-k8s-sa
cd vault-k8s-sa
vim service-account.yaml
# service-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: vault-local
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: vault-local-role
rules:
- apiGroups: [""]
resources: ["serviceaccounts/token"]
verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: vault-default-rolebinding
subjects:
- kind: ServiceAccount
name: vault-local
roleRef:
kind: Role
name: vault-local-role
apiGroup: rbac.authorization.k8s.io
Apply the configuration:
kubectl apply -f service-account.yaml
Now we will copy our exsisting application to a new folder and update the annotations to support Vault
cp -r app app-vault
cd app-vault
rm deployment.yaml
vim deployment.yaml
Update the application deployment:
# app/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: app-vault
spec:
replicas: 1
selector:
matchLabels:
app: app-vault
template:
metadata:
labels:
app: app-vault
annotations:
# Vault Agent Injection annotations to enable Vault integration in the pod
vault.hashicorp.com/agent-inject: "true" # Tells Vault to inject a Vault agent into the pod
vault.hashicorp.com/role: "vault-local" # Role for Vault to access secrets, providing permission for the pod
# Specifies which Vault secret path to inject
vault.hashicorp.com/agent-inject-secret-config: "kv-v2/vault-local/local-k8s" # Secret path to fetch from Vault
# Template to format the secrets and inject them into the container as environment variables
vault.hashicorp.com/agent-inject-template-config: |
{{- with secret "kv-v2/data/vault-local/local-k8s" -}}
export SECRET="{{ .Data.data.secret }}" # Extract the "secret" field from the Vault secret
{{- end }}
spec:
serviceAccountName: vault-local # Service account with Vault permissions assigned to the pod
containers:
- name: app-vault
image: matcham89/app:latest
ports:
- containerPort: 5000
command:
- "sh"
- "-c"
args:
- ". /vault/secrets/config && exec python app.py" # Load the Vault secrets into environment variables before running the app
env:
- name: APP_MESSAGE
value: "Test Application" # The environment variable for the application message
Update the service manifest
vim service.yaml
apiVersion: v1
kind: Service
metadata:
name: app-vault
spec:
selector:
app: app-vault
ports:
- protocol: TCP
port: 80
targetPort: 5000
Deploy the new application & service:
kubectl apply -f deployment.yaml
kubectl apply -f service.yaml
Navigate to the Ingress folder and update the configuration
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: default-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
ingressClassName: nginx
rules:
- host: app.local
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: app
port:
number: 80
- host: app-vault.local
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: app-vault
port:
number: 80
kubectl apply -f default.yaml
Update the etc host files to reflect the new ingress map
(remember to checkout the external loadbalncer IP and ensure the kind api is running)
vim /etc/hosts
# local cluster
172.18.0.5 app.local
172.18.0.5 app-vault.local
Check on the newly created pod:
kubectl get pods
You should see a count of 2/2
for the ready status.
One is the init container to inject the secrets, the second is our applicaiton.
With the application ready, we can test to see if the secret injection is working:
curl app-vault.local
Expected output:
<html>
<head><title>Application Title</title></head>
<body>
<h1>Test Application</h1>
<p><strong>Vault Secret:</strong> this is a secret stored in vault and exported with vault injector</p>
</body>
</html>
You can also verify the injected secret directly:
kubectl exec -i -t app-XXXXXX -c app-vault -- cat /vault/secrets/config
Expected output:
export SECRET="this is a secret stored in vault and exported with vault injector" # Extract the "secret" field from the Vault secret
Conclusion
We have successfully integrated Vault into our Kubernetes application to dynamically inject secrets.
This setup forms the foundation for managing secrets securely in production environments.