Vault in Kubernetes

High availability

I want to have more than 1 instance of vault in cluster, to avoid errors when node is unavailable + store secrets in database.

Earlier, i had a vault as a single instance in EC2 and DynamoDB as a HA storage.

So using RDS is a tradeoff and need to migrate secrets to Postgres by instructions on site. It’s easy to be honest.

Eventually i get the same data, but in Postgres.

What i do?

1. create namespace vault
2. use the following configuration in vault. I use AWS KMS to auto unseal

storage "postgresql" {
  connection_url = "postgres://{{vault.dbuser}}:{{vault.dbpassword}}@{{vault.dbhost}}:{{vault.dbport}}/{{vault.dbname}}?sslmode=disable"
  ha_enabled = "true"
}

seal "awskms" {
  region = "{{vault.seal.region}}"
  kms_key_id = "{{vault.seal.kms}}"
  access_key = "{{vault.seal.accesskey}}"
  secret_key = "{{vault.seal.secretkey}}"
}


or version with consul

storage "consul" {
  address = "consul-consul-server.consul:8500"
  path    = "vault"
}

seal "awskms" {
  region = "eu-central-1"
  kms_key_id = "aa"
  access_key = "aa"
  secret_key = "bb"
}

3. Create secret by configuration and install vault via helm in HA mode with 3 replicas

kubectl create secret generic vault-storage-config -n vault --from-file=config.hcl"

helm repo add hashicorp https://helm.releases.hashicorp.com
kubectl create ns vault
helm install vault hashicorp/vault -n vault --set='injector.enabled=true' --set 'server.ha.enabled=true' --set 'server.ha.replicas=3' --set='server.dataStorage.size=1Gi' --set='server.extraVolumes[0].type=secret' --set='server.extraVolumes[0].name=vault-storage-config' --set='server.extraArgs=-config=/vault/userconfig/vault-storage-config/config.hcl'

Auto unseal

I faced a problem when need to auto unseal vault in Kubernetes when node restarts. It happens for example if you use spot instances, otherwise you need to unseal each instance manually by kube-proxy or connecting directly to pod

The easiest way:

1. create iam user with appropriate policy to encrypt/decrypt/describe key
2. create key and link it to user
3. update vault config to use seal:

seal "awskms" {
  region = "eu-west-1"
  kms_key_id = "xxx"
  access_key = "aaa"
  secret_key = "bbb"
}

4. unseal with special key: vault operator unseal -migrate

that’s it

Vault secrets

Basically it has a full explanation on official site, but again, the simplest way:

1. create policy where we describe what path can be used by kubernetes service account:

vault policy write "app-policy" -<<EOF
path "kv*" {
capabilities = ["read"]
}
EOF

2. enable kubernetes authentication

vault auth enable kubernetes

3. connect vault config with kubernetes account credentials

vault write auth/kubernetes/config token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

4. allow namespaces and service accounts to use vault (in this case allows all)

vault write auth/kubernetes/role/app-role bound_service_account_names=* bound_service_account_namespaces=* policies=app-policy ttl=1h

5. in deployment section of manifest need to add the following metadata

spec:
  selector:
    matchLabels:
      app: yourapp
  replicas: 3
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: true
        vault.hashicorp.com/agent-inject-secret-app: kv/data/yourapp
        vault.hashicorp.com/role: app-role

then vault mounts secrets as a file app by path /vault/secrets/app

app – because it has suffix agent-inject-secret-app

keep in mind that only values created by kv v.1 engine will be transformed directly to env property file like this:

AA=BB

otherwise need to use mapping by annotation:

vault.hashicorp.com/agent-inject-template-xxx

Vault secrets as env variables via ExternalSecret

Some applications unable to load parameters from file, so need connect Vault with secrets. We can do it via ExternalSecret: https://github.com/external-secrets/kubernetes-external-secrets

helm repo add external-secrets https://external-secrets.github.io/kubernetes-external-secrets/
helm install vault-secrets external-secrets/kubernetes-external-secrets --set env.VAULT_ADDR=http://vault:8200 -n vault

as result you will see:
NAME: vault-secrets
NAMESPACE: vault
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The kubernetes external secrets has been installed. Check its status by running:
  kubectl --namespace vault get pods -l "app.kubernetes.io/name=kubernetes-external-secrets,app.kubernetes.io/instance=vault-secrets"

Visit https://github.com/external-secrets/kubernetes-external-secrets for instructions on how to use kubernetes external secrets

Then need to create a resource with auto secret

apiVersion: "kubernetes-client.io/v1"
kind: ExternalSecret
metadata:
  name: nameofyoursecret
spec:
  backendType: vault
  vaultRole: app-role
  dataFrom:
   - kv/data/testsecret

After that you will see that secret has been created and can be linked with pod