Local Kubernetes Development with Hashi Corp Vault | Side Car Injection

Today we will learn how to install Hashi Corp Vault in local Kubernetes cluster and how to manage secrets in Vault and also how to inject these secrets via side car to .Net Core container.

Before we start we need a few things to get started,

Docker Desktop with Kubernetes enabled (or you can use other tools such as minikube) to have a local cluster
Helm to install Hashi Corp Vault
kubectl to check deployments and pod status
Visual Studio 2022 for .Net Core Appliction
Lens ( I will be using lens as Kubernetes Dashboard, it is very powerful dashboard tool)
Argo CD and GitHub actions ( for CI/CD purposes, you can skip it and directly deploy from local)

If you want to learn more on Argo CD and GitHub Actions, please follow below link,

https://medium.com/c-sharp-progarmming/azure-function-app-on-kubernetes-with-github-actions-and-argo-cd-series-part-i-624b08233795

https://medium.com/c-sharp-progarmming/azure-function-app-on-kubernetes-with-github-actions-and-argo-cd-series-part-ii-c23631824ad4

Before we dive in to hands on, let’s understand what is vault and how it works in Kubernetes cluster,

What is Vault?

Hashi Corp Vault is an identity-based secrets and encryption management system. A secret is anything that you want to tightly control access to, such as API encryption keys, passwords, and certificates. Vault provides encryption services that are gated by authentication and authorization methods. Using Vault’s UI, CLI, or HTTP API, access to secrets and other sensitive data can be securely stored and managed, tightly controlled (restricted), and auditable.

A modern system requires access to a multitude of secrets, including database credentials, API keys for external services, credentials for service-oriented architecture communication, etc. It can be difficult to understand who is accessing which secrets, especially since this can be platform-specific. Adding on key rolling, secure storage, and detailed audit logs is almost impossible without a custom solution. This is where Vault steps in.

Vault validates and authorizes clients (users, machines, apps) before providing them access to secrets or stored sensitive data.

(Reference: https://www.vaultproject.io/docs/what-is-vault)

How Secrets are injected in to the containers by Vault?

So let’s get started, first we need to Install Vault,

First run below command to add Hashi Corp Helm repository,

                helm repo add hashicorp https://helm.releases.hashicorp.com

2. Now update the Helm repo to get the latest changes,

                helm repo update

3. Now create a new namespace named hscvault, in K8s, this where all our vault related deployment will be kept,

                kubectl create namespace hscvault

4. Now set this namespace to current kubectl config context,

                kubectl config set-context --current --namespace hscvault

5. Now we will deploy, Vault in development mode, there are different profiles available for different purposes such as dev, standalone, ha, and external. For our purpose, we will use dev.

                helm install vault hashicorp/vault --set "server.dev.enabled=true"

6. Now check the pods, we can see that vault has been installed successfully.

Sealed/Unsealed

Whenever a fresh installation of Vault is done, Vault is in sealed state, it is meant that, vault is aware of the storage but unaware of decryption.

For this, we need to unseal the vault, we will need to obtain the root key, so that vault will be able to decrypt it.

For dev installation we do not need to unseal it, as it will be unsealed mode but for standalone and ha mode, it needs to be unsealed.

Now, we will learn how to extract root key and setup Vault for our usage,

Let’s extract the root key by running below command,

                kubectl exec vault-0 -- vault token create

Copy and keep the token safe with you. ( I am sharing the full image as I will be deleting the pod once my blog writing is done, but please be careful, while handling these secrets)

Now we need to setup K8s authentication, so that services or applications that we are deploying can access Vault without using root key, as root key will have maximum privileges. Let’s set it up now,

Start an interactive session in the vault-0 pod,

                kubectl exec --stdin=true --tty=true vault-0 -- /bin/sh

Run below command to enable Kubernetes Authetication,

                vault auth enable kubernetes

Once Kubernetes authentication is completed, we need to provide Kubernetes cluster API details to vault so that it can authenticate it,

                vault write auth/kubernetes/config \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"

Now, we have provided our cluster details to Vault. Now we will create a policy for our applications fetching secrets from vault. Mostly applications only need read access, so we will create a policy with read access for this,

                vault policy write firstvaultapp-policy - <<EOF
path "k8s/data/firstvaultapp/*" {
  capabilities = ["read"]
}
EOF

*Please remember, we need to create a KV engine which will satisfy the above path “”k8s/data/firstvaultapp/*” for which we have created the role. We will create it later using UI. Our path should be “k8s/firstvaultapp”.

Now our policy is also created, so we will create a new role named “firstvaultapp-role” and map it with a Kubernetes Service Account Name and policy “firstvaultapp-policy” (which we just created).

                vault write auth/kubernetes/role/firstvaultapp-role \
    bound_service_account_names=firstvaultapp-sa \
    bound_service_account_namespaces=demo-vault \
    policies=firstvaultapp-policy \
    ttl=24h

The role connects the Kubernetes service account firstvaultapp-sa, and namespacedemo-vault, with the Vault policy, firstvaultapp-policy. The tokens returned after authentication are valid for 24 hours.

Now exit the interactive shell,

                kubectl create sa firstvaultapp-sa -n demo-vault

To verify the same things from UI, you can do port forward for vault-0 pod and login using the root key, extracted earlier,

                kubectl port-forward pods/vault-0 8200:8200

http://localhost:8200/

Now you can go to Access and Policies to verify recently created Policies and Access rules,

Now we are done with our Vault Setup, let’s move on to Side Car Injection Part,

To do that, let’s create a KV secret engine for all our applications related secrets via UI (path to be same as provided in the policy),

Go to Secrets → Enable New Engine → Choose “KV”

click on Next

Let’s create Some Secrets in k8s engine,

Click on Save

Now let’s quickly create a template Web API application in .Net Core, that will return the Secrets in one of the endpoint. We will skip the application creation steps this time, as it is pretty simple and I will share the repo for your reference (https://github.com/arkapravasinha/FirstVaultApp.API )

Our deployment.yml looks like this,

                apiVersion: apps/v1
kind: Deployment
metadata:
  name: firstvaultappapi-deployment
spec:
  selector:
    matchLabels:
      app: firstvaultappapi
  template:
    metadata:
      labels:
        app: firstvaultappapi
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/agent-inject-secret-secret1.txt: k8s/firstvaultapp/testsecrets
        vault.hashicorp.com/agent-inject-secret-secret2.txt: k8s/firstvaultapp/testsecrets
        vault.hashicorp.com/agent-inject-template-secret1.txt: |
          {{ with secret "k8s/firstvaultapp/testsecrets" }}{{ .Data.data.secret1 }}{{ end }}
        vault.hashicorp.com/agent-inject-template-secret2.txt: |
          {{ with secret "k8s/firstvaultapp/testsecrets" }}{{ .Data.data.secret2 }}{{ end }}
        vault.hashicorp.com/role: firstvaultapp-role
    spec:
      serviceAccountName: firstvaultapp-sa
      containers:
        - name: firstvaultappapi-container
          image: ghcr.io/arkapravasinha/firstvaultappapi:v0.1.12
          resources:
            limits:
              memory: 128Mi
              cpu: 500m
          ports:
            - containerPort: 80

If you look closely, we are injecting secrets via annotations and we are also templating it, so that it can be understandable to the application. It is very important to provide Service Account Name so that our pod can authenticate with vault.

And also in the application, we are injecting the secrets in middleware, via Configuration Builder,

Program.cs

I have added a Controller, which will return all the secrets injected,

SecretController.cs

Let’s deploy our application and see, I am using GitHub Actions to build my application and Argo CD to deploy my application using GitOps, you can always deploy manually.