Cloud Native CI/CD with GitLab

Automating the Deployment with GitLab CI/CD

In this section, we will consider out K3s cluster as a staging environment where we will deploy as soon as we push changes to the repository. We will imagine that our developers are working on different feature branches and they want to deploy their changes to this staging environment to test them before merging them into the main branch. As example, we have the following branches:

• feature-A
• feature-B
• feature-C

As a developer, you can create a branch from the main branch, make changes, and push them to the repository. Once you push the code, GitLab CI/CD will build the Docker image, package the Helm chart, and deploy the application to a dedicated namespace in the K3s cluster.

Therefore, as a result, we will have 3 namespaces in the K3s cluster:

• feature-a
• feature-b
• feature-c

This multi-namespace setup will allow us to test different features in isolation.

ℹ️ This approach is opionated and can be adapted to your needs but whatever you choose, what you will learn here will be useful.

In order to achieve this, we will use the following stages in the pipeline:

• build: This stage will build the Docker image and push it to the GitLab Container Registry.
• package: This stage will package the Helm chart and upload it to the GitLab Package Registry.
• deploy: This stage will deploy the Helm chart to the K3s cluster.

Let's start step by step.

The Docker Build Stage

Since this stage will use Docker to build and push the image to the GitLab Container Registry, therefore we need to run Docker in Docker (DinD). In GitLab CI/CD, we can use the docker:dind image as a service that will be available to the build job when running commands like docker build and docker push.

This is how the .gitlab-ci.yml file looks like:

variables:
  DEPLOYMENT_ID: $CI_COMMIT_REF_SLUG-$CI_COMMIT_SHORT_SHA
  IMAGE_NAME: $CI_REGISTRY_IMAGE:$DEPLOYMENT_ID

build:
  stage: build
  image: docker:24.0.5
  services:
      - docker:24.0.5-dind
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker build -t $IMAGE_NAME .
    - docker push $IMAGE_NAME
  rules:
    - if: '$CI_COMMIT_BRANCH =~ /^feature-/ && $CI_COMMIT_MESSAGE =~ /deploy/'

Once the image is built and pushed to the GitLab Container Registry, we can move to the next stage.

The Helm Package Stage

In this stage, we will package the Helm chart and upload it to the GitLab Package Registry. We are going to use the okteto/helm:3.16.2 image to run Helm commands without the need to install Helm on the runner. We are also going to create a use the unique deployment ID to create a unique Helm chart.

variables:
  DEPLOYMENT_ID: $CI_COMMIT_REF_SLUG-$CI_COMMIT_SHORT_SHA
  HELM_CHART_FOLDER: $CI_PROJECT_DIR/manifests/helm/todo
  HELM_CHART_NAME: todo-$DEPLOYMENT_ID
  HELM_CHANNEL: "stable"
  HELM_PACKAGE_REGISTRY_API_ENDPOINT: $CI_API_V4_URL/projects/$CI_PROJECT_ID/packages/helm/api/$HELM_CHANNEL/charts

package:
  stage: package
  image: okteto/helm:3.16.2
  script:
    - 'cd $HELM_CHART_FOLDER'
    - 'rm -f *.tgz'
    - 'sed -i "s/^name:.*/name: $HELM_CHART_NAME/" Chart.yaml'
    - 'sed -i "s/^appVersion:.*/appVersion: $DEPLOYMENT_ID/" Chart.yaml'
    - 'sed -i "s/^tag:.*/tag: $DEPLOYMENT_ID/" values.yaml'
    - 'helm package .'
    - 'export HELM_PACKAGE_PATH=$(ls -d *.tgz)'
    - >
      curl --fail-with-body --request POST
      --form "chart=@$HELM_PACKAGE_PATH"
      --user gitlab-ci-token:$CI_JOB_TOKEN
      "$HELM_PACKAGE_REGISTRY_API_ENDPOINT"
  rules:
    - if: '$CI_COMMIT_BRANCH =~ /^feature-/ && $CI_COMMIT_MESSAGE =~ /deploy/'

The Helm Deploy Stage

Since we are going to use Helm to deploy the application to the K3s cluster, we need to be able to authenticate with the cluster. Before we deploy the application, we need to generate a Kubeconfig file, store it as a secret in GitLab, and use it in the pipeline.

ℹ️ The Kubeconfig file contains the necessary information to connect to the Kubernetes cluster, including the cluster server, the certificate authority, and the service account token. It is used by the kubectl command-line tool to interact with the cluster.

The following script will help you generate the Kubeconfig file. It will:

• Create a service account in the cluster. A service account is an identity that can be used by pods to authenticate with the Kubernetes API.
• Grant cluster-wide admin permissions to the service account. This is necessary to deploy applications to the cluster.
• Get a fresh service account token with a long duration (1 year). This token will be used to authenticate with the cluster.
• Get the cluster details. This includes the cluster server and the certificate authority data.
• Generate the Kubeconfig file and test it.
• Encode the Kubeconfig file in base64 and store it in a file.

# Variables
SERVICE_ACCOUNT_NAME="gitlab-ci"
NAMESPACE="gitlab-ci"
CLUSTERROLE="cluster-admin"
KUBECONFIG_FILE="/tmp/kubeconfig-gitlab.yaml"

# Step 0: Create a namespace
echo "Creating namespace '${NAMESPACE}'..."
kubectl create namespace $NAMESPACE || {
    echo "Namespace '${NAMESPACE}' already exists."
}

# Step 1: Create a service account
echo "Creating service account '${SERVICE_ACCOUNT_NAME}' in namespace '${NAMESPACE}'..."
kubectl create serviceaccount $SERVICE_ACCOUNT_NAME -n $NAMESPACE || {
    echo "Service account '${SERVICE_ACCOUNT_NAME}' already exists."
}

# Step 2: Grant cluster-wide admin permissions
echo "Granting cluster-wide admin rights to '${SERVICE_ACCOUNT_NAME}'..."
kubectl create clusterrolebinding "${SERVICE_ACCOUNT_NAME}-admin" \
    --clusterrole=$CLUSTERROLE \
    --serviceaccount=$NAMESPACE:$SERVICE_ACCOUNT_NAME || {
    echo "ClusterRoleBinding '${SERVICE_ACCOUNT_NAME}-admin' already exists."
}

# Step 3: Get a fresh service account token with a long duration (1 year)
echo "Fetching a fresh token for the service account..."
TOKEN=$(kubectl create token $SERVICE_ACCOUNT_NAME -n $NAMESPACE --duration=8766h)
if [[ -z "$TOKEN" ]]; then
    echo "Failed to retrieve the token. Please check the service account and cluster configuration."
    exit 1
fi

# Step 4: Get cluster details
echo "Retrieving cluster details..."
CLUSTER_SERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
# The cluster server will contain 127.0.0.1 and we need to replace it with the IP address of the "dev" machine
PUBLIC_IP=$(curl -s http://ifconfig.me)
CLUSTER_SERVER=$(echo $CLUSTER_SERVER | sed "s/127.0.0.1/$PUBLIC_IP/")
CA_CERTIFICATE_DATA=$(kubectl config view --raw --minify -o jsonpath='{.clusters[0].cluster.certificate-authority-data}')

# Step 5: Generate kubeconfig file
echo "Generating kubeconfig file '${KUBECONFIG_FILE}'..."
cat < $KUBECONFIG_FILE
apiVersion: v1
kind: Config
clusters:
- name: kubernetes
  cluster:
    server: $CLUSTER_SERVER
    certificate-authority-data: $CA_CERTIFICATE_DATA
contexts:
- name: gitlab-ci-context
  context:
    cluster: kubernetes
    namespace: $NAMESPACE
    user: gitlab-ci
current-context: gitlab-ci-context
users:
- name: gitlab-ci
  user:
    token: $TOKEN
EOF

echo "Kubeconfig file '${KUBECONFIG_FILE}' created successfully."

# Step 6: Test the kubeconfig file
echo "Testing the kubeconfig file..."
KUBECONFIG=$KUBECONFIG_FILE kubectl get pods -n $NAMESPACE || {
    echo "Failed to use the kubeconfig file. Please check the permissions and configuration."
    exit 1
}

# Step 7: Encode the kubeconfig file in base64
echo "Encoding the kubeconfig file in base64..."
KUBECONFIG_BASE64=$(base64 -w 0 $KUBECONFIG_FILE)
echo "$KUBECONFIG_BASE64" > $KUBECONFIG_FILE.base64

The above commands should be run on the "dev" machine where the K3s cluster is running. Once you have the Kubeconfig file, view its content and copy it to the clipboard:

cat /tmp/kubeconfig-gitlab.yaml.base64

In GitLab, go to Settings > CI/CD > Variables and create a new variable with the following details:

• Type: Variable
• Environments: All
• Visibility: Masked
• Key: KUBECONFIG_CI
• Value: The content of the Kubeconfig file

Save the variable.

Since we are going to pull the image from our CI k8s cluster, we need to have a secret to pull the image from the GitLab private registry. This was already done in the previous sections using the following command:

kubectl create secret docker-registry gitlab-registry \
--docker-server=registry.gitlab.com \
--docker-username=$GITLAB_USERNAME \
--docker-password=$GITLAB_TOKEN

However, the secret is created in the default namespace. We need to create the secret in the namespace where we are going to deploy the application. Since the name of the namespace is the same as the $CI_COMMIT_REF_SLUG, and since this value is dynamic, we need to create the secret dynamically in the pipeline.

We will use the predefined variables $CI_REGISTRY, $CI_REGISTRY_USER and $CI_REGISTRY_PASSWORD, as well as the $HELM_NAMESPACE