Using a Distributed Cache to Share Data Between Runners | Cloud Native CI/CD with GitLab

Cloud Native, Scalable and Observable GitLab Runner on Kubernetes

87%

Imagine your Kubernetes cluster is configured to autoscale its nodes based on the load (Cluster Autoscaler) and your GitLab Runner is configured to autoscale its pods based on the number of jobs in the queue (Runner Autoscaler). This combination is powerful and enables you to run multiple jobs in parallel on different nodes in the cluster. However, there is a challenge when it comes to caching data between runners.

There are some scenarios where you need to share data between runners that run on different nodes or even different clusters. This is where a distributed cache comes into play. GitLab supports different types of distributed caches, including S3-compatible storage like AWS S3 and Minio, Google Cloud Storage, and Azure Blob Storage.

If you have an account on S3, you can try the S3 cache, start by creating a bucket. Create a user and attach the following policy:

{
  "Version":"2012-10-17",
  "Statement":[
    {
      "Sid":"GitlabRunnerCachePolicy",
      "Effect":"Allow",
      "Action":[
        "s3:PutObject",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:DeleteObject"
      ],
      "Resource":[
        "arn:aws:s3:::/*",
        "arn:aws:s3:::"
      ]
    }
  ]
}

Change YOUR_BUCKET_NAME by the name of your bucket. Note that you can fine-tune the policy to allow only the necessary actions and resources. Then export these variables as environment variables; we will use them later:

# The name of the bucket to use for the cache
export BUCKET_NAME=
# The location of the bucket
export BUCKET_LOCATION=
# Your S3 access
export ACCESS_KEY=
# Your S3 secret
export SECRET_KEY=

Now, create a Kubernetes secret to store the S3 access and secret keys:

kubectl create secret generic gitlab-runner-s3-cache \
--from-literal=accesskey="$ACCESS_KEY" \
--from-literal=secretkey="$SECRET_KEY" \
--namespace default

Finally, update the GitLab Runner Helm values file to use the S3 cache:

cat < $HOME/todo/gitlab-runner/helm/values.yaml
gitlabUrl: https://gitlab.com/
runnerRegistrationToken: "$GITLAB_RUNNER_TOKEN"
rbac:
  create: true
serviceAccount:
  create: true
runners:
  privileged: true
  config: |
    [[runners]]
      [runners.kubernetes]
        namespace = "{{.Release.Namespace}}"
        image = "python:3.12"
      [runners.cache]
        Type = "s3"
        Shared = true
        [runners.cache.s3]
          ServerAddress = "s3.$BUCKET_LOCATION.amazonaws.com"
          BucketName = "$BUCKET_NAME"
          BucketLocation = "$BUCKET_LOCATION"
          Insecure = false

  cache:
    secretName: gitlab-runner-s3-cache
EOF

Cloud Native CI/CD with GitLab

From Commit to Production Ready

Enroll now to unlock all content and receive all future updates for free.

Unlock now ($29.99) Learn More

Previous Next