Cloud Native, Scalable and Observable GitLab Runner on Kubernetes

The definition of running "Cloud Native" workloads is running applications in containers - usually microservices - orchestrating them with Kubernetes or other container orchestration tools, and deploying them in the cloud - not necessarily public cloud like AWS, Azure, or GCP. The term "cloud" is far from being limited to the commercial definition of public cloud providers. It's more about how we design, build, deploy, and manage applications.

Cloud Native applications are also "observable" which means they provide the necessary metrics, logs, and distributed traces to monitor, debug, troubleshoot and even scale the application based on the collected data.

GitLab Runner is a critical component in the CI/CD pipeline and hopefully, it provides the necessary metrics to monitor its performance and health, it's also lightweight (microservice) and can be easily containerized and orchestrated with Kubernetes. However, the lack of documentation and examples as well as the complexity of the Kubernetes ecosystem make it difficult for to benefit from the advantages of running GitLab Runner in a Cloud Native environment.

Our goal in this chapter is to provide a step-by-step guide to run GitLab Runner in a Kubernetes cluster, monitor its performance and health, and autoscale it based on custom metrics. We will use many technologies and tools like GitLab, Kubernetes, Helm