Monitoring Kubernetes with Prometheus; a Practical Example

To better understand how to use Prometheus to monitor the state of a cluster and its apps, we will deploy a Hello World app to the cluster to see some metrics in action:

# Create the NS
kubectl create ns hello-world

# Deploy a sample app
kubectl -n hello-world \
  create deployment hello-world \
  --replicas=2 \
  --image=k8s.gcr.io/echoserver:1.10

Our goal here is to monitor the status of the pods running our app. For this use case, kube-state-metrics exporter is a good fit since it provides metrics about the state of Kubernetes objects.

For example, we can use:

kube_pod_status_phase metric to monitor the current phase of pods.
kube_pod_container_status_running to describe whether the container is currently in running state
kube_pod_info to simply get metadata of our pods.
Or other metrics listed in the official documentation.

In Prometheus, these information can be queried using PromQL queries like the following:

# Filter by namespace
kube_pod_status_phase{namespace="hello-world"}

# Filter by pod
kube_pod_status_phase{pod="hello-world-xxxxx-xxxxx"}

# .. and so on

In the following steps, we are going to install Prometheus in our cluster and configure it using Helm (values.yaml):

# Install Helm (if not already installed)
# Choose the desired version
export version="v3.19.0"

# Download the installation script
curl -fsSL -o \
  get_helm.sh \
  https://raw.githubusercontent.com/helm/helm/$version/scripts/get-helm-3
chmod 700 get_helm.sh
# Run the installation script
./get_helm.sh

# Add the Prometheus Helm repository
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

# Update the Helm repositories
helm repo update

Create a values.yaml file to configure Prometheus. Our intention, as described earlier, is to scrape metrics from all pods. Therefore, we need to configure Prometheus to use the pod role in the kubernetes_sd_configs section. The configuration should also replace the namespace and pod labels with the actual values from Kubernetes metadata. This can be done by creating the following values.yaml file:

cat < $HOME/values.yaml
server:
  persistentVolume:
    # Disable persistent volume for the sake of simplicity
    # Should be enabled in production
    enabled: false

kube-state-metrics:
  ## If false, kube-state-metrics sub-chart will not be installed
  enabled: true

serverFiles:
  prometheus.yml:
     # Add our testing job to the Prometheus configuration
     scrape_configs:
      - job_name: "hello-world"
        # Do not change labels
        honor_labels: true
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          # Map all Kubernetes pod labels to Prometheus labels
          - action: labelmap
            regex: __meta_kubernetes_pod_label_(.+)
          # Replace the 'namespace' label
          # with the value from Kubernetes namespace metadata
          - action: replace
            source_labels: [__meta_kubernetes_namespace]
            target_label: namespace
          # Replace the 'pod' label
          # with the value from Kubernetes pod name metadata
          - action: replace
            source_labels: [__meta_kubernetes_pod_name]
            target_label: pod
EOF

After creating the values.yaml file, install Prometheus using Helm:

# Install Prometheus
helm upgrade \
  --install \
  prometheus \
  prometheus-community/prometheus \
  --version 27.45.0 \
  -f $HOME/values.yaml

Port-forward the Prometheus service to access the UI:

# Get the pod name
export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=prometheus,app.kubernetes.io/instance=prometheus" -o jsonpath="{.items[0].metadata.name}")

# Define a port on localhost:
export PROMETHEUS_LOCALHOST_PORT=30003
kubectl --namespace default port-forward \
  $POD_NAME $PROMETHEUS_LOCALHOST_PORT:9090 > /dev/null 2>&1 &

We can now access the Prometheus UI. Since we don't have a web browser on our server, we will create an SSH tunnel from our local machine to the control node (server1 or whatever your control node is called):

# In your local machine:
export CONTROL_NODE_IP="CHANGEME"
export PROMETHEUS_LOCALHOST_PORT="30003"

Observability with Prometheus and Grafana

A Complete Hands-On Guide to Operational Clarity in Cloud-Native Systems

Enroll now to unlock all content and receive all future updates for free.

Unlock now ($36.99) Learn More

Previous Next