Observability with Prometheus and Grafana

Watch Out For Cardinality Explosion

Relabeling can inadvertently increase metric cardinality if not used carefully.

Increasing Cardinality

When you combine or concatenate label values, you create new unique label combinations. For example, if label-1 has 100 unique values and label-2 has 100 unique values, joining them produces up to 10,000 unique time series:

scrape_configs:
  - job_name: 'example'
    static_configs:
      - targets: ['example:8080']
    relabel_configs:
      - source_labels: ['label-1', 'label-2']
        target_label: 'new-label'
        separator: "-"

This might look harmless, but at a large scale it can dramatically inflate the number of time series Prometheus needs to handle, increasing both memory and storage usage.

Reducing Cardinality

Relabeling can also help reduce cardinality by grouping many distinct labels into broader categories.

For example, consider a metric http_requests_total labeled by a unique user path:

http_requests_total{path="/api/v1/user/@marcus-aurelius"} 100
http_requests_total{path="/api/v1/user/@al-ma'arri"} 300
http_requests_total