Relabeling: Rules and Actions

39%

Relabeling Actions

In Prometheus, relabeling configurations can perform various actions on labels based on conditions. Here is the list of actions, including the ones we have seen in the previous examples and some new ones.

Replace

The replace action replaces the value of the target label with a new value.

Example:

We want to change the label __scheme__ to scheme:

cat < /etc/prometheus/prometheus.yml && killall -HUP prometheus
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
    relabel_configs:
      - action: replace
        source_labels: ['__scheme__']
        target_label: 'scheme'
EOF

Result:

# Before relabeling:
# Prometheus discovers the target with these default labels:
# __address__="localhost:9090"
# __scheme__="http"
# job="prometheus"

# No "scheme" label exists yet in the exported metrics.

prometheus_build_info{instance="localhost:9090", job="prometheus"} 1
prometheus_tsdb_head_chunks{instance="localhost:9090", job="prometheus"} 4321
process_cpu_seconds_total{instance="localhost:9090", job="prometheus"} 0.42


# After applying relabeling:
# The relabel rule copies the value of "__scheme__" into a new label "scheme".
# __scheme__ is an internal label, so it is dropped automatically after relabeling.
# Result: a new visible label scheme="http" appears in all metrics.

prometheus_build_info{instance="localhost:9090", job="prometheus", scheme="http"} 1
prometheus_tsdb_head_chunks{instance="localhost:9090", job="prometheus", scheme="http"} 4321
process_cpu_seconds_total{instance="localhost:9090", job="prometheus", scheme="http"} 0.42

# Internal labels are dropped

Example:

We want to add the new label env to the target labels and set its value to production:

cat < /etc/prometheus/prometheus.yml && killall -HUP prometheus
scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
    relabel_configs:
      - action: replace
        target_label: env
        replacement: production
EOF

The replace action is usually used with source_labels and target_label because it replaces the value of the target label with a new value, but since we are creating a new label, we don't need to use source_labels. The replacement field acts as the new value for the target label.

Since the default value of the action field is replace, we can omit it:

cat < /etc/prometheus/prometheus.yml && killall -HUP prometheus
scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
    relabel_configs:
      - target_label: env
        replacement: production
EOF

Result:

# Before relabeling:
# Default discovered target labels.
# job="prometheus"
# No "env" label exists yet.

prometheus_build_info{instance="localhost:9090", job="prometheus"} 1
prometheus_tsdb_head_chunks{instance="localhost:9090", job="prometheus"} 4321
process_cpu_seconds_total{instance="localhost:9090", job="prometheus"} 0.42


# After applying relabeling:
# The rule adds a new label env="production" to every time series scraped from this target.

prometheus_build_info{instance="localhost:9090", job="prometheus", env="production"} 1
prometheus_tsdb_head_chunks{instance="localhost:9090", job="prometheus", env="production"} 4321
process_cpu_seconds_total{instance="localhost:9090", job="prometheus", env="production"} 0.42


# Effect summary:
# + Added label: env="production"
# - No targets or metrics dropped
# - Internal labels are dropped

Note that we can do almost the same thing without relabeling using the configuration below. The only difference is that adding labels like this affects both target labels and discovered labels, which is not exactly what we want in this case:

cat < /etc/prometheus/prometheus.yml && killall -HUP prometheus
scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
        labels:
          env: production
EOF

Example:

In our case, we have a label job with the value prometheus. If we want to replace the value of this label with my-prometheus, we can use the following configuration:

cat <<'EOF' > /etc/prometheus/prometheus.yml && killall -HUP prometheus
scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
    relabel_configs:
      - action: replace
        source_labels: ['job']
        target_label: 'job'
        regex: '(.*)'
        replacement: 'my-$1'
EOF

Result:

# Before relabeling:
# Default discovered target labels:
# __address__="localhost:9090"
# job="prometheus"

prometheus_build_info{instance="localhost:9090", job="prometheus"} 1
prometheus_tsdb_head_chunks{instance="localhost:9090", job="prometheus"} 4321
process_cpu_seconds_total{instance="localhost:9090", job="prometheus"} 0.42


# After applying relabeling:
# The rule captures the value of "job" (prometheus) using the regex (.*)
# and replaces it with "my-prometheus".
# Result: every metric from this target now has job="my-prometheus".

prometheus_build_info{instance="localhost:9090", job="my-prometheus"} 1
prometheus_tsdb_head_chunks{instance="localhost:9090", job="my-prometheus"} 4321
process_cpu_seconds_total{instance="localhost:9090", job="my-prometheus"} 0.42

# Dropped internal labels as usual

ℹ️ The $1 in the replacement field refers to the first capture group in the regex, which is (.*). If the job label is app-prometheus, the new value will be my-app-prometheus.

Since the default value of the regex field is (.*), we can do the same thing without specifying it:

cat <<'EOF' > /etc/prometheus/prometheus.yml && killall -HUP prometheus
scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
    relabel_configs:
      - action: replace
        source_labels: ['job']
        target_label: 'job'
        replacement: 'my-$1'
EOF

If we had a label app that could have values like app-24, app-54, app-12, and we want to replace the app label with my-app and its values with 24, 54, and 12, we can use the following configuration:

scrape_configs:
  - job_name: prometheus
    static_configs:
      - tigs:
      - action: replace
        source_labels: ['app']
        target_label: 'my-app'
        regex: 'app-(.*)'
        replacement: '$1'argets: ['localhost:9090']
    relabel_conf

Result:

# Before relabeling:
# app="app-24"
# app="app-54"
# app="app-12"

prometheus_build_info{instance="localhost:9090", job="prometheus", app="app-24"} 1
prometheus_build_info{instance="localhost:9090", job="prometheus", app="app-54"} 1
prometheus_build_info{instance="localhost:9090", job="prometheus", app="app-12"} 1


# After relabeling:
# The rule extracts the numeric part from "app-(.*)"
# and creates a new label "my-app" with only the captured digits.
# The original "app" label is overwritten by "my-app".

prometheus_build_info{instance="localhost:9090", job="prometheus", my-app="24"} 1
prometheus_build_info{instance="localhost:9090", job="prometheus", my-app="54"} 1
prometheus_build_info{instance="localhost:9090", job="prometheus", my-app="12"} 1


# Internal labels are dropped

Example:

You can use multiple source labels to create a new target label. In this example, we will concatenate the __address__ and __metrics_path__ labels to create a new label called full_address. The separator field is used to join the labels together.

For example, if __address__ is localhost:9090 and __metrics_path__ is /metrics, the full_address label will be localhost:9090/metrics. localhost:9090/metrics has the following format: .

This is how we can do it:

cat < /etc/prometheus/prometheus.yml && killall -HUP prometheus
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
    relabel_configs:
      - action: replace
        source_labels: ['__address__', '__metrics_path__']
        target_label: full_address
        separator: ""
EOF

Result:

# Before relabeling:
# __address__="localhost:9090"
# __metrics_path__="/metrics"

prometheus_build_info{instance="localhost:9090", job="prometheus"} 1
prometheus_tsdb_head_chunks{instance="localhost:9090", job="prometheus"} 4321
process_cpu_seconds_total{instance="localhost:9090", job="prometheus"} 0.42


# After relabeling:
# The rule concatenates __address__ and __metrics_path__ with an empty separator ("")
# and assigns the result to a new label "full_address".
# full_address="localhost:9090/metrics"

prometheus_build_info{instance="localhost:9090", job="prometheus", full_address="localhost:9090/metrics"} 1
prometheus_tsdb_head_chunks{instance="localhost:9090", job="prometheus", full_address="localhost:9090/metrics"} 4321
process_cpu_seconds_total{instance="localhost:9090", job="prometheus", full_address="localhost:9090/metrics"} 0.42


# Internal labels are dropped

Remember that the default value of the separator field is ;; that's why we set it to an empty string to avoid having localhost:9090;/metrics as the value of the full_address label.

Example:

Here is another example where replace is used to replace the env label with environment and set its value to test before sending the metrics to a remote storage:

scrape_configs:
  - job_name: 'demo-app'
    static_configs:
      - targets: ['localhost:8000']
        labels:
          env: 'staging'   # provide the 'env' label for the demo app

remote_write:
  - url: 'http://remote-storage.example.com/api/v1/write'
    write_relabel_configs:
      # Rename 'env' -> 'environment' and set its value to 'remote-test' for REMOTE WRITE only
      - action: replace
        source_labels: ['env']
        target_label: 'environment'
        replacement: 'remote-test'

Result:

# Before relabeling:
# env="staging"
# instance="localhost:8000"
# job="demo-app"

demo_temperature_celsius{instance="localhost:8000", job="demo-app", env="staging"} 21.3
demo_requests_total{instance="localhost:8000", job="demo-app", env="staging"} 125
demo_request_latency_seconds_sum{instance="localhost:8000", job="demo-app", env="staging"} 3.72


# After relabeling:
# For remote_write only: 'env' is replaced with 'environment' and value set to 'remote-test'.
# Local TSDB still has env="staging"; the transformation applies to the remote destination.

demo_temperature_celsius{instance="localhost:8000", job="demo-app", environment="remote-test"} 21.3
demo_requests_total{instance="localhost:8000", job="demo-app", environment="remote-test"} 125
demo_request_latency_seconds_sum{instance="localhost:8000", job="demo-app", environment="remote-test"} 3.72


# Internal labels are dropped

Example:

Finally, here is an example to illustrate how to use the action replace to change the severity label to severity_level and set its value to high when the value of the severity label is critical. This is done using alert relabeling:

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']
    alert_relabel_configs:
      # Rename the 'severity' label to 'severity_level'
      # and replace 'critical' with 'high'
      - source_labels: ['severity']
        regex: 'critical'
        target_label: 'severity_level'
        replacement: 'high'

Result:

# Before relabeling:
# severity="critical"
# alertname="PrometheusTargetDown"
# instance="localhost:9090"
# job="prometheus"

# ALERTS indicates whether the alert is firing (1) or not (0)
ALERTS{alertname="PrometheusTargetDown", severity="critical", instance="localhost:9090", job="prometheus", state="firing"} 1

# ALERTS_FOR_STATE shows how long the alert has been in this state (in seconds)
ALERTS_FOR_STATE{alertname="PrometheusTargetDown", severity="critical", instance="localhost:9090", job="prometheus", state="firing"} 300


# After relabeling:
# The rule renames the label "severity" to "severity_level"
# and replaces its value "critical" with "high" before sending to Alertmanager.
# The local alert data in Prometheus remains unchanged.

ALERTS{alertname="PrometheusTargetDown", severity_level="high", instance="localhost:9090", job="prometheus", state="firing"} 1
ALERTS_FOR_STATE{alertname="PrometheusTargetDown", severity_level="high", instance="localhost:9090", job="prometheus", state="firing"} 300


# Internal labels are dropped

For every alert defined in your Prometheus server, new metrics called ALERTS and ALERTS_FOR_STATE are created automatically to represent the alert and its state over time.

ℹ️ The ALERTS metric indicates whether an alert is currently firing (1) or not (0). The ALERTS_FOR_STATE metric provides additional context about the state of the alert, such as how long it has been firing or pending.

Keep and Drop

If you want to filter out targets and metrics based on whether label values match a regex pattern, you can use the keep and drop actions.

Example:

Let's say we have a job that scrapes two targets: localhost:8000 (our dockerized demo application) and localhost:9090 (Prometheus itself). If we want to keep only the metrics from Prometheus and drop the metrics from the demo application, we can use the following configuration:

cat < /etc/prometheus/prometheus.yml && killall -HUP prometheus
scrape_configs:
  - job_name: demo-app-and-prometheus
    static_configs:
      - targets:
          - 'localhost:8000'
          - 'localhost:9090'
    relabel_configs:
      - action: keep
        source_labels: [__address__]
        regex: 'localhost:9090'
EOF

Result:

# Before relabeling:
# __address__="localhost:8000"
# __address__="localhost:9090"

demo_temperature_celsius{instance="localhost:8000", job="demo-app-and-prometheus"} 21.3
demo_requests_total{instance="localhost:8000", job="demo-app-and-prometheus"} 125
prometheus_build_info{instance="localhost:9090", job="demo-app-and-prometheus"} 1
prometheus_tsdb_head_chunks{instance="localhost:9090", job="demo-app-and-prometheus"} 4321


# After relabeling:
# The rule keeps only targets where __address__ matches "localhost:9090".
# The demo-app target at localhost:8000 is dropped before scraping.

prometheus_build_info{instance="localhost:9090", job="demo-app-and-prometheus"} 1
prometheus_tsdb_head_chunks{instance="localhost:9090", job="demo-app-and-prometheus"} 4321


# Effect summary:
# - Kept target: localhost:9090
# - Dropped target: localhost:8000
# - Internal labels are dropped as usual

Example:

Our application (Prometheus exporter demo) exports a custom metric called demo_requests_created with labels such as env, region, status, and version. Here are some example metrics exported by the application:

demo_requests_created{env="production",region="us-east-1",status="error",version="v1.2.0"} 1
demo_requests_created{env="staging",region="us-east-1",status="success",version="v1.1.0"} 2
demo_requests_created{env="staging",region="ap-south-1",status="success",version="v1.2.0"} 3
demo_requests_created{env="production",region="us-west-2",status="success",version="v2.0.0"} 5
demo_requests_created{env="dev",region="eu-west-1",status="error",version="v1.2.0"} 4
demo_requests_created{env="dev",region="us-east-1",status="timeout",version="v1.0.0"} 2

If you want to filter out all metrics for region="ap-south-1" and env="staging", you can use the following configuration:

cat < /etc/prometheus/prometheus.yml && killall -HUP prometheus
scrape_configs:
  - job_name: 'demo-app'
    static_configs:
      - targets: ['localhost:8000']
    metric_relabel_configs:
      - action: drop
        source_labels: [region, env]
        regex: 'ap-south-1;staging'
EOF

The same thing can be achieved by setting the separator to _ and adjusting the regex accordingly:

cat < /etc/prometheus/prometheus.yml && killall -HUP prometheus
scrape_configs:
  - job_name: 'demo-app'
    static_configs:
      - targets: ['localhost:8000']
    metric_relabel_configs:
      - action: drop
        source_labels: [region, env]
        separator: '_'
        regex: 'ap-south-1_staging'
EOF

Result:

# Before relabeling:
# region="us-east-1", env="production"
# region="us-east-1", env="staging"
# region="ap-south-1", env="staging"
# region="us-west-2", env="production"
# region="eu-west-1", env="dev"
# region="us-east-1", env="dev"

demo_requests_created{env="production",region="us-east-1",status="error",version="v1.2.0"} 1
demo_requests_created{env="staging",region="us-east-1",status="success",version="v1.1.0"} 2
demo_requests_created{env="staging",region="ap-south-1",status="success",version="v1.2.0"} 3
demo_requests_created{env="production",region="us-west-2",status="success",version="v2.0.0"} 5
demo_requests_created{env="dev",region="eu-west-1",status="error",version="v1.2.0"} 4
demo_requests_created{env="dev",region="us-east-1",status="timeout",version="v1.0.0"} 2


# After relabeling:
# The rule combines source labels (region, env) into "region;env" or "region_env".
# It drops metrics where region="ap-south-1" and env="staging".
# Only one series matches that pattern and is removed.

demo_requests_created{env="production",region="us-east-1",status="error",version="v1.2.0"} 1
demo_requests_created{env="staging",region="us-east-1",status="success",version="v1.1.0"} 2
# (dropped) demo_requests_created{env="staging",region="ap-south-1",status="success",version="v1.2.0"} 3
demo_requests_created{env="production",region="us-west-2",status="success",version="v2.0.0"} 5
demo_requests_created{env="dev",region="eu-west-1",status="error",version="v1.2.0"} 4
demo_requests_created{env="dev",region="us-east-1",status="timeout",version="v1.0.0"} 2

# Internal labels are dropped

Example:

Let's see a more advanced example. We have dynamic staging environments that we want to monitor with Prometheus. Each staging environment exports a metric called demo_requests_total with labels such as env, region, status, and version. When the status of a staging environment is timeout, we know that the environment was destroyed and we don't want to scrape it anymore to prevent getting stale data and to avoid unnecessary load on Prometheus.

To summarize, we want to drop targets where:

env label is set to staging.
status label is set to timeout.

Dropping targets means that we don't want to scrape them at all - in other words, we want to filter them before scraping them and in a dynamic way (whenever the combination of label values matches the criteria).

This is an example of our time series before relabeling:

All env values are set to staging.
Some status values are set to success, error, or timeout.

demo_requests_total{env="staging",region="us-east-1",status="success",version="v1.2.0"} 100
demo_requests_total{env="staging",region="us-west-2",status="error",version="v2.0.0"} 50
demo_requests_total{env="staging",region="eu-west-1",status="success",version="v1.1.0"} 75
demo_requests_total{env="staging",region="ap-south-1",status="timeout",version="v1.0.0"} 30

This is what we could do:

cat < /etc/prometheus/prometheus.yml && killall -HUP prometheus
scrape_configs:
  - job_name: 'demo-app'
    static_configs:
      - targets: ['localhost:8000']
    relabel_configs:
      - action: drop
        source_labels: [env, status]
        regex: 'staging;timeout'
EOF

The problem is that relabel_configs runs before scraping and can only see target-level labels - in other words, the labels you see on the Service Discovery page. The env, for example, exists as a metric label, not a target label: it's only visible after scraping. Because of that, the env label is not available during relabeling and the drop never matches. The same applies to the status label.

To fix this problem, we can use metric_relabel_configs, which runs after scraping and can see metric labels like env:

cat < /etc/prometheus/prometheus.yml && killall -HUP prometheus
scrape_configs:
  - job_name: 'demo-app'
    static_configs:
      - targets: ['localhost:8000']
    metric_relabel_configs:
      - action: drop
        source_labels: [env, status]
        regex: 'staging;timeout'
EOF

Result: Prometheus still scrapes the target, but drops matching time series before storage.

This is not exactly what we wanted, because the target is still scraped. We wanted to drop the target before scraping.

To achieve that, we can make the env and status labels available at the target level using static_configs labels:

cat < /etc/prometheus/prometheus.yml && killall -HUP prometheus
scrape_configs:
  - job_name: 'demo-app'
    static_configs:
      - targets: ['localhost:8000']
        labels:
          env: 'staging'  # or any other value
          status: 'timeout'  # or any other value
    relabel_configs:
      - action: drop
        source_labels: [env, status]
        regex: 'staging;timeout'
EOF

Observability with Prometheus and Grafana

A Complete Hands-On Guide to Operational Clarity in Cloud-Native Systems

Enroll now to unlock all content and receive all future updates for free.

Unlock now ($36.99) Learn More

Previous Next