Observability with Prometheus and Grafana

The Push Process

To push metrics to the Pushgateway, you can use the curl command or a Prometheus client library. This is a simple example of pushing a metric to the Pushgateway using curl.

echo "my_batch_job_duration_seconds{job=\"my_batch_job\",instance=\"localhost\"} 123" | \
    curl --data-binary \
    @- http://localhost:9091/metrics/job/my_batch_job

In this example:

• We are pushing a metric called my_batch_job_duration_seconds with a value of 123.
• The metric has two labels: job and instance.
• The job label is set to my_batch_job, and the instance label is set to localhost.

my_batch_job_duration_seconds{job="my_batch_job",instance="localhost"} 123

• We are using --data-binary @- to read the metric from standard input (stdin) and send it to the Pushgateway.
• The URL http://localhost:9091/metrics/jobs is the endpoint for pushing metrics to the Pushgateway.
• The /my_batch_job part of the URL specifies the job name for the metric.

# Full URL:
http://localhost:9091/metrics/job/my_batch_job

In a realistic scenario, you would replace the metric name, value, job, and instance labels with dynamic values generated by your batch job in your programming/scripting language of choice. The job could run on any server that has access to the Pushgateway, so you would replace localhost with the Pushgateway's hostname or IP address.

To see the pushed metrics, use curl.

curl -s http://localhost:9091/metrics | \
    grep my_batch_job_duration_seconds

You should be able to see this line representing the pushed metric in the output.

# TYPE my_batch_job_duration_seconds untyped
my_batch_job_duration_seconds{instance="localhost",job="my_batch_job"} 123

Let's try another example using Python's library. Start by creating a virtual environment and installing the library. We will use the monitoring server for this.

apt update
apt install python3-pip -y
pip3 install virtualenv --break-system-packages
virtualenv $HOME/pushgateway_example/venv
source $HOME/pushgateway_example/venv/bin/activate
pip install prometheus_client==0.20.0
mkdir -p $HOME/pushgateway_example
cd