Observability with Prometheus and Grafana

Adding Scraping Targets

To better illustrate how Prometheus works, we are going to use the Node Exporter to monitor two additional servers.

Proceed by creating two new Ubuntu 24.04 servers and name them:

• server1
• server2

Our monitoring server should be able to reach both of them; preferably all servers should be in the same private network.

Export the IP addresses as environment variables, as we are going to use them in the configuration file:

# Don't forget to replace the IP addresses with yours
cat <> ~/.bashrc && source ~/.bashrc
export server1=
export server2=
EOF

The Node Exporter is a tool that collects hardware and kernel metrics and exposes them in a format that Prometheus can understand.

It provides comprehensive visibility into the host's health and performance, including:

• CPU: Usage time broken down by mode (user, system, idle, iowait).
• Memory: Total, free, used, cached, and buffered memory.
• Disk I/O: Read/write operations and bytes transferred per device.
• Network: Bandwidth usage, packet errors, and drops per interface.
• Filesystem: Disk space usage, inode usage, and mount point information.
• System Load: The 1-minute, 5-minute, and 15-minute load averages.

We can start by configuring Prometheus to scrape them using this configuration. Since the exporter uses port 9100 to expose its data, our configuration will be as follows:

cat < /etc/prometheus/prometheus.yml
global:
  scrape_interval: 5s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['$server1:9100', '$server2:9100']
EOF

Validate the configuration file:

promtool check config /etc/prometheus/prometheus.yml