Observability with Prometheus and Grafana

Requirements

In this chapter, we are going to install and configure Prometheus on a Linux server. Before proceeding, let's understand the requirements for running this tool.

For storage, Prometheus uses the local disk to store data. Therefore, you need to have enough disk space. The amount of disk space required depends on the number of metrics, time series, and samples per second that you are collecting, and the retention period you configure (the default is 15 days). In general, SSDs are recommended for better performance, but you can also use HDDs or any other storage device as long as it is POSIX-compliant.

ℹ️ POSIX-compliant means that the storage device must support the POSIX standard, which is a set of standards that define the API for the Unix operating system. Most modern file systems are POSIX-compliant, but some network file systems like NFS may not be fully POSIX-compliant.

NFS, as a shared file system that multiple clients can access simultaneously, inherently conflicts with several POSIX file system requirements. These requirements include unique behaviors, such as the ability to continue using a file even after it has been deleted. Such behaviors can be unusual and unexpected, creating compatibility issues and potential data integrity risks when used with systems that rely on strict POSIX compliance. Therefore, most NFS-based storage systems, including cloud-based ones like AWS EFS, are not recommended for Prometheus. The Prometheus maintainers strongly recommend using local file systems.

The storage requirements could be estimated using the following formula:

storage size (GiB) = (retention time (seconds) * ingested samples (per second) * bytes per sample) / 1024³

ℹ️ Prometheus stores an average of 1 to 2 bytes per sample and the default retention time is 15 days (1296000 seconds).

Prometheus is a single binary that can be run on any operating system. In this example, we are going to install it on an Ubuntu server. We will call this server monitoring for the sake of this guide. We will also need 2 other similar servers later to demonstrate how to monitor multiple targets. We will call these servers server1 and server2.

Since our primary focus is monitoring rather than networking, our servers are in the same network, with Internet access and public IP addresses for an easy setup.

When it comes to CPU, you can start with 2 CPU cores. This is enough to start and scale later if you find that bottlenecks are occurring.