Kubernetes Practice — Logging with Logstash and FluentD by Sidecar Container

We are going to learn how to use the Sidecar Container pattern to install Logstash and FluentD on Kubernetes for log aggregation.

For any system, log aggregation is very important. When you use Kubernetes to run your application, the log only belongs to one Pod. If that Pod is deleted, the log is also lost.

Therefore, if we want to track system failures, we must have a log aggregation system. At this time, two popular log stacks are ELK (Elasticsearch Logstash Kibana) and EFK (Elasticsearch FluentD Kibana).

To collect logs on each Pod, we use Sidecar Container.

Sidecar Container

Instead of implementing the log collection process on the application containers, we can separate that process to another container to avoid affecting the performance of the application containers. That container is called Sidecar Container.

Sidecar containers are the containers that should run along with the main container in the pod. This sidecar container extends and enhances the application containers in some way.

Logging with Logstash

The original task of Logstash is monitoring logs and transforming them into a meaningful set of fields and eventually streaming the output to a defined destination. However, it has an issue with performance.

So, Elastic has launched Filebeat that use for monitoring logs and streaming the output to a defined destination.

And Logstash acts as an aggregator that ingests data from a multitude of sources, transforms it, and then sends it to your favorite “stash.”

So we’re done with the theory, let’s get to work. First we deploy a Pod with an application container that writes logs to the file /var/log/access.log, and we deploy a sidecar container on the same Pod that runs Filebeat to collect logs and output logs to Logstash.

Create a file named filebeat.cm.yaml to store the Filebeat configuration file.