Prototyping a Log Collector Service

The different SDKs allow you to interact with the Docker API to create, manage, and monitor containers. They open up a world of possibilities for building applications that leverage Docker. In this section, we're going to prototype a log collector service that streams logs from running containers and saves them to a SQLite database.

First of all, start a verbose container to generate logs:

docker run -d \
    --name random-logger \
    alpine \
    sh -c "while true; do echo hello; sleep 10; done"

Now, we're going to use the Python SDK. This is the initial program:

import docker
import threading

def stream_container_logs(container):
    """
    Stream logs from a single container.
    """
    for log in container.logs(stream=True):
        print(f"Logs from {container.name}: {log.decode().strip()}")

def main():
    # Create a Docker client
    client = docker.from_env()

    # List all running containers
    containers = client.containers.list()

    # Create and start a thread for each container to stream logs
    threads = []
    for container in containers:
        thread = threading.Thread(target=stream_container_logs, args=(container,))
        thread.start()
        threads.append(thread)

    # Wait for all threads to complete (optional)
    for thread in threads:
        thread.join()

if __name__ == "__main__":
    main()

The idea here is to create a thread for each container to stream logs. However, this program doesn't store logs yet. For the sake of simplicity, we're going to use SQLite to store these logs.

First, create a virtual environment and a folder for the application:

# Create a folder for the log collector service
mkdir ${HOME}/log-collector
cd ${HOME}/log-collector

# Create a Python virtual environment
python3.14 -m venv .venv

We need to run the following commands that create the database and the table to store logs:

cat <<EOF > ${HOME}/log-collector/init.py
import sqlite3

def create_db():
    conn = sqlite3.connect('container_logs.db')
    cursor = conn.cursor()
    cursor.execute('''
        CREATE TABLE IF NOT EXISTS logs (
            timestamp DATETIME,
            host TEXT,
            container_id TEXT,
            log_line TEXT
        )
    ''')
    conn.commit()
    conn.close()

if __name__ == "__main__":
    create_db()
EOF

Run the script to initialize the database:

python init.py

Now, let's go back to the initial code and make it save logs to the database we just created:

cat <<EOF > ${HOME}/log-collector/main.py
import docker
import sqlite3
import datetime
import threading
import socket

def save_log_to_db(container_id, log_line):
    """Save log line to SQLite database."""
    conn = sqlite3

Painless Docker - 2nd Edition

A Comprehensive Guide to Mastering Docker and its Ecosystem

Enroll now to unlock all content and receive all future updates for free.

Unlock now ~~$31.99~~$25.59

Hurry! This limited time offer ends in:

To redeem this offer, copy the coupon code below and apply it at checkout:

Learn More

Previous Next