Painless Docker - 2nd Edition

Optimizing Docker Builds: Leveraging BuildKit Caching & Advanced Techniques

Docker uses a smart layering and caching mechanism to optimize the build process, thanks to its intelligent algorithms and its BuildKit backend. Here are some advanced techniques to further optimize caching in your Dockerfiles.

However, Docker neither knows the content of your application nor the build and runtime tools you use. Therefore, it cannot make intelligent decisions about caching beyond the basic instruction-level caching. The developer, in this case, can help Docker by structuring the Dockerfile in a way that maximizes cache efficiency.

Let's go back to our Python application example:

# Use the official Python base image
FROM python:3.14-slim
# Set the working directory
WORKDIR /app
# Copy the requirements file
COPY requirements.txt .
# Install the dependencies
RUN pip install -r requirements.txt
# Copy the application code
COPY . .
# Run the application
CMD ["python", "app.py"]

The Python package installer (pip) has a feature called a "wheel" - a built package format that allows for faster installations. When pip installs packages, it first looks for pre-built wheels in the local cache before downloading and building packages from source. This means that if you have previously installed a package, pip can reuse the cached wheel for subsequent installations. To leverage this feature in Docker builds, we can use BuildKit's caching capabilities. Here's how you can modify the RUN pip install -r requirements.txt instruction to use a cache mount:

RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt

Here is a clean continuation that sticks strictly to your example and explains what actually improves and why, without introducing new techniques.

With this change, the Dockerfile now looks like this:

# syntax=docker/dockerfile:1
FROM python:3.14-slim
WORKDIR /app
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt
COPY . .
CMD ["python", "app.py"]

At first glance, this may look similar to the previous version, but the behavior during rebuilds is different in an important way.

Normally, when Docker needs to re-run the pip install step (for example, because requirements.txt changed), pip starts from scratch. It downloads packages again and may rebuild them from source, which can be slow, especially for packages with native extensions.

By adding the cache mount, we are telling BuildKit to preserve pip's cache directory (/root/.cache/pip) between builds. This directory is where pip stores downloaded wheels and build artifacts.

Now, when the RUN pip install -r requirements.txt step is executed again:

• pip first checks its cache (the mounted directory)
• If a required package version is already present, pip reuses it
• Only missing or changed packages need to be downloaded or rebuilt

This means that even when Docker cannot reuse the layer cache for the RUN pip install instruction, the command itself becomes much faster because pip is no longer starting from an empty state.

It's important to understand the distinction here:

• Docker layer caching decides whether a step runs at all
• BuildKit cache mounts make a step faster when it must run

In this example:

• If requirements.txt doesn't change, Docker reuses the cached layer and the pip install step is skipped entirely
• If requirements.txt changes, Docker re-runs the step, but pip can reuse cached downloads and builds instead of starting from scratch

Note that we also added the line # syntax=docker/dockerfile:1 at the top of the Dockerfile. This line tells Docker which Dockerfile frontend to use. While it's not required for basic Dockerfiles, it's required in practice when using BuildKit-specific features such as RUN --mount=type=cache