Dockerfile Frontends: The build file upgrade we need

(Originally from my cloud native DevOps newsletter #36.)

Did you know the Dockerfile keeps getting new features in something known as Dockerfile frontends? Have you seen this first line in a Dockerfile example?

# syntax=docker/dockerfile:1
FROM something

That first commented line is optional but tells the BuildKit image builder in Docker to support new features that expand on the original Dockerfile specification from a decade ago.

The before times

Before BuildKit and Dockerfile frontends existed, if we wanted to take advantage of some new syntax in a Dockerfile, say, when COPY --chown was added, or multi-stage, we'd have to ensure we had an updated version of Docker Engine to build Dockerfiles with that feature. That wasn't great, especially if it was updated on our local machine and "worked on my machine" but CI was building on an older Docker Engine version that failed the build. That happened to me in 2016-2020, a lot.

Since 2018, BuildKit has externalized that Dockerfile parser into a "frontend" so that neither Docker Engine nor BuildKit decided what features you could use in your Dockerfile.

The builder of 2023

In 2023, BuildKit became the default image builder in all editions of Docker, and is on by default in Docker's official GitHub Actions image builder. This is awesome because it means as long as we have Docker Engine v23+ installed, and add that first syntax line in our Dockerfiles, we will always build images with the latest Dockerfile 1.x features, regardless of our Docker Engine version or BuildKit version. Here's a short history of Docker+BuildKit, how the relate, and the various ways to use BuildKit in Docker CLI/Engine.

But why do we even need that syntax line?

If you've never put that syntax line in your Dockerfile, you may have been using newer features (like RUN --mount for ssh, cache, and secrets) and not realizing the backup behavior of BuildKit if you don't specify a syntax.

A BuildKit install (which is usually installed when you install Docker Engine or Docker Desktop) is bundled with the latest version of the Dockerfile frontend spec that was released when your Buildkit version was released. It will use that frontend spec by default when you use any docker build or docker buildx build command.

But what if your Docker Engine's BuildKit version is just a little older, and you want to use a brand new feature like ADD git@github.com/... to git clone a repo into your image without ever needing git installed? Cool, right? It was shipped in Dockerfile frontend v1.6 from June 2023.

Well, adding that feature caused a build failure for me today because I didn't put the # syntax=docker/dockerfile:1 line in my Dockerfile, so rather than downloading the latest Dockerfile frontend 1.x version from the docker/dockerfile repo on Docker Hub, BuiltKit just used its backup plan: defaulting to the frontend that was shipped with it. In my case, having BuildKit v0.11.6 installed included the older frontend v1.5, so I got a weird error that failed to build because that frontend parser didn't know how to ADD git URLs yet.

Remember to always...

# syntax=docker/dockerfile:1
FROM something

But what about non-BuildKit builders?

Did you know there is no OCI standard for how an image is built, or the build file itself?

OCI standards are only concerned with the resulting image format, the registry API that stores it, and the runtime that creates a container. There is a lot of confusion on the web around a "Dockerfile OCI standard" that doesn't exist. Some image builders outside of the official docker build claim to be "Dockerfile compatible," but what they really mean is the pre-BuildKit Dockerfile 1.0, circa 2018. They may support a few modern features, but it's spotty and far from 100% compatible. If you were trying to make an advanced Dockerfile with all the latest Dockerfile frontend features and yet keep it "working with any image builder" you'd be disappointed.

I'm not saying another build tool isn't worth looking at, but you'll want to eventually standardize on one builder ecosystem and stick with their pros and cons.

My favorite Dockerfile frontend features

When Docker announced Dockerfile frontends back in 2018, I was unsure of its future, if it would catch on, and if I would use its features. Six years later, the BuildKit and Dockerfile maintainers are still adding valuable features to the Dockerfile syntax that I recommend and use. Here are a few:

• v1.8 added a linter built into BuildKit, which runs on each build. A significant advantage over external linters is that this reports by default on each build and can also block builds if BuildKit thinks the Dockerfile is invalid or if you enable it to block on lint fails (which you can override.)
• v1.7 added string substitution in variable expansion. This should reduce complexity when needing to replace strings dynamically in Dockerfile statements (e.g. replacing arm64 with aarch64 in download URLs.)
• v1.6 added the ability to use ADD for git URLs to avoid needing git in the image.
• v1.4 added COPY --link and ADD --link for injecting/rebasing image builds without breaking the cache of downstream Dockerfile commands. It's a niche feature I've only used a few times, but once you understand its advantages, it may significantly speed up some builds where you often need to inject a new external dependency early in the Dockerfile that won't affect the remaining steps in the build. I would describe this feature as "side-load some files into a new image without rebuilding the whole thing."
• v1.4 added Heredocs support for long RUN chained commands to save keystrokes and make them more readable. I've not used this as much as I should.
• v1.2 added the popular RUN --mount for injecting secrets, ssh, and caches into the building environment.