Why you should pin your docker images with SHA instead of tags

Docker is an amazing tool. I’ve been utilising it from development all the way through to production and it made my life so much easier. Doesn’t matter how big or small your company is, there’s a good chance you can make use of containers in some shape or form.

Docker has gained a lot of popularity over the last 6 years. More and more companies are adopting it and are building cool technologies on top of it.

As with any technology that’s rapidly adopted, there’s going to be a widespread difference in emerging practices amongst different companies. One such practice is versioning.

Docker offers a built-in versioning technique through tagging. A typical docker image will take on the form of image_name:tag. The default tag is called latest, so for example if you want to grab the latest ubuntu image you can just docker pull ubuntu and docker will infer that you’re looking to pull ubuntu:latest. You can also specify tags if you want to grab a specific release of ubuntu. For example, by issuing docker pull ubuntu:18.04 you’re instructing docker to grab the latest available upstream version of Ubuntu Bionic.

Each image is also associated with a digest SHA — very much like a git commit. By specifying the SHA instead of the version tag, you can pull a specific version of the image. Something like docker pull ubuntu@sha256:sha_digest_here.

So why is this important?

Let’s take a docker-based CI/CD system for example. Imagine that your build pipeline is using docker containers to build/test/deploy your application. What this means is that your build/test/deploy commands will be running inside a docker container. Ideally you’re using very similar docker containers for building and running your application. For the sake of simplicity, let’s consider you’re always pulling the ubuntu:18.04 image to launch your build container off of. Something like

build:
  image: ubuntu:18.04
  commands:
    - ./build_my_wonderful_app.sh

—

It’s your regular Monday morning and you have a couple of changes that you’d want to push through. You create your PR and wait for the build to get green… but it doesn’t. You see some weird error that you’ve never seen before but you go through the usual hoops to troubleshoot.

Is it an actual code issue? Is it a flaky test? Is it a poorly-written dependency that does not pin transitive dependency versions correctly? Is it a bit flip caused by a cosmic ray?

Several hours down the drain you’re pulling your hair out and start contemplating your life choices. Then it hits you — it must be your build container. Sure enough, you go to the official image’s github page (hoping it exists) only to see there was a new commit merged back into ubuntu:18.04.

You go back to your old build logs and compare the container’s SHA to the SHA in your new build and sigh in a disdainful relief; that’s it.

So what do you do? Well, you either update your build process to work with the new version or you can pin your build image using the container SHA and create a ticket to solve this correctly later. After all, the change might be a legit cause for concern — such as a critical CVE or bugfix — but I’ll leave that decision up to you.

—

I think as developers we are used to seeing versions as “that’s the exact thing I need, so I’m safe to use that”, however this way of looking at docker tags is not quite correct. Long time support for certain base images is expected and so changes will be introduced over time.

All in all, if you want to save yourself a headache, just pin your image to a SHA version like so:

build:
  image: ubuntu@sha256:3235326357dfb65f1781dbc4df3b834546d8bf914e82cce58e6e6b676e23ce8f
  commands:
    - build_my_wonderful_app.sh

You can find the SHA by either pulling the image (it shows the digest once the image is pulled) or by inspecting the image.

—

As with everything on the internet, take this article with a grain of salt. By pinning the container version to a specific SHA you’re trading off “avoiding potential build failures” for “getting potential security patches for free”. I personally prefer controlling these upgrades myself, but it’s completely valid if you don’t.