Docker-node: Tracking Docker Hub digests changes

Created on 5 May 2018  路  8Comments  路  Source: nodejs/docker-node

Hi,

I'm hoping that someone on the "devops" side of this repository can help me get to the bottom of a potential problem with Docker Hub's sha256 digests for node images.

As you're probably aware, the only way to get "immutable" builds with Docker is to use exact digests, so that is what I'm using. However, in the past ~1 month I've observed 8 different digests for the node:8.11.1 image:

2018-04-03 sha256:9657809c879d6f5064904c4f03fff3cb1b644f659a93ed798d31bd6a1e268c19
2018-04-04 sha256:a4c4106ddda19c2c228cbdfbc0f0d7a5f27c383b0486f88fc2c2c40153763cf5
2018-04-17 sha256:26e4c77f9f797c3993780943239fa79419f011dd93ae4e0097089e2145aeaa24
2018-04-28 sha256:7c3a236a4f4b61c1e390930225209ddc24d3d9a08a0dee97d13fff2df91c1cf7
2018-04-29 sha256:4fe84bf04ebb3da3ff5e6524db3dd5085eb3c61a85928b594924aae8529f376f
2018-05-03 sha256:b802a02ee6496c34e2dc0eb0379eb1c738331414956d650c0dffdfa0866acb2f
2018-05-03 sha256:9fef54193f605eb77fcb7eaae564e7cc9b3444a12d76a2e1c87b9789752852b3
2018-05-05 sha256:74133ece3418b6e307d3f9f63cf02ebc3ea1e76b2ff13037f5bf1d423d6a1cdd

This level of churn can be frustrating for users who (a) want immutable build, but (b) want to keep updated.

In this repo you can see however that the Dockerfile itself has not changed between 30th March until today (5th May): https://github.com/nodejs/docker-node/commits/master/8/Dockerfile

These digest values are found using the Docker Hub API, in this case: /v2/library/node/manifests/8.11.1 with headers.accept = 'application/vnd.docker.distribution.manifest.v2+json' and then looking at the docker-content-digest header of the response.

I'm suspecting one of these:

  1. Some kind of "continuous delivery" that is regularly rebuilding and updating the node images on Docker Hub when the base image
  2. My method for retrieving the digest value is wrong
  3. Docker Hub is at fault, changing digest even though the underlying image hasn't changed
meta question
Source
picture rarkins
👍1

Most helpful comment

Part of the problem is that you are tracking the manifest list (node:8.11.1) and so as builds on other arches finish, it shows up as a manifest change. You may want to switch to the arch specific image so that you don't get all the unrelated arch build updates (amd64/node).

You can see the full history of the sha256 manifests on https://github.com/docker-library/repo-info/tree/master/repos/node/remote, specifically the history of 8.11.1.md.

I have the repo locally and did a quick git log to see what changes are included in each of the digests that you observed:

picture yosifkit on 8 May 2018
👍2

All 8 comments

@tianon @yosifkit might be able to weigh in.

I know that the images are rebuilt when the base images changes (so security fixes in alpine are available without having to wait for a new node release). But I don't know any details, so hopefully they'll be able to provide a better answer! 馃檪

SimenB picture SimenB on 5 May 2018
👍1

Using a tool called docker-diff, I've compared the last two digests above:


diff

450a451
> -rw-r--r-- 60         etc/networks
480a482
> -rw-r--r-- 2932       etc/protocols
554a557
> -rw-r--r-- 887        etc/rpc
569a573
> -rw-r--r-- 19605      etc/services
4770c4774
< -rw-r--r-- 4987       usr/include/linux/spi/spidev.h
---
> -rw-r--r-- 5012       usr/include/linux/spi/spidev.h
4824c4828
< -rw-r--r-- 31393      usr/include/linux/usb/ch9.h
---
> -rw-r--r-- 31434      usr/include/linux/usb/ch9.h
5384c5388
< -rw-r--r-- 22125      usr/include/x86_64-linux-gnu/asm/msr-index.h
---
> -rw-r--r-- 22293      usr/include/x86_64-linux-gnu/asm/msr-index.h
19597c19601
< -rw-r--r-- 378013     usr/share/doc/linux-libc-dev/changelog.Debian.gz
---
> -rw-r--r-- 391243     usr/share/doc/linux-libc-dev/changelog.Debian.gz
19761a19766,19768
> drwxr-xr-x 0          usr/share/doc/netbase/
> -rw-r--r-- 7724       usr/share/doc/netbase/changelog.gz
> -rw-r--r-- 535        usr/share/doc/netbase/copyright
31201a31209,31213
> -rw-r--r-- 38         var/lib/dpkg/info/netbase.conffiles
> -rw-r--r-- 182        var/lib/dpkg/info/netbase.list
> -rw-r--r-- 135        var/lib/dpkg/info/netbase.md5sums
> -rwxr-xr-x 1563       var/lib/dpkg/info/netbase.postinst
> -rwxr-xr-x 756        var/lib/dpkg/info/netbase.postrm
31391,31392c31403,31404
< -rw-r--r-- 349126     var/lib/dpkg/status
< -rw-r--r-- 349126     var/lib/dpkg/status-old
---
> -rw-r--r-- 349669     var/lib/dpkg/status
> -rw-r--r-- 349669     var/lib/dpkg/status-old
31428,31429c31440,31441
< -rw-r--r-- 12989      var/log/apt/history.log
< -rw-r----- 66766      var/log/apt/term.log
---
> -rw-r--r-- 13011      var/log/apt/history.log
> -rw-r----- 66912      var/log/apt/term.log
31432c31444
< -rw-r--r-- 159497     var/log/dpkg.log
---
> -rw-r--r-- 160027     var/log/dpkg.log

So it looks like OS-level changes.

The build-deps/jessie Dockerfile is also unchanged for half a year: https://github.com/docker-library/buildpack-dockdeps/commits/d7da72aaf3bb93fecf5fcb7c6ff154cb0c55d1d1/jessie/Dockerfile

However, digging into the latest node:8.11.1 digest by cat'ing /var/log/apt/history.log indeed shows a recent date: End-Date: 2018-05-04 17:37:22

Doing the same command on buildpack-deps:jessie shows the same log file. So I'm guessing it's the base jessie image that's also getting continuous updates and then these are pulled through to node.

The next question is: is this ideal for node users?

First part: it's entirely reasonable and perhaps even recommended to pin node image digests. e.g. when the node images were broken for half a day earlier this year for yarn users, the only insurance against this was to have pinned digests.

Second part: Is it reasonable that a pinned version of Node.js (e.g. v8.11.1) gets 8 updates in a month? I understand it, but it also seems undesirable that we get such churn at the language/application level due to Operating system updates despite using containers.

rarkins picture rarkins on 5 May 2018

IMO it's entirely reasonable to get OS-level security updates, and one of the biggest advantages of official images over just installing node into stretch (or whatever) yourself.

But showing what changed in a simple way (throughout the tree) sounds like something we'd want.

SimenB picture SimenB on 5 May 2018

Security updates: agreed. But have there been 7 security updates to Debian Jessie in the last month or is something else going on? Difficult for node project to do much about it though if it鈥檚 an upstream buildpack

rarkins picture rarkins on 5 May 2018

Part of the problem is that you are tracking the manifest list (node:8.11.1) and so as builds on other arches finish, it shows up as a manifest change. You may want to switch to the arch specific image so that you don't get all the unrelated arch build updates (amd64/node).

You can see the full history of the sha256 manifests on https://github.com/docker-library/repo-info/tree/master/repos/node/remote, specifically the history of 8.11.1.md.

I have the repo locally and did a quick git log to see what changes are included in each of the digests that you observed:

yosifkit picture yosifkit on 8 May 2018
👍2

@yosifkit thanks for the wonderful explanation.

I wasn't aware that cross-platform builds ultimately resulted in digest changes, so switching from library/node lookups to amd64/node ones should immediately help reduce the noise. Can you tell me if every - or practically every - official library/x image has a corresponding amd64/x twin, or would I need to query the Hub API to check against the list of architectures to be sure?

rarkins picture rarkins on 8 May 2018

Yes, all but one library/ image is pushed to amd64/ and that is because it only supports s390x. The available tags between an arch and library/ should be fairly close in most instances (minus things like windows images). The images are actually pushed first to the architecture specific namespaces (amd64, arm32v5, arm32v6, arm32v7, arm64v8, i386, ppc64le, s390x, winamd64) and then the manifest list is created in the library/ from those images. You can see the list of arches with links on the Docker Hub description under "Supported architectures".

yosifkit picture yosifkit on 8 May 2018

@yosifkit thank you again for your help. To wrap it up, I conclude:

(a) reduce some digest churn by pulling digests from amd64/node instead of library/node
(b) remaining digest churn is due to genuine security updates

rarkins picture rarkins on 8 May 2018
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

sam-github picture sam-github  路  4Comments
eyaylagul picture eyaylagul  路  3Comments
kmetsalu picture kmetsalu  路  5Comments
tigrus picture tigrus  路  4Comments
jtcmedia picture jtcmedia  路  5Comments