Official-images: No DCT signing for recently published images?

[davejhilton]

I apologize in advance if this isn't the right place for this...

I'm seeing that a majority (if not all) of the "official images" have recently published tags that are not being signed.

For example, the php and golang images—which have always had new tags published as signed (to my knowledge) in the past—have recently stopped being signed when published.

$ docker pull --disable-content-trust=false php:7.1.27 # successfully pulls
$ docker pull --disable-content-trust=false php:7.1.28 # fails: No valid trust data for 7.1.28
$ docker pull --disable-content-trust=false php:7.1.29 # fails: No valid trust data for 7.1.29
$ docker pull --disable-content-trust=false php:7.1    # successfully pulls the same digest as the signed 7.1.27 tag

$ docker pull --disable-content-trust=false golang:1.12.1 # successfully pulls
$ docker pull --disable-content-trust=false golang:1.12.2 # fails: No valid trust data for 1.12.2
$ docker pull --disable-content-trust=false golang:1.12.3 # fails: No valid trust data for 1.12.3
$ docker pull --disable-content-trust=false golang:1.12.4 # fails: No valid trust data for 1.12.4
$ docker pull --disable-content-trust=false golang:1.12.5 # fails: No valid trust data for 1.12.5
$ docker pull --disable-content-trust=false golang:1.12   # successfully pulls the same digest as the signed 1.12.1 tag
$ docker pull --disable-content-trust=false golang:1      # successfully pulls the same digest as the signed 1.12.1 tag
$ docker pull --disable-content-trust=false golang:latest # successfully pulls the same digest as the signed 1.12.1 tag

Digging a little further... I see that, in addition to the latest SIGNED golang:1.12.1 image, there is a also a more recently-published UNSIGNED version of the same tag as well:

####### SIGNED image: #######
$ docker pull golang:1.12.1 --disable-content-trust=false
$ docker inspect golang:1.12.1 --format="{{index .RepoDigests 0}}"
golang@sha256:ea29c32b677885c58e03a3b27ee2f5db5f65f8a57c4ab73676d5ef07ff9deca6
$ docker inspect golang:1.12.1 --format="{{.Created}}"
2019-03-15T20:25:02.144222262Z



####### UNSIGNED image: #######
$ docker pull golang:1.12.1
$ docker inspect golang:1.12.1 --format="{{index .RepoDigests 0}}"
golang@sha256:62538d25400afa368551fdeebbeed63f37a388327037438199cdf60b7f465639
$ docker inspect golang:1.12.1 --format="{{.Created}}"
2019-03-27T04:40:59.136612521Z

I admittedly don't know much about the docker image publishing process used here, or how new tags get signed... but did something change with those processes between March 15th and March 27th (when those two tags were published)?

I've observed this same pattern of a sudden drop-off in signed tags for the php, golang, alpine, node, busybox, and several other of the official images I've checked, all potentially within that same time window.

Was this the result of a known, intentional change to the publishing process for these official images? Or can it be expected that at some future point, DCT signed versions of these image tags will be published?

Thanks!

[tianon]

Doh, this isn't great -- thanks for pinging us. The current process to my understanding is still what's discussed over in #1516, so I'll see if I can find some Docker Inc folks who can help us look into what's going on. :heart:

[davejhilton]

Thanks! 🙏

[davejhilton]

By the way... do you have suggestions about where else I could have filed this issue, to have it be seen by the correct set of Docker Inc folks directly, and avoid using this project as the middle-man? I'm pretty ignorant about the structure and relationships between all the docker-related groups/orgs/hierarchies 😅

[justincormack]

This should be fixed very soon, sorry for delay.

@davejhilton you can contact hub support, through the form on https://success.docker.com/support or you can email [email protected] if its security related.

[rhuddleston]

@justincormack is this an ongoing issue or is this fixed now? For example the sentry image isn't signed https://github.com/getsentry/docker-sentry/issues/173 can it be corrected now or what all is involved in fixing already published versions that were not signed?

[ahamlinman]

I was just able to pull signed versions of Rust 1.34 and 1.35. @rhuddleston It looks like the latest Sentry image is signed too.

I can't speak for all images, but if you've been experiencing this it's probably worth trying again now.

[tianon]

Awesome, thanks all (especially those on the Docker side who got this fixed 👍❤️)!

[manishtomar]

This has been fixed. I believe most of the images that were not signed are all signed. Please let us know if there is still any image that is not signed.

[tianon]

Awesome, thanks for confirming here @manishtomar! :+1: :heart:

[dpn]

@manishtomar Node is broken as of almost a month ago (See above linked issue)-- any chance this will be fixed soon?

And any chance there's a way to get this process fixed internally so this doesn't happen again?

[abeluck]

All node images using buster as a base are not signed as of 2019-08-29

10.16.3-buster
10.16-buster
10-buster
dubnium-buster
lts-buster
10.16.3-buster-slim
10.16-buster-slim
10-buster-slim
dubnium-buster-slim
lts-buster-slim

Edit: after more investigation, even more images are not signed. The latest stretch releases are also not signed. This wasn't apparent at first because docker pull --disable-content-trust=false node:lts-stretch worked, however when you inspect the version of node inside the images, it is an older 10.15 version, not the purported 10.16 as it should be.

[abeluck]

20 days later. The node image still has no valid trust signature. What's going on? @manishtomar

I've emailed [email protected] but haven't received any updates after an initial "looking into it" message.

@tianon perhaps you could ping some contacts to get this issue visibility?

[mermaid]

I too am having issues with node images not being singed

docker pull --disable-content-trust=false  node:12.4.0-alpine
No valid trust data for 12.4.0-alpine
docker pull --disable-content-trust=false  node:12.8.0-alpine
No valid trust data for 12.8.0-alpine
docker pull --disable-content-trust=false node:12-alpine
No valid trust data for 12-alpine
docker pull --disable-content-trust=false node:12.11.0
No valid trust data for 12.11.0

Any change we can get these signed?