Official-images: Guidance on GPG best practices

[jsternberg]

I currently maintain the InfluxData images for chronograf, kapacitor, influxdb, and telegraf. When we release a new version, we go through a process where all of these images get built usually multiple times before a pull request is eventually created.

Unfortunately, we keep running into an issue with the GPG keys and I'm not sure what to do. It usually delays the upload by about a day while I wait for the GPG keyservers to sort themselves out.

We use the following run line in every image to retrieve the GPG key and we will usually get this at least once during our regular build:

RUN set -ex && \
    for key in \
        05CE15085FC09D18E99EFB22684A14CF2582E0C5 ; \
    do \
        gpg --keyserver ha.pool.sks-keyservers.net --recv-keys "$key" || \
        gpg --keyserver pgp.mit.edu --recv-keys "$key" || \
        gpg --keyserver keyserver.pgp.com --recv-keys "$key" ; \
    done

+ gpg --keyserver ha.pool.sks-keyservers.net --recv-keys 05CE15085FC09D18E99EFB22684A14CF2582E0C5
gpg: directory '/root/.gnupg' created
gpg: new configuration file '/root/.gnupg/dirmngr.conf' created
gpg: new configuration file '/root/.gnupg/gpg.conf' created
gpg: keybox '/root/.gnupg/pubring.kbx' created
gpg: keyserver receive failed: Address not available
+ gpg --keyserver pgp.mit.edu --recv-keys 05CE15085FC09D18E99EFB22684A14CF2582E0C5
gpg: keyserver receive failed: No data
+ gpg --keyserver keyserver.pgp.com --recv-keys 05CE15085FC09D18E99EFB22684A14CF2582E0C5
gpg: keyserver receive failed: Host is unreachable

Do you have any advice or best practices for GPG keys? Whatever we are doing, it isn't working well for us. I've seen some reports on issues that switching to an ipv4-only GPG keyserver works, but I also found another comment here that says it's a red herring and doesn't really help.

Thanks for any help.

[yosifkit]

TLDR; GPG servers are extremely flaky and we have yet to solve it reliably.

For automated testing, we usually just set our travis builds to repeat multiple times (like php) since it is usually just gpg failures.

For builds on my local machine (which is were most build tests for official image PRs are done) I run an nginx reverse proxy server to man-in-the-middle apt, apk, and gpg requests and cache them.

For the servers that build and push the official images we take an eventually consistent approach and mark builds as "unstable" if one or more tags fail to build and just allow the build to run later. They also have multiple retries for each tag.

Related issues: https://github.com/docker-library/cassandra/pull/131#issuecomment-358444537, https://github.com/docker-library/tomcat/issues/87, https://github.com/docker-library/tomcat/pull/108, https://github.com/docker-library/mysql/issues/263#issuecomment-354025886, https://github.com/docker-library/httpd/issues/66#issuecomment-316832441, https://github.com/docker-library/php/issues/586, https://github.com/docker-library/wordpress/pull/291

[chorrell]

With the node image, we had a lot of issues with the GPG servers, particularly when building under Alpine. We ended up with the following gpg servers which seem to be working well:

; do \
    gpg --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys "$key" || \
    gpg --keyserver hkp://ipv4.pool.sks-keyservers.net --recv-keys "$key" || \
    gpg --keyserver hkp://pgp.mit.edu:80 --recv-keys "$key" ; \
  done

We ended up dropping keyserver.pgp.com entirely.

[chorrell]

For reference:

• https://github.com/nodejs/docker-node/issues/591
• https://github.com/nodejs/docker-node/pull/632

[yosifkit]

New initiative is to just use one gpg server and then use a proxy for travis builds (and now on the official build servers!). Example here: https://github.com/docker-library/php/pull/666.

[damianb]

Forgive me for asking, but has there been any consideration as to why it's de-facto to import keys one at a time, instead of wadding them together in one large bulk request to each keyserver?

For projects that have three or more keys to import, I would wager this is not helping the reliability of the keyservers. Perhaps a better solution is in order?

[yosifkit]

has there been any consideration as to why it's de-facto to import keys one at a time, instead of wadding them together in one large bulk request to each keyserver?

gpg doesn't do a "bulk" request, it still does one http GET per key. Most importantly, if you give it a list of keys and it fails to import one of them, it will not return a failing exit code.

[jasonmp85]

TLDR; GPG servers are extremely flaky and we have yet to solve it reliably

I don't know a _ton_ about the GPG ecosystem, but couldn't Docker run some sort of proxy or mirror for this purpose? Like, if the main place it's a PITA is building Docker images, maybe just we point our keyserver addresses to a Docker mirror?

[wglambert]

Well you can run a proxy in a Docker container for handling gpg requests, as we do when building the images https://github.com/docker-library/official-images/issues/4252#issuecomment-405749749

[tianon]

I don't know a _ton_ about the GPG ecosystem, but couldn't Docker run some sort of proxy or mirror for this purpose?

Indeed, as @wglambert has pointed out, this is exactly how we solve this problem on all our Travis builds and our official build infrastructure today: https://github.com/tianon/pgp-happy-eyeballs

There's a script there called "hack-my-builds" that is designed to be used exactly as in https://github.com/docker-library/php/pull/666, as in, adding the following to .travis.yml:

install:
  - wget -qO- 'https://github.com/tianon/pgp-happy-eyeballs/raw/master/hack-my-builds.sh' | bash

[tianon]

Just to thread the needle all the way through: https://github.com/docker-library/faq#openpgp--gnupg-keys-and-verification