Buildkit: ADD from url fails with 'invalid not-modified ETag'

[mcfedr]

Build fails at:

ADD https://getcomposer.org/composer.phar /usr/local/bin/composer

Log:

 => [1/17] FROM docker.io/library/php:7.3-fpm@sha256:cf8e94d24d94329f13bcd430ae586f80278247e1c43e5f8b3d52c4ab16d2464f                                                                                                         0.0s
 => ERROR https://getcomposer.org/composer.phar                                                                                                                                                                               0.3s
 => [internal] load build context                                                                                                                                                                                             0.0s
------
 > https://getcomposer.org/composer.phar:
------
invalid not-modified ETag: "5c912760-1d3e0e"

I'm wondering if this is related to https://github.com/moby/buildkit/pull/835 but thought worth reporting so it can also be tested.

The etag seems to work just fine, i.e. curl -I -H 'if-none-match: "5c912760-1d3e0e"' https://getcomposer.org/composer.phar returns 304.

[ahmetb]

Also seeing it from github releases, such as:

    ADD https://github.com/Yelp/dumb-init/releases/download/v1.2.2/dumb-init_1.2.2_amd64 /bin/dumb_init

[rwe]

I believe I understand the cause.

The fix should be very simple ~, however I'm not set up for building in this ecosystem. Hopefully someone can provide a patch~ . EDIT see (untested) PR #1159.

I was seeing this too with a Google Cloud Storage URL and so decided to investigate. It seems unlikely that both GitHub releases and Google Cloud Storage are "misbehaving" with respect to ETags, and it's likely that instead BuildKit may be misinterpreting the If-None-Match spec.

Background info

Docker version info

Docker for Mac 2.1.1.0 (27260)

Client:
 Debug Mode: false
 Plugins:
  app: Docker Application (Docker Inc., v0.8.0)
  buildx: Build with BuildKit (Docker Inc., v0.2.2-10-g3f18b65-tp-docker)

Server:
 Containers: 15
  Running: 0
  Paused: 0
  Stopped: 15
 Images: 2126
 Server Version: 19.03.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.14.131-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 1.945GiB
 Name: docker-desktop
 ID: WESG:FJU6:DS3Q:6OS3:D7SX:VK5N:Q4UH:A5BS:T46V:2MJH:YKWB:XQPX
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 29
  Goroutines: 44
  System Time: 2019-09-04T19:03:23.537795672Z
  EventsListeners: 2
 HTTP Proxy: gateway.docker.internal:3128
 HTTPS Proxy: gateway.docker.internal:3129
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

Reproduction Steps

### `Dockerfile`

FROM scratch
ADD https://storage.googleapis.com/gpt-2/models/774M/checkpoint

The first run (or a run after the cache is cleared) succeeds:

DOCKER_BUILDKIT=1 docker build .

> ``` > [+] Building 0.3s (5/5) FINISHED > => [internal] load .dockerignore 0.0s > => => transferring context: 244B 0.0s > => [internal] load build definition from Dockerfile 0.0s > => => transferring dockerfile: 121B 0.0s > => https://storage.googleapis.com/gpt-2/models/774M/checkpoint 0.0s > => [1/1] ADD https://storage.googleapis.com/gpt-2/models/774M/checkpoint . 0.0s > => exporting to image 0.0s > => => exporting layers 0.0s > => => writing image sha256:9a630453cc77705a4f57c121393a38a4f45f65eba6155ba6c3f27ecab18e2b05 0.0s > ``` Running the same command again: > ``` > [+] Building 0.2s (3/4) > => [internal] load build definition from Dockerfile 0.0s > => => transferring dockerfile: 36B 0.0s > => [internal] load .dockerignore 0.0s > => => transferring context: 35B 0.0s > => ERROR https://storage.googleapis.com/gpt-2/models/774M/checkpoint 0.1s > ------ > > https://storage.googleapis.com/gpt-2/models/774M/checkpoint: > ------ > invalid not-modified ETag: > ``` Clearing the cache with `docker builder prune` resets the state.

Cause

If a single ETag is requested in If-None-Match, the server may not include that (unambiguous) ETag header in the response.

Detailed demonstration

Requesting the file directly:

curl --http1.1 -s -L -D /dev/stderr -o /dev/null https://storage.googleapis.com/gpt-2/models/774M/checkpoint

Response headers. Notice the ETag. > ```http > HTTP/1.1 200 OK > X-GUploader-UploadID: AEnB2Up6PhdYRb_UZ18VAl30f6XLzFkOoBPnSYSjSKqzk90Go8Zqk-zZoenkbL3inKQz1ozoLjcObKKuIbvOV7XFlSSFO6aW0Q > Expires: Wed, 04 Sep 2019 19:23:16 GMT > Date: Wed, 04 Sep 2019 19:23:16 GMT > Cache-Control: private, max-age=0 > Last-Modified: Tue, 20 Aug 2019 15:50:08 GMT > ETag: "ca0368fcd3c4c1a99aca42511d0c1f12" > x-goog-generation: 1566316208157027 > x-goog-metageneration: 1 > x-goog-stored-content-encoding: identity > x-goog-stored-content-length: 77 > Content-Type: application/octet-stream > x-goog-hash: crc32c=BI0EFw== > x-goog-hash: md5=ygNo/NPEwamaykJRHQwfEg== > x-goog-storage-class: MULTI_REGIONAL > Accept-Ranges: bytes > Content-Length: 77 > Server: UploadServer > Alt-Svc: quic=":443"; ma=2592000; v="46,43,39" > ``` Requesting the file `If-None-Match` with that ETag:

curl --http1.1 -s -L -D /dev/stderr -o /dev/null -H 'If-None-Match: "ca0368fcd3c4c1a99aca42511d0c1f12"' https://storage.googleapis.com/gpt-2/models/774M/checkpoint

Notice that the ETag is *not* included in the response headers. > ```http > HTTP/1.1 304 Not Modified > X-GUploader-UploadID: AEnB2UpEqNrI5fDJjglD2f4--3CCzskMyUg-Fo1RZxoqqHq17HG8W_gURMO6uUVy9B6Mg4450GyA4yRTjPqEJY8v6dtxhuHcLQ > Expires: Wed, 04 Sep 2019 19:23:36 GMT > Date: Wed, 04 Sep 2019 19:23:36 GMT > Cache-Control: private, max-age=0 > Last-Modified: Tue, 20 Aug 2019 15:50:08 GMT > Content-Length: 0 > Server: UploadServer > Alt-Svc: quic=":443"; ma=2592000; v="46,43,39" > ``` Requesting the file with multiple ETags in `If-None-Match`:

curl --http1.1 -s -L -D /dev/stderr -o /dev/null -H 'If-None-Match: "ca0368fcd3c4c1a99aca42511d0c1f12", "foobar"' https://storage.googleapis.com/gpt-2/models/774M/checkpoint

Now the ETag is included to disambiguate. > ```http > HTTP/1.1 200 OK > X-GUploader-UploadID: AEnB2UpD_nKA4ZMNpJvC97lMJfyXjcr9myMAxojFypxW8lUNiEGiwdaOtezf74-OBCHDXn7T4Ru57oelrDHb01wY9IMU1Qdl6A > Expires: Wed, 04 Sep 2019 19:26:02 GMT > Date: Wed, 04 Sep 2019 19:26:02 GMT > Cache-Control: private, max-age=0 > Last-Modified: Tue, 20 Aug 2019 15:50:08 GMT > ETag: "ca0368fcd3c4c1a99aca42511d0c1f12" > x-goog-generation: 1566316208157027 > x-goog-metageneration: 1 > x-goog-stored-content-encoding: identity > x-goog-stored-content-length: 77 > Content-Type: application/octet-stream > x-goog-hash: crc32c=BI0EFw== > x-goog-hash: md5=ygNo/NPEwamaykJRHQwfEg== > x-goog-storage-class: MULTI_REGIONAL > Accept-Ranges: bytes > Content-Length: 77 > Server: UploadServer > Alt-Svc: quic=":443"; ma=2592000; v="46,43,39" > ```

Fix

The following lines of code should be updated so that, if only one ETag was requested, and no ETag header was returned in the response, the requested ETag is assumed:

• https://github.com/moby/buildkit/blob/bcd466a748e950507826c30e835b087289aaf384/source/http/httpsource.go#L174-L177
• https://github.com/moby/buildkit/blob/bcd466a748e950507826c30e835b087289aaf384/source/http/httpsource.go#L197-L203

[mcfedr]

Not that this should change the fix, as its clearly need for the real world cases, but, out of interest, RFC 7232 describing the 304 response appear does require the server to send back the ETag

The server generating a 304 response MUST generate any of the
following header fields that would have been sent in a 200 (OK)
response to the same request: Cache-Control, Content-Location, Date,
ETag, Expires, and Vary.

[deeky666]

Seems like this should have been fixed by #1159. Is the patch released already? I'm on 19.03.8 and still experiencing this issue when trying to add PHP Composer (as described above).

[delboy1978uk]

I was getting this issue, but I found doing a docker system prune meant that I could rebuild again. Maybe this will help someone.

[tobia]

I'm also still having this issue (with the same exact URL) with docker 20.10.5

Which version is this supposed to have been fixed in?