Skaffold: Cryptic error "manifest digest: [...] does not match Docker-Content-Digest: [...]" for base images

[ahmetb]

I just instrumented https://github.com/dockersamples/example-voting-app with skaffold.yaml and I'm seeing these logs from skaffold dev:

$ skaffold dev -f skaffold.yaml
2018/05/02 01:07:24 No matching credentials found for index.docker.io, falling back on anoynmous
2018/05/02 01:07:25 manifest digest: "sha256:ee8b362b5afc1e9db00b79c4db7d7744986507a8ba1d88f53e531e46e93454ec" does not match Docker-Content-Digest: "sha256:a346d13efede199fe161e5146acf080a205256431d82cc26f845eeb9c47b7bf7" for "index.docker.io/library/python:2.7-alpine"
2018/05/02 01:07:25 No matching credentials found for index.docker.io, falling back on anoynmous
2018/05/02 01:07:26 manifest digest: "sha256:01528b7048c72d7a36f4c216ce1b2c60751308b27538c140f12e0b10ba392246" does not match Docker-Content-Digest: "sha256:f4ea9cdf980bb9512523a3fb88e30f2b83cce4b0cddd2972bc36685461081e2f" for "index.docker.io/microsoft/dotnet:2.0.0-sdk"
2018/05/02 01:07:26 No matching credentials found for index.docker.io, falling back on anoynmous
2018/05/02 01:07:27 manifest digest: "sha256:14b627a91c92566d489d9d9073e465563be0e0c598c9537aa32e871a812018f5" does not match Docker-Content-Digest: "sha256:6bb963d58da845cf66a22bc5a48bb8c686f91d30240f0798feb0d61a2832fc46" for "index.docker.io/library/node:8.9-alpine"

Information

Skaffold version: v0.5.0
Operating system: macOS
Content of skaffold.yaml:

apiVersion: skaffold/v1alpha2
kind: Config
build:
  artifacts:
  - imageName: vote
    workspace: voting-app/vote/
  - imageName: worker
    workspace: voting-app/worker/
  - imageName: result
    workspace: voting-app/result/
deploy:
  kubectl:
    manifests:
    - ./voting-app/k8s-specifications/*

Repro steps

1. git clone https://github.com/dockersamples/example-voting-app.git voting-app
2. save skaffold.yaml listed above
3. run skaffold dev -f skaffold.yaml

[jonjohnsonjr]

Yeah :-/

That's going to be there until either:

1. Docker fixes https://github.com/docker/distribution/issues/2395, or
2. I fix https://github.com/google/go-containerregistry/issues/119.

We (google/go-containerregistry) are currently relying on the manifest list -> schema 2 backwards compatibility translation purely out of laziness. I intend to fix that, but I'm reluctant to remove this warning in the interim given that it would defeat the entire purpose of Docker-Content-Digest. IMO, this should be a fatal error, but it would break all of the official images...

Also, y'know, PR's welcome ;-)

[priyawadhwa]

Fixed by updating go-containerregistry.