Skaffold: Problems with private docker images on quay.io

[sdwolfz]

Setup:

• OS:

Darwin uk-c02vl0d9hv2j 18.7.0 Darwin Kernel Version 18.7.0: Thu Jun 20 18:42:21 PDT 2019; root:xnu-4903.270.47~4/RELEASE_X86_64 x86_64 i386 MacBookPro14,1 Darwin

• minikube:

minikube version: v1.5.2 
commit: 792dbf92a1de583fcee76f8791cff12e0c9440ad

• kubectl:

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-14T04:24:34Z", GoVersion:"go1.12.13", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:09:08Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

• docker:

Docker version 19.03.5, build 633a0ea
Docker for mac: 2.1.0.5 (40693)

• skaffold: 1.0.1 and v1.0.0-144-g760edb47-dirty (built from master 760edb4703f764877fcbd8a9dec36abe19f33da4)
  

  Steps:

1. Create a fresh minikube vm:

minikube delete
minikube start

Note: you can also use the Docker for Mac kubernetes functionality, it does not change the outcome.

1. Authenticate to quay.io in docker and kubernetes:

docker login -u "<<USER>>" -p "<<TOKEN>>" quay.io
kubectl create secret docker-registry regcred \
  --docker-server=quay.io                     \
  --docker-username="<<USER>>"                \
  --docker-password="<<TOKEN>>"
kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "regcred"}]}'

1. Create the following files:
2. skaffold.yml:

apiVersion: skaffold/v1
kind: Config
build:
  artifacts:
  - image: my-app
  local:
    push: false
    useBuildkit: true
deploy:
  kubectl:
    manifests:
      - ./pod.yml

• pod.yml:

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
    - name: my-app
      image: my-app

• Dockerfile:

FROM quay.io/some-org/some-private-image:version
WORKDIR /work
COPY ./ /work

Note: make sure you use a private image with this.

1. Run skaffold dev

Expected:

The image is build and pushed to minikube, a pod is started and running.

Observed:

Listing files to watch...
 - my-app
FATA[0001] watching files for artifact remote-service: listing files: listing files: expanding ONBUILD instructions: parsing ONBUILD instructions: processing base image (quay.io/some-org/some-private-image:version) for ONBUILD triggers: getting remote config: getting image: unsupported MediaType: "application/vnd.docker.distribution.manifest.v1+prettyjws", see https://github.com/google/go-containerregistry/issues/377

Note that running: skaffold run or skaffold debug works properly, the issue is only with skaffold dev. Also if you do a run or debug before a dev and it downloads the image successfully, the problem goes away.

Since debug is basically dev with the following extra config:

    if len(opts.TargetImages) == 0 {
        opts.TargetImages = []string{"none"}
    }
    opts.PortForward.ForwardPods = true
    deploy.AddManifestTransform(debugging.ApplyDebuggingTransforms)

I'm assuming the problem is related to target images, and from the log and looking through the code what I understand is that it tries to parse the remote image in order to determine which files to watch, but does not have access to the image to do that (unless the image has already been pulled).

Also, the error message could be better, listing an authentication error instead of a mime type related one.

Other

I would like an option to disable artifact caching and file watching from the skaffold.yml, or an option to have port forwarding enabled from a run command. I could not find a reference to something like this in the docs.

[balopat]

Thank you for the super detailed issue @sdwolfz!

Also, the error message could be better, listing an authentication error instead of a mime type related one.

Hmm...if this is an auth error it is really a weird error message. Especially that the https://github.com/google/go-containerregistry/issues/377 was fixed in April and we are on a December version of go-containerregistry (unless it's a regression).

Can you try to run eval minikube docker-env _before_ running docker login -u "<<USER>>" -p "<<TOKEN>>" quay.io? Skaffold uses the docker daemon inside minikube (or Docker for desktop).

I would like an option to disable artifact caching and file watching from the skaffold.yml, or an option to have port forwarding enabled from a run command. I could not find a reference to something like this in the docs.

Portforwarding is coming to run: https://github.com/GoogleContainerTools/skaffold/pull/3263

[sdwolfz]

@balopat Same error when running eval $(minikube docker-env) before docker login.
Running skaffold dev -w none starts pulling the image, while omitting -w none gives:

FATA[0001] watching files for artifact remote-service: listing files: listing files: expanding ONBUILD instructions: parsing ONBUILD instructions: processing base image (quay.io/some-org/some-private-image:version) for ONBUILD triggers: getting remote config: getting image: unsupported MediaType: "application/vnd.docker.distribution.manifest.v1+prettyjws", see https://github.com/google/go-containerregistry/issues/377

[dgageot]

Anyone can confirm that this is still an issue?

[sdwolfz]

Anyone can confirm that this is still an issue?

I'll try to re-run it and get back to you within a few days.

[sdwolfz]

@dgageot I can confirm the issue still occurs exactly as described above.

[chanseokoh]

getting remote config: getting image: unsupported MediaType: "application/vnd.docker.distribution.manifest.v1+prettyjws", see https://github.com/google/go-containerregistry/issues/377

I believe go-containerregistry never supported the deprecated image manifest v2 schema 1. The next major Docker version won't work on schema 1 images either; you need to make your private repo (remote-service) return a schema 2 manifest for the images inside. I believe Quay.io recently added and enabled schema 2 support globally. Maybe the solution would be as easy as re-pushing (after re-building the image if necessary) the image as explained in the deprecated notice.

Remember the schema v1 is ancient and by now you should consider it dead. The Docker CLI currently returns this warning:
[DEPRECATION NOTICE] registry v2 schema1 support will be removed in an upcoming release. Please contact admins of the localhost:5000 registry NOW to avoid future disruption. More information at https://docs.docker.com/registry/spec/deprecated-schema-v1/

$ docker push localhost:5000/ubuntu:tag2
The push refers to repository [localhost:5000/ubuntu]
e0b3afb09dc3: Layer already exists 
6c01b5a53aac: Layer already exists 
2c6ac8e5063e: Layer already exists 
cc967c529ced: Layer already exists 
[DEPRECATION NOTICE] registry v2 schema1 support will be removed in an upcoming release. Please contact admins of the localhost:5000 registry NOW to avoid future disruption. More information at https://docs.docker.com/registry/spec/deprecated-schema-v1/
tag2: digest: sha256:c1719faa7caaca0273424beb7e316d474cc072193046e186a396f17edf034ebf size: 4925

Once @sdwolfz confirms the issue is gone after upgrading the image (or re-configure Quary.io) to return a schema 2 manifest, probably we can close this issue.

[sdwolfz]

@chanseokoh Thank you for the details. I'll give it a go and see what happens.

[sdwolfz]

@chanseokoh you are right, the problem is CircleCI uses an ancient version of docker by default: 17.09.0-ce: https://circleci.com/docs/2.0/building-docker-images/ which pushed images with the old schema. Works well after updating the docker version or re-pushing.

[tejal29]

I am seeing this error again for a sample project i am running with this dockerfile

FROM library/ruby:2.3.0

# Install essential Linux packages
RUN apt-get update -qq && apt-get install -y \
    build-essential \
    nodejs

WORKDIR /app

# Copy Gemfile and Gemfile.lock
COPY Gemfile /app/

# Speed up nokogiri install
ENV NOKOGIRI_USE_SYSTEM_LIBRARIES 1

RUN gem install bundler
RUN bundle install

COPY . /app
CMD bundle exec dashing start

error:

➜  github-dashing git:(docker-support) ✗ skaffold dev -d gcr.io/tejal-test
Listing files to watch...
 - github-dash
watching files for artifact "github-dash": listing files: parsing ONBUILD instructions: retrieving image "library/ruby:2.3.0": unsupported MediaType: "application/vnd.docker.distribution.manifest.v1+prettyjws", see https://github.com/google/go-containerregistry/issues/377

This is also reason where users see proto.StatusCode_DEVINIT_REGISTER_BUILD_DEP /cc @briandealwis and @nelango

[sdwolfz]

FROM library/ruby:2.3.0

:point_up: may be the old ruby image? assuming that image was built a long time ago. Anyway, my issue went away when upgrading the docker version in out CircleCI jobs, so it might be a different issue you are facing, with the same symptoms. Might I suggest you open a new issue instead, and close this one again? I'll be unsubscribing from this thread as my problem was fixed.

[chanseokoh]

ruby:2.3.0 seems to be an old image pushed with the deprecated manifest v2 schema 1. Apparently Docker Hub doesn't auto-convert it to schema 2.

$ docker manifest inspect library/ruby:2.3.0
unsupported manifest media type and no default available: application/vnd.docker.distribution.manifest.v1+prettyjws

[tejal29]

@chanseokoh we have a a Pr in progress #4896 to not error but ask users to manually docker pull and proceed.

[briandealwis]

Closing this: go-containerregistry will not support these old manifests, and using a more modern docker to build and push images solves this issue.