Minikube: cache with quay.io issues: Docker-Content-Digest and 405 Method Not Allowed

Created on 28 Mar 2019 · 19Comments · Source: kubernetes/minikube

Hello,

With minikube 1.0.0 on windows, I am not able to cache images from quay.io docker repository.

$ minikube cache add quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0
2019/03/28 14:30:05 No matching credentials were found, falling back on anonymous

!   Failed to cache and load images: caching images: caching image C:\Programs\Cygwin\home\cvila\.minikube\cache\images\quay.io\kubernetes-ingress-controller\nginx-ingress-controller_0.23.0: manifest digest: "sha256:603b7018c941b1117da4eb980c8043992ad37abdde3636fcbae1f34bcfff443f" does not match Docker-Content-Digest: "sha256:e1292564ba5f1fd75005a4575778523d3309fb5d5d57f6597234c0b1567641f6" for "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0"

*   Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
-   https://github.com/kubernetes/minikube/issues/new

From what I understood, it comes from https://github.com/google/go-containerregistry/issues/119, which has been recently fixed (beg March) by https://github.com/google/go-containerregistry/pull/387. Maybe we just need to make sure we compile minikube with the latest release of go-containerregistry ?

Side question, is there a way to avoid the No matching credentials were found, falling back on anonymous, when we put in cache a public image (which does not require any credentials to be passed)

Thanks !

help wanted kinbug prioritbacklog 2019q2

Source

cvila84

👍2

All 19 comments

This would be fixed with PR #3899, which updates google/go-containerregistry (for another reason)

afbjorklund on 28 Mar 2019

Hmm, or maybe not. Are you sure that update fixes this issue ?

💣  Failed to cache and load images: caching images: caching image /home/anders/.minikube/cache/images/quay.io/kubernetes-ingress-controller/nginx-ingress-controller_0.23.0: unsupported status code 405; body: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>

It works OK with docker pull, so I guess the issue is somewhere in google/go-containerregistry ...

afbjorklund on 28 Mar 2019

Upgrading google/go-containerregistry to master (note: it doesn't have any releases) gives:

💣 Failed to cache and load images: caching images: caching image /home/anders/.minikube/cache/images/quay.io/kubernetes-ingress-controller/nginx-ingress-controller_0.23.0: fetching remote image: unsupported MediaType: https://github.com/google/go-containerregistry/issues/377

So it looks like https://github.com/google/go-containerregistry/issues/377 (problem with quay.io images)

Schema 1 images are generally super ancient and will have tons of unaddressed vulnerabilities. The exception to this is quay.io, which for reasons I cannot understand didn't implement schema 2 support until very recently. Poking around, most images are still schema1 so it seems not to be completely rolled out yet?

afbjorklund on 28 Mar 2019

@cvila84 : unfortunately there seems to be no away to avoid the spam from google/gcr:

        if auth == authn.Anonymous {
            log.Println("No matching credentials were found, falling back on anonymous")
        }

afbjorklund on 28 Mar 2019

@afbjorklund you are right, the root cause does not come from the issue #119 (according to this comment)

I guess we have to wait for compatible images pushed on quay.io but they do not seem to be worried about that (since they recently pushed images with old format)

For the spam, I saw there is already issue 4007 (no link made on purpose) that covers it

cvila84 on 29 Mar 2019

It's a bug in the library IMHO, waiting for external parties to upgrade doesn't seem like a good approach

afbjorklund on 30 Mar 2019

Agree.

But the guy closed the issue and in this comment, he is quite clear :

I guess I'm reluctant to enable people to continue to depend on schema 1 images, because people should basically never be depending on schema 1 images, except for quay, which I think is being fixed.

So he is relying on quay.io to ensure next images pushed on this repo will be compatible with schema 2...

cvila84 on 30 Mar 2019

There are other image libraries, that might work better... We don't _have_ to use google/gcr.
We moved away* from containers/image, but could consider going back if it works better...

in #2730

afbjorklund on 30 Mar 2019

This is not an issue specific to v1.0.0 as I notice the same issue with v0.35.0.

aakarshg on 1 Apr 2019

I believe this issue was fixed in minikube v1.1. That said, the example still fails:

minikube cache add quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0 --alsologtostderr -v=8
I0523 16:21:54.215412   80236 notify.go:128] Checking for updates...
I0523 16:21:54.418000   80236 cache_images.go:307] Attempting to cache image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0 at /usr/local/google/home/tstromberg/.minikube/cache/images/quay.io/kubernetes-ingress-controller/nginx-ingress-controller_0.23.0
I0523 16:21:54.873611   80236 cache_images.go:331] OPENING:  /usr/local/google/home/tstromberg/.minikube/cache/images/quay.io/kubernetes-ingress-controller/nginx-ingress-controller_0.23.0


I0523 16:24:24.189035   80236 cache_images.go:304] 2019/05/23 16:21:54 Unable to read "/usr/local/google/home/tstromberg/.docker/config.json": open /usr/local/google/home/tstromberg/.docker/config.json: no such file or directory
2019/05/23 16:21:54 No matching credentials were found, falling back on anonymous
W0523 16:24:24.189239   80236 exit.go:100] Failed to cache and load images: caching images: caching image /usr/local/google/home/tstromberg/.minikube/cache/images/quay.io/kubernetes-ingress-controller/nginx-ingress-controller_0.23.0: unsupported status code 405; body: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>

💣  Failed to cache and load images: caching images: caching image /usr/local/google/home/tstromberg/.minikube/cache/images/quay.io/kubernetes-ingress-controller/nginx-ingress-controller_0.23.0: unsupported status code 405; body: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>

However, docker pull works just fine.

tstromberg on 24 May 2019

I am also getting the issue on minikube 1.2.0:

~ minikube cache add quay.io/dexidp/dex:v2.14.0

💣  Failed to cache and load images: caching images: caching image /Users/csreegn/.minikube/cache/images/quay.io/dexidp/dex_v2.14.0: unsupported status code 405; body: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>


😿  Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
👉  https://github.com/kubernetes/minikube/issues/new

reegnz on 28 Jun 2019

Are there any updates on this issue? I'm getting the same error:

minikube cache add quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0

💣  Failed to cache and load images: caching images: caching image /Users/xyz/.minikube/cache/images/quay.io/kubernetes-ingress-controller/nginx-ingress-controller_0.23.0: unsupported status code 405; body: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>


😿  Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
👉  https://github.com/kubernetes/minikube/issues/new

AdnanHodzic on 1 Aug 2019

This was fixed in minikube v1.4:

minikube cache add quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0

tstromberg on 20 Sep 2019

👍1

➜  ~ minikube cache add quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0


E1009 13:42:20.927214   47862 cache_images.go:79] CacheImage quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0 -> /Users/reegnz/.minikube/cache/images/quay.io/kubernetes-ingress-controller/nginx-ingress-controller_0.23.0 failed: manifest digest: "sha256:603b7018c941b1117da4eb980c8043992ad37abdde3636fcbae1f34bcfff443f" does not match Docker-Content-Digest: "sha256:e1292564ba5f1fd75005a4575778523d3309fb5d5d57f6597234c0b1567641f6" for "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0"

💣  Failed to cache and load images: caching images: caching image /Users/reegnz/.minikube/cache/images/quay.io/kubernetes-ingress-controller/nginx-ingress-controller_0.23.0: manifest digest: "sha256:603b7018c941b1117da4eb980c8043992ad37abdde3636fcbae1f34bcfff443f" does not match Docker-Content-Digest: "sha256:e1292564ba5f1fd75005a4575778523d3309fb5d5d57f6597234c0b1567641f6" for "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0"

😿  Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
👉  https://github.com/kubernetes/minikube/issues/new/choose
➜  ~ minikube version
minikube version: v1.4.0
commit: 7969c25a98a018b94ea87d949350f3271e9d64b6

Doesn't look fixed to me. Although now we only have a the Content-Digest issue. (I am running the MacOS version).

Should I open a new issue about that? Still only affects the quay.io images.

reegnz on 9 Oct 2019

👀1

I am getting the same does not match Docker-Content-Digest error as above in minikube 1.4.0 and 1.5.2 only on quay.io

chris-free on 7 Nov 2019

I assume this is an issue with the go-containerregistry library we use. Has
anyone tried updating to the latest version in the go.mod file?

On Thu, Nov 7, 2019, 8:22 AM Chris Freeman notifications@github.com wrote:

I am getting the same does not match Docker-Content-Digest error as above
in minikube 1.4.0 and 1.5.2

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/minikube/issues/4016?email_source=notifications&email_token=AAAYYMCB7WVQTHVQMOO63T3QSQ6FJA5CNFSM4HCAXJMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDM7ABA#issuecomment-551153668,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAYYMFX3N3XB4RLP6HIE6TQSQ6FJANCNFSM4HCAXJMA
.

tstromberg on 7 Nov 2019

Commit https://github.com/google/go-containerregistry/commit/e3df2a4fd30e0756e8fc2031f5a02c5c0b25f10d removes the docker digest check.

Using this version Minikube make then gives:
/home/chris/go/pkg/mod/github.com/google/[email protected]/pkg/v1/daemon/image.go:80:5: cli.NegotiateAPIVersion undefined (type *client.Client has no field or method NegotiateAPIVersion) /home/chris/go/pkg/mod/github.com/google/[email protected]/pkg/v1/daemon/write.go:43:5: cli.NegotiateAPIVersion undefined (type *client.Client has no field or method NegotiateAPIVersion)

Could be wrong here - but this looks to be a dependency conflict whereby go-containerregistry@e3df2a4 expects a different version of docker/docker

chris-free on 16 Nov 2019

@chris-free - Thanks for looking into this. We don't directly depend on docker, so it shouldn't be a problem to update our dependency on docker/docker with whatever version go-containerregistry looks for.

Hopefully there isn't a libmachine conflict, anyways =)

tstromberg on 17 Nov 2019

Hello, i also tried @chris-free approach with same results.

If I put the latest version of go-containerregistry AND remove the indirect dependency on docker/docker 1.13.1 in go.mod, a "go mod tidy" command will restore the dependency to 1.13.1...

I tried "go mod graph" to understand which other component may require this version 1.13.1 but did not find anything, so I'm a bit lost here.

BTW, why is this issue closed ?

cvila84 on 18 Nov 2019

Was this page helpful?

0 / 5 - 0 ratings