Kaniko: Unable to pull older tag from DockerHub

[midN]

Actual behavior
Unable to pull older tag from Docker Hub, results in 404.

Expected behavior
Pulls it without issues.

To Reproduce
Steps to reproduce the behavior:

1. Create a Dockerfile with FROM ruby:2.1.2
2. See it fail miserably
3. Change Dockerfile to FROM ruby:2.5.0
4. See it succeed
5. :wat:?

Additional Information
Newer tag, works

INFO[0000] Downloading base image ruby:2.5.0            
2019/01/08 19:46:43 No matching credentials were found, falling back on anonymous
INFO[0000] Executing 0 build triggers                   
INFO[0000] Taking snapshot of full filesystem...        
INFO[0000] Skipping paths under /kaniko, as it is a whitelisted directory 
INFO[0000] Skipping paths under /home/jenkins, as it is a whitelisted directory 
INFO[0000] Skipping paths under /var/run, as it is a whitelisted directory 
INFO[0000] Skipping paths under /dev, as it is a whitelisted directory 
INFO[0000] Skipping paths under /sys, as it is a whitelisted directory 
INFO[0000] Skipping paths under /proc, as it is a whitelisted directory 
INFO[0000] Skipping paths under /busybox, as it is a whitelisted directory 
INFO[0000] Skipping push to container registry due to --no-push flag 

Older tag, works locally and everywhere else, fails miserably with Kaniko

INFO[0000] Downloading base image ruby:2.1.2            
2019/01/08 19:49:00 No matching credentials were found, falling back on anonymous
error building image: getting stage builder for stage 0: unsupported status code 404; body: 404 page not found

Tag exists and works.

docker pull ruby:2.1.2
2.1.2: Pulling from library/ruby
a3ed95caeb02: Already exists
ca4ef1007b96: Already exists
935b2cc382fb: Already exists
34695db0bdf0: Already exists
7bfcfa20e94c: Already exists
5bd64402bda1: Already exists
ffca5e0e0718: Already exists
Digest: sha256:2ebbb68619ba12d07a4fa78cdc6597744b6d9bd047cd52b1695a40175e9427e6
Status: Image is up to date for ruby:2.1.2

[dlorenc]

Can you share the full Dockerfile you tried this with?

[midN]

FROM ruby:2.1.2

:arrow_up: Exact Dockerfile
To be precise - thats 1 LINE Dockerfile, nothing else is in it

[dlorenc]

Wow ok thanks. I was able to repro this. cc @jonjohnsonjr any guesses here?

[jonjohnsonjr]

It's a schema 1 manifest, which we don't want to support. It's "fixable" by doing a docker pull && docker push, which will auto-upgrade it to schema 2.

FWIW, you almost certainly don't want to be using a docker image from 2014 if you care about security.

[midN]

@jonjohnsonjr There are tons of older docker images that people might be using, such as older nginx/apache versions as well as random application pieces like APM agents etc which are not easy to upgrade.

Why not add support for Schema 1 manifests or at least add a meaningful error?
Having it output an "404 not found" is quite confusing.

[midN]

Also, it failed to pull ruby:2.4.0 as well, which is not old and still an supported version.

[jonjohnsonjr]

I can't repro a failure for ruby:2.4.0, does it also 404?

Ack on better error message, we can do that at least.

We could possibly rewrite schema 1 manifests to schema 2 on pulls, but it would be pretty slow because we have to unzip the layers to compute diffids :(

I'll look into it when I get a chance, we might be able to do it lazily.

[priyawadhwa]

Looks like this issue has been resolved in go-containerregistry, so I'm going to close this issue.