Buildkit: Support schema1 push for quay?

[AkihiroSuda]

Astonishingly Quay.io still does not support schema2: https://github.com/bazelbuild/rules_docker/issues/102

DEBU[0011] do request                                    digest=sha256:eb300a827decea6de23bda3e4ec5a60dcb3fb59bd01792fe3b54c08c10f68214 mediatype="application/vnd.docker.distribution.manifest.v2+json" request.headers=map[Content-Type:[application/vnd.docker.distribution.m
anifest.v2+json]] request.method=PUT size=1245 url="https://quay.io/v2/****/****/manifests/latest"
DEBU[0012] fetch response received                       digest=sha256:eb300a827decea6de23bda3e4ec5a60dcb3fb59bd01792fe3b54c08c10f68214 mediatype="application/vnd.docker.distribution.manifest.v2+json" response.headers=map[Server:[nginx/1.13.12] Date:[Thu, 24 May 2018 03:1
1:16 GMT] Content-Type:[application/json] Content-Length:[131]] size=1245 status="415 Unsupported Media Type" url="https://quay.io/v2/****/****/manifests/latest"
ERRO[0012] /moby.buildkit.v1.Control/Solve returned error: unexpected status: 415 Unsupported Media Type

Do we want to support pushing as schema1?

I hesitate to add support for such deprecated format, but probably we should do if there are also other registry implementations that lack support for schema2.

cc @alexellis
cc @dmcgowan @stevvooe

[AkihiroSuda]

It neither supports application/vnd.oci.image.manifest.v1+json :cry:

[dmcgowan]

Do we want to support pushing as schema1?

I can give my perspective here, no

We can continue to support pull but schema1 is less secure and the push problem can quickly be remedied by the registries.

[alexellis]

:-1: this rules out what appears to be the second most popular and successful registry from being used with Buildkit.

[jzelinskie]

We're actively working on implementing Schema v2-2 on Quay and you should expect it in the coming months.

We were initially hesitant to implement it because of the lack of tooling around ManifestLists and real world usage of multi-arch/os. We now have real world use cases and additional tooling built into the docker client, which has incentivized us to get this work done.

As with all previous implementations, images pushed to Quay with v2-2 will also have backwards compatibility for older clients.

[AkihiroSuda]

I'm trying to implement schema1, but still failing with 400 Bad Request.
https://github.com/AkihiroSuda/buildkit_poc/commits/schema1

[alexellis]

Thanks @AkihiroSuda for Quay (used in large banks and enterprise) it currently rules-out the use of buildkit. This support would be really useful for those people who are evaluating OpenFaaS / Cloud with Buildkit.

[alexellis]

That's great @jzelinskie, will that also filter into the on-premise versions of Quay as well as the online version? 👍

[AkihiroSuda]

@alexellis

Generating schema1 manifest seems very complicated than I thought.

Is is possible to just export OCI archive and let skopeo push to Quay?
(I confirmed skopeo works with quay.io)

The code for executing skopeo will be on of-builder repo, not on buildkit repo probably.

But I'm not sure how to detect on-premise quay.

[AkihiroSuda]

@jzelinskie What's current status? ^^

[AkihiroSuda]

ping @jzelinskie

[johanneswuerbach]

@jzelinskie any update on this? We also run into this issue when preparing prometheus for multi-arch images https://github.com/prometheus/busybox/issues/25

[jzelinskie]

Quay is incrementally rolling out both OCI and v2_2. You might already be in the percentage that are enabled, if not, you can message me the namespace and I can get it whitelisted for you.

[AkihiroSuda]

Thanks @jzelinskie

can we close this issue? @alexellis

[HerrmannHinz]

@jzelinskie i have an on prem quay installation running quay.io/coreos/quay:v2.9.3
can we expect a fix for this as well?

[jzelinskie]

@HerrmannHinz you'll have to wait for a v3 release for on-prem.

[HerrmannHinz]

@jzelinskie that will be when? approx?

[chanseokoh]

Manifest V2.2 still not supported (on cloud), returning 415 Unsupported Media Type:

$ curl -v -X PUT -H 'Content-Type: application/vnd.docker.distribution.manifest.v2+json' https://quay.io/v2/francium25/test/manifests/add-tag
...
< HTTP/2 415 
...
{"errors":[{"code":"MANIFEST_INVALID","detail":{"message":"manifest schema version not supported"},"message":"manifest invalid"}]}

OTOH, Manifest V2.1 works, as it has been:

$ curl -v -X PUT -H 'Content-Type: application/vnd.docker.distribution.manifest.v1+json' https://quay.io/v2/francium25/test/manifests/add-tag
...
< HTTP/2 401 
...
{"errors":[{"code":"UNAUTHORIZED","detail":{},"message":"access to the requested resource is not authorized"}]}

[webvictim]

@jzelinskie Hey - has the rollout of support for V2_2 manifests on quay.io got to 100% yet? If not, is there some way to get whitelisted for it?

[vielmetti]

https://www.redhat.com/en/blog/introducing-red-hat-quay-3-registry-your-linux-and-windows-containers is the announcement from today.

[natalia-pokrovskaya]

Quay is incrementally rolling out both OCI and v2_2. You might already be in the percentage that are enabled, if not, you can message me the namespace and I can get it whitelisted for you.

@jzelinskie Can you please whitelist "altissia"? 🙏

[webvictim]

Quay is incrementally rolling out both OCI and v2_2. You might already be in the percentage that are enabled, if not, you can message me the namespace and I can get it whitelisted for you.

@jzelinskie We'd also appreciate a whitelist for "gravitational" - thanks.

[masih]

@jzelinskie Can you please whitelist solarwinds?

Many thanks

[sameeakhan]

@jzelinskie does namespace here mean the same thing as organization on Quay? If so, whitelist "upskill" please?

[visheshtanksale]

@jzelinskie Any updates on when will support for 2_2 schema roll out to all repos?

[HerrmannHinz]

@jzelinskie could you please add mercedesbenzio ?

[crazy-max]

@jzelinskie Could you add crazymax and ftpgrab namespaces please ? :pray:

[BenHizak]

in this discussion, quay.io said again that they are planning to upgrade schema

[cmoulliard]

Problem is still there for me on Quay.io using this org - https://quay.io/repository/cmoulliard

[INFO] Containerizing application to quay.io/cmoulliard/hello-sb...
[WARNING] Base image 'gcr.io/distroless/java:8' does not use a specific image digest - build may not be reproducible
[INFO] Using base image with digest: sha256:529c741e75e153005747da4b805226c2a48d63fdd05167f117a8329e78eb1286
[INFO] 
[INFO] Container entrypoint set to [java, -cp, /app/resources:/app/classes:/app/libs/*, io.halkyon.HelloWorldApplication]
[INFO] Executing tasks:
[INFO] [===========================   ] 90.0% complete
[INFO] > launching layer pushers
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6.876 s
[INFO] Finished at: 2019-10-26T09:23:53+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:1.7.0:build (default-cli) on project hello: Tried to push image manifest for quay.io/cmoulliard/hello-sb:latest but failed because: 
Registry may not support pushing OCI Manifest or Docker Image Manifest Version 2, Schema 2 | If 
this is a bug, please file an issue at https://github.com/GoogleContainerTools/jib/issues/new: 415 
UNSUPPORTED MEDIA TYPE
[ERROR] {"errors":[{"code":"MANIFEST_INVALID","detail":{"message":"manifest schema version not 
supported"},"message":"manifest invalid"}]}
[ERROR] -> [Help 1]

[atuljha-optimizely]

@jzelinskie Can you whitelist optimizely namespace?

[zhangguanzhang]

@jzelinskie
When will quay.io fully use v2's scheme2.manifest

[chanseokoh]

For those who want their Quay.io namespace whitelisted for V2 Schema 2:

I don't really know if any Red Hat member is responding to whitelist requests raised here and if they are being handled. Asking @jzelinskie here may be working, I don't know. However, this repo is Docker's Buildkit and not about Quay.io, so it is weird to see so many people asking for Quay.io whitelisting in this thread.

In any case, I've seen that Red Hat folks are quite actively responding in this official quay-sig forum: https://groups.google.com/forum/#!forum/quay-sig
Perhaps it's a better strategy to also make a whitelist request there.

About when Quay.io will fully use Schema 2, one of the posts says

We aim to switch it on globally at the end of the year but in case the underlying changes will take longer than expected it would slip to next year.

And for filing bugs for the opensource Quay, I think this is the right place: https://issues.jboss.org/projects/PROJQUAY

Hope this helps.

[morvencao]

at the end of the year? Just a few days left.

[sbueringer]

at the end of the year? Just a few days left.

not anymore :)

[eedwards-sk]

really dropping the ball here, quay

[DEPRECATION NOTICE] registry v2 schema1 support will be removed in an upcoming release. Please contact admins of the quay.io registry NOW to avoid future disruption. More information at https://docs.docker.com/registry/spec/deprecated-schema-v1/

[gyliu513]

@eedwards-sk so what is the plan to support registry v2 schema2 ? Thanks !

[willejs]

@jzelinskie can i get OCI & V2.2 enabled on the ometria org please?

[jonjohnsonjr]

According to https://groups.google.com/d/msg/quay-sig/lImWndQIrE0/2mRVOJOhBQAJ quay should support schema 2 everywhere now.