Origin: some `oc image mirror` issues

[legionus]

1. __Not canonical repository name__

According to the documentation:

# Copy image to S3 (pull from <bucket>.s3.amazonaws.com/image:latest)
oc image mirror myregistry.com/myimage:latest s3://s3.amazonaws.com/<region>/<bucket>/image:latest

Actual result:

$ oc image mirror docker.io/library/busybox:latest s3://s3.amazonaws.com/us-east-1/aos-agladkov-registry/myproject/toybox

error: unable to connect to s3.amazonaws.com/us-east-1/aos-agladkov-registry/myproject/toybox: repository name must be canonical
error: an error occurred during planning

It happens because this command uses ParseDockerImageReference to parse image reference, but this function requires that reference be canonical.

https://github.com/openshift/origin/blob/master/pkg/oc/cli/cmd/image/mirror/mappings.go#L29
https://github.com/openshift/origin/blob/master/pkg/oc/cli/cmd/image/mirror/mappings.go#L46

One way to make this link canonical is to specify a domain with a dot or add localhost:

$ oc image mirror docker.io/library/busybox:latest s3://s3.amazonaws.com/us-east-1/aos-agladkov-registry/example.com/myproject/toybox
$ oc image mirror docker.io/library/busybox:latest s3://s3.amazonaws.com/us-east-1/aos-agladkov-registry/localhost/myproject/toybox

But even if it turns out so to put something like that, the URL to the image will be:

aos-agladkov-registry.s3.amazonaws.com/example.com/myproject/toybox
aos-agladkov-registry.s3.amazonaws.com/localhost/myproject/toybox

1. __Unable to push manifest__

$ oc image mirror docker.io/library/busybox:latest s3://s3.amazonaws.com/us-east-1/aos-agladkov-registry/example.com/myproject/toybox
...
phase 0:
  s3.amazonaws.com example.com/myproject/toybox blobs=16 mounts=0 manifests=1 shared=0

info: Planning completed in 2.93s
error: unable to upload blob sha256:4639278eb7ec424805d31508723f2868f2b3d26475c76f09f4aa87be6d55572c to s3.amazonaws.com/example.com/myproject/toybox: S3 target blob store requires blobs to have mount stats that include a digest
error: unable to upload blob sha256:b6467317087572599bdc9ff215788f7bf01aa201b0b377971901f1b4ca5c025d to s3.amazonaws.com/example.com/myproject/toybox: S3 target blob store requires blobs to have mount stats that include a digest
error: unable to upload blob sha256:03f1257c1e23796261321cc0b8b518b35e65a965f16547f087c9b7f89afc13b4 to s3.amazonaws.com/example.com/myproject/toybox: S3 target blob store requires blobs to have mount stats that include a digest
error: unable to upload blob sha256:1f4144adca0f4a32f35c8aeaa6e33a2470dbf226cd06cec72eae8754e071dd15 to s3.amazonaws.com/example.com/myproject/toybox: S3 target blob store requires blobs to have mount stats that include a digest
error: unable to upload blob sha256:45354500ca49c359a901800c06c9664e336d4652e6a7ed39b91e594f7c5d5f09 to s3.amazonaws.com/example.com/myproject/toybox: S3 target blob store requires blobs to have mount stats that include a digest
error: unable to upload blob sha256:f70adabe43c0cccffbae8785406d490e26855b8748fc982d14bc2b20c778b929 to s3.amazonaws.com/example.com/myproject/toybox: S3 target blob store requires blobs to have mount stats that include a digest
error: unable to push docker.io/library/busybox: failed to upload blob sha256:b8a401baa6b218ffa521c651eba0c876a7a211d840734b33c009c4d8af89cda1: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
error: unable to upload blob sha256:52894eca45f5ddc393bb8211350d916060ddf791d480160f538465dfc76048b3 to s3.amazonaws.com/example.com/myproject/toybox: S3 target blob store requires blobs to have mount stats that include a digest
error: unable to upload blob sha256:3f46e1095de266d1af2b069542c947ebe417063bedd12d781dcd39c9f6759ed3 to s3.amazonaws.com/example.com/myproject/toybox: S3 target blob store requires blobs to have mount stats that include a digest
error: unable to push docker.io/library/busybox: failed to upload blob sha256:8ac48589692a53a9b8c2d1ceaa6b402665aa7fe667ba51ccc03002300856d8c7: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
error: unable to push docker.io/library/busybox: failed to upload blob sha256:b62c88e2a3d11d887c7ca2bd57be02a7998d4255f349c02a39ee6cca408891e4: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
error: unable to push docker.io/library/busybox: failed to upload blob sha256:1063bcd1bb2a382f78d888cf6c6f891c35123352e1c2950079da2f362086f5bb: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
error: unable to push docker.io/library/busybox: failed to upload blob sha256:f688e3dede8cf452d4c06a3f91eab2cba12a2ffcbbe3467469aa89bbb3a93afd: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
error: unable to push docker.io/library/busybox: failed to upload blob sha256:420f307579300330442e3cadbd50b43bfd8870ff79dc90be8492826ec20e412a: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
error: unable to push docker.io/library/busybox: failed to upload blob sha256:3d9f9d95c0302dfaa04f09d22cb49ed6de296a190cf0ebbb582e9ff665c5ed3d: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
error: unable to push docker.io/library/busybox: failed to upload blob sha256:ba3a78826904c625e65a2eed1f247bbab59898f043490e7113e88907bf7c6b3b: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
info: Mirroring completed in 520ms (0B/s)
error: one or more errors occurred while uploading images

I did not find a way to fix it.

1. __S3 credentials__

I found the only way to specify the login and password for S3: The S3 credentials must be encoded and written to the ~/.docker/config.json manually.

{                                                                                
    "auths": {                                                                    
       "s3.amazonaws.com": {                                                      
           "auth": "dXNlcm5hbWU6cGFzc3dvcmQ="                                      
       }                                                                          
    }                                                                             
}      

Is this really the only way?

1. __Docker defaults__

When you are using a reference to dockerhub, there are a number of defaults that allow not to specify docker.io, latest as tag andlibrary component in case the image is official. This works in the docker client, it works in other oc sub-commands, but does not work here. The error message does not reflect the essence of the problem:

$ oc image mirror docker.io/busybox:latest docker.io/foo

error: unable to retrieve source image docker.io/library/busybox by tag: errors:
denied: requested access to the resource is denied
unauthorized: authentication required

error: an error occurred during planning

Moreover, the message shows a normalized repository name, which is even more confusing.

[legionus]

Also I tried to test it with minio. The integrated registry server works fine with it, but oc image mirror not:

$ oc image mirror docker.io/library/busybox:latest s3://127.0.0.1:9000/us-east-1/mybucket/example.com/myproject/toybox
127.0.0.1:9000/
  example.com/myproject/toybox
    blobs:
      docker.io/library/busybox sha256:b62c88e2a3d11d887c7ca2bd57be02a7998d4255f349c02a39ee6cca408891e4 1.457KiB
      docker.io/library/busybox sha256:3d9f9d95c0302dfaa04f09d22cb49ed6de296a190cf0ebbb582e9ff665c5ed3d 1.459KiB
      docker.io/library/busybox sha256:1063bcd1bb2a382f78d888cf6c6f891c35123352e1c2950079da2f362086f5bb 1.46KiB
      docker.io/library/busybox sha256:f688e3dede8cf452d4c06a3f91eab2cba12a2ffcbbe3467469aa89bbb3a93afd 1.46KiB
      docker.io/library/busybox sha256:420f307579300330442e3cadbd50b43bfd8870ff79dc90be8492826ec20e412a 1.462KiB
      docker.io/library/busybox sha256:8ac48589692a53a9b8c2d1ceaa6b402665aa7fe667ba51ccc03002300856d8c7 1.462KiB
      docker.io/library/busybox sha256:ba3a78826904c625e65a2eed1f247bbab59898f043490e7113e88907bf7c6b3b 1.462KiB
      docker.io/library/busybox sha256:b8a401baa6b218ffa521c651eba0c876a7a211d840734b33c009c4d8af89cda1 1.464KiB
      docker.io/library/busybox sha256:3f46e1095de266d1af2b069542c947ebe417063bedd12d781dcd39c9f6759ed3 657.2KiB
      docker.io/library/busybox sha256:4639278eb7ec424805d31508723f2868f2b3d26475c76f09f4aa87be6d55572c 671.2KiB
      docker.io/library/busybox sha256:1f4144adca0f4a32f35c8aeaa6e33a2470dbf226cd06cec72eae8754e071dd15 688.2KiB
      docker.io/library/busybox sha256:f70adabe43c0cccffbae8785406d490e26855b8748fc982d14bc2b20c778b929 706.2KiB
      docker.io/library/busybox sha256:45354500ca49c359a901800c06c9664e336d4652e6a7ed39b91e594f7c5d5f09 743.9KiB
      docker.io/library/busybox sha256:52894eca45f5ddc393bb8211350d916060ddf791d480160f538465dfc76048b3 835.4KiB
      docker.io/library/busybox sha256:03f1257c1e23796261321cc0b8b518b35e65a965f16547f087c9b7f89afc13b4 2.018MiB
      docker.io/library/busybox sha256:b6467317087572599bdc9ff215788f7bf01aa201b0b377971901f1b4ca5c025d 2.044MiB
    manifests:
      {}
  stats: shared=0 unique=16 size=8.275MiB ratio=1.00

phase 0:
  127.0.0.1:9000 example.com/myproject/toybox blobs=16 mounts=0 manifests=1 shared=0

info: Planning completed in 3.15s
error: unable to upload blob sha256:1f4144adca0f4a32f35c8aeaa6e33a2470dbf226cd06cec72eae8754e071dd15 to 127.0.0.1:9000/example.com/myproject/toybox: S3 target blob store requires blobs to have mount stats that include a digest
error: unable to push docker.io/library/busybox: failed to upload blob sha256:ba3a78826904c625e65a2eed1f247bbab59898f043490e7113e88907bf7c6b3b: RequestError: send request failed
caused by: Put https://127.0.0.1:9000/mybucket/v2/: EOF
error: unable to upload blob sha256:4639278eb7ec424805d31508723f2868f2b3d26475c76f09f4aa87be6d55572c to 127.0.0.1:9000/example.com/myproject/toybox: S3 target blob store requires blobs to have mount stats that include a digest
error: unable to upload blob sha256:b6467317087572599bdc9ff215788f7bf01aa201b0b377971901f1b4ca5c025d to 127.0.0.1:9000/example.com/myproject/toybox: S3 target blob store requires blobs to have mount stats that include a digest
error: unable to upload blob sha256:f70adabe43c0cccffbae8785406d490e26855b8748fc982d14bc2b20c778b929 to 127.0.0.1:9000/example.com/myproject/toybox: S3 target blob store requires blobs to have mount stats that include a digest
error: unable to upload blob sha256:45354500ca49c359a901800c06c9664e336d4652e6a7ed39b91e594f7c5d5f09 to 127.0.0.1:9000/example.com/myproject/toybox: S3 target blob store requires blobs to have mount stats that include a digest
error: unable to push docker.io/library/busybox: failed to upload blob sha256:1063bcd1bb2a382f78d888cf6c6f891c35123352e1c2950079da2f362086f5bb: RequestError: send request failed
caused by: Put https://127.0.0.1:9000/mybucket/v2/: EOF
error: unable to upload blob sha256:52894eca45f5ddc393bb8211350d916060ddf791d480160f538465dfc76048b3 to 127.0.0.1:9000/example.com/myproject/toybox: S3 target blob store requires blobs to have mount stats that include a digest
error: unable to push docker.io/library/busybox: failed to upload blob sha256:b62c88e2a3d11d887c7ca2bd57be02a7998d4255f349c02a39ee6cca408891e4: RequestError: send request failed
caused by: Put https://127.0.0.1:9000/mybucket/v2/: EOF
error: unable to push docker.io/library/busybox: failed to upload blob sha256:f688e3dede8cf452d4c06a3f91eab2cba12a2ffcbbe3467469aa89bbb3a93afd: RequestError: send request failed
caused by: Put https://127.0.0.1:9000/mybucket/v2/: EOF
error: unable to upload blob sha256:03f1257c1e23796261321cc0b8b518b35e65a965f16547f087c9b7f89afc13b4 to 127.0.0.1:9000/example.com/myproject/toybox: S3 target blob store requires blobs to have mount stats that include a digest
error: unable to push docker.io/library/busybox: failed to upload blob sha256:3d9f9d95c0302dfaa04f09d22cb49ed6de296a190cf0ebbb582e9ff665c5ed3d: RequestError: send request failed
caused by: Put https://127.0.0.1:9000/mybucket/v2/: EOF
error: unable to upload blob sha256:3f46e1095de266d1af2b069542c947ebe417063bedd12d781dcd39c9f6759ed3 to 127.0.0.1:9000/example.com/myproject/toybox: S3 target blob store requires blobs to have mount stats that include a digest
error: unable to push docker.io/library/busybox: failed to upload blob sha256:8ac48589692a53a9b8c2d1ceaa6b402665aa7fe667ba51ccc03002300856d8c7: RequestError: send request failed
caused by: Put https://127.0.0.1:9000/mybucket/v2/: EOF
error: unable to push docker.io/library/busybox: failed to upload blob sha256:420f307579300330442e3cadbd50b43bfd8870ff79dc90be8492826ec20e412a: RequestError: send request failed
caused by: Put https://127.0.0.1:9000/mybucket/v2/: EOF
error: unable to push docker.io/library/busybox: failed to upload blob sha256:b8a401baa6b218ffa521c651eba0c876a7a211d840734b33c009c4d8af89cda1: RequestError: send request failed
caused by: Put https://127.0.0.1:9000/mybucket/v2/: EOF
info: Mirroring completed in 1.17s (0B/s)
error: one or more errors occurred while uploading images

[bparees]

@smarterclayton any feedback on this?

@legionus i did have a hallway conversation with clayton in which he suggested the s3 aspects of oc image mirror should be marked experimental at best and probably don't need tests/doc/etc right now (of course the entire command is marked experimental today, but if we lifted that, we'd want to keep it for the s3 use cases)

[smarterclayton]

Sorry, I need to come back to this. It's possible minio fails because it doesn't support setting HTTP headers? It's also possible that my fix to docker/distribution registry client got accidentally reverted in origin (the one that handles an edge case correctly regarding the absence of a digest).

[smarterclayton]

s3 credentials Is this really the only way?

Yeah, if you had two S3 buckets how would you publish to different ones with different credentials? s3 config gets complex, I was trying to keep it simple.

Docker defaults

Some of the other oc commands deal with origin specifically, i think for more general images we should be explicit. It's always possible the docker hub might not be around forever, and we need to wean people off the defaults somehow. Not the end of the world to support it, but I don't like relying on the defaulting in the long term.

Will look at the other two.

[dmage]

@smarterclayton if you are talking about the change https://github.com/openshift/origin/commit/b8366d5a50d69f81d4770e2cfacc49fabf135532, then yes, it was reverted.

You can see it in the history of vendor/github.com/docker/distribution/registry/client/repository.go. We've lost it in transition to the publisher bot.

Origin's master uses openshift-3.10-docker-edc3ab2 from openshift/docker-distribution which doesn't have these changes. So, if someone will do glice up -v, it will revert the change https://github.com/openshift/origin/commit/81426ee753a648bd00ab006f369c50141b7f1465#diff-dd244c0674a710542a3925a985c3dd71 as well. And that is sad, it means that our dependency management is broken and no one understand how to do it right. And it means nobody can (and never did) glide up -v in the origin repo.

So far, we need to cherry-pick those 2 commits into the docker-distribution tree and cherry pick one of them into origin (probably this is not the only problem, so glide up -v is still dangerous to use).

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[bparees]

/remove-lifecycle stale
/lifecycle frozen

[matihost]

Also:

• _oc image import_ has also no documentation except command help.
  it would be nice to mention in documentation that file ~/.docker/config.json has to be present on the machine
• it doesn't support referencing internal registry images (aka ImageStreamTags) as source image (w/o providing internal Docker registry address explicitly)

[bparees]

@matihost

oc image-import documentation (perhaps you meant oc image mirror?):
https://docs.okd.io/latest/dev_guide/managing_images.html#importing-tag-and-image-metadata

oc image mirror documentation:
https://docs.okd.io/latest/dev_guide/managing_images.html#managing-images-mirror-registry-images

If the container image registry requires authentication to pull or push an image, you you need to log in manually, using the docker login command, before executing the oc image mirror command.

while not a direct statement of the need to have a .docker/config.json, it is implicit if you understand how docker login + .docker/config.json work.

[matihost]

Thanks, the 'oc image mirror'

Is it is a valid use case to use 'oc image mirror' as part of BuildConfig Jenkinsfile pipeline inside OKD cluster Jenkins agent to push image from OKD ImageStream (internal Docker registry) to external docker registry?

[bparees]

Is it is a valid use case to use 'oc image mirror' as part of BuildConfig Jenkinsfile pipeline inside OKD cluster Jenkins agent to push image from OKD ImageStream (internal Docker registry) to external docker registry?

yes. (or you can use skopeo).

[Gladskih]

So, has anyone managed to get "oc image mirror" working?
Or I just spent the day flogging a dead horse.

[dmage]

@Gladskih yes, the next command copied the image successfully:

oc image mirror docker.io/library/busybox:latest \
    s3://s3.amazonaws.com/us-west-1/obulatov-test-1/test/image-mirror:latest

[Gladskih]

@dmage
But not in my case: running on Windows for two internal docker registries of different Openshift clusters even if I have plain credentials in my config.js.

[dmage]

@Gladskih feel free to open a new issue. Please include steps to reproduce and output of the mirror command with --loglevel=6.

[Gladskih]

Your @dmage advice about --loglevel=6 is really hellpfull! Thank you, Oleg!
I just figured out that oc image mirror command looking for a config by paths other than the default for docker (%USERPROFILE%.docker).

When Jenkins pipeline (on Windows worker node) run oc:

• C:\jenkins\workspace\MyProject\config.json
• C:\jenkins\workspace\MyProject\config.json
• C:\cygwin64\home\jenkins_user.docker\config.json
• C:.docker\config.json
• C:\jenkins\workspace\MyProject.dockercfg
• C:\jenkins\workspace\MyProject.dockercfg
• C:\cygwin64\home\jenkins_user.dockercfg
• C:.dockercfg

When I run oc (from C:\tools):

• C:\tools\config.json
• C:\tools\config.json
• C:\tools.docker\config.json
• C:.docker\config.json
• C:\tools.dockercfg
• C:\tools.dockercfg
• C:\tools.dockercfg
• C:.dockercfg

So when I place a copy of config.json and daemon.json from C:\Users\MyUserName.docker to C:\tools the command completes successfully on my local machine.
Then I can to specify --config option on docker login command to create config.json in expected by oc place. But also I found that oc image mirror works only if creds specified explicitely in config. With wincred it logs "Found valid config.json at C:\tools\config.json" and "Found secret to match" and but makes GET scope token without Authorisation header and receives {"access_token":"anonymous","token":"anonymous"} and then 401 Unauthorized returns on HEAD and GET of manifest URL.

I created Issue