Harbor: Cannot replicate images from Red Hat registry (registry.access.redhat.com)

[rfc1459]

Trying to replicate an image from the public Red Hat Container Registry (for instance ubi7/ubi-minimal:7.7-10) fails even though the same image can be pulled by a standard Docker daemon and Red Hat's own podman.

Expected behavior and actual behavior

Expected

Harbor should pull the image from the registry and add it to a local project.

Actual

The replication job fails with strconv.ParseInt: parsing "": invalid syntax while trying to pull the first blob referenced by the manifest.

Steps to reproduce the problem

1. Create a registry with name redhat-registry, type docker-registry and URL https://registry.access.redhat.com
2. Create a project (in my case: ubi7)
3. Create a replicaiton rule with the following settings:
   
   
   
   • Name: redhat-ubi7
   
   
   • Replication mode: Pull-based
   
   
   • Source registry: redhat-registry
   
   
   • Source resource filter:
   
   
   • Name: ubi7/ubi-minimal
   
   
   • Tag: 7.7-10
   
   
   • Resource: image
   
   
   • Destination namespace: ubi7
   
   
   • Trigger Mode: Manual
   
   
   • Override: Yes
   
   
   • Enable rule: Yes
   
   
4. Trigger the replication rule

Versions

• harbor version: [1.9.0]
• docker engine version: [18.09.4]
• docker-compose version: [1.22.0]

Additional context

Relevant excerpt from the job log:

2019-09-23T12:36:21Z [INFO] [/replication/transfer/image/transfer.go:95]: client for source registry [type: docker-registry, URL: https://registry.access.redhat.com, insecure: false] created
2019-09-23T12:36:21Z [INFO] [/replication/transfer/image/transfer.go:105]: client for destination registry [type: harbor, URL: http://core:8080, insecure: true] created
2019-09-23T12:36:21Z [INFO] [/replication/transfer/image/transfer.go:157]: copying ubi7/ubi-init:7.7-10(source registry) to ubi7/ubi-init:7.7-10(destination registry)...
2019-09-23T12:36:21Z [INFO] [/replication/transfer/image/transfer.go:261]: pulling the manifest of image ubi7/ubi-init:7.7-10 ...
2019-09-23T12:36:22Z [INFO] [/replication/transfer/image/transfer.go:272]: the manifest of image ubi7/ubi-init:7.7-10 pulled
2019-09-23T12:36:22Z [INFO] [/replication/transfer/image/transfer.go:294]: trying abstract a manifest from the manifest list...
2019-09-23T12:36:22Z [INFO] [/replication/transfer/image/transfer.go:306]: a manifest(architecture: amd64, os: linux) found, using this one: sha256:c88406b7e200c6a518d594fa5747131c412e38918faf2eb7104e8e1dad641b2c
2019-09-23T12:36:22Z [INFO] [/replication/transfer/image/transfer.go:261]: pulling the manifest of image ubi7/ubi-init:sha256:c88406b7e200c6a518d594fa5747131c412e38918faf2eb7104e8e1dad641b2c ...
2019-09-23T12:36:23Z [INFO] [/replication/transfer/image/transfer.go:272]: the manifest of image ubi7/ubi-init:sha256:c88406b7e200c6a518d594fa5747131c412e38918faf2eb7104e8e1dad641b2c pulled
2019-09-23T12:36:23Z [INFO] [/replication/transfer/image/transfer.go:231]: copying the blob sha256:a99e238f6cc16b6fbd1d613195a50b4ba7bb6358bebad982215807b62331ccb1...
2019-09-23T12:36:23Z [ERROR] [/replication/transfer/image/transfer.go:244]: failed to pulling the blob sha256:a99e238f6cc16b6fbd1d613195a50b4ba7bb6358bebad982215807b62331ccb1: strconv.ParseInt: parsing "": invalid syntax
2019-09-23T12:36:23Z [ERROR] [/replication/transfer/image/transfer.go:143]: strconv.ParseInt: parsing "": invalid syntax

Drilling down from replication/transfer/image/transfer.go to the place where strconv.ParseInt is called when processing blobs, we end up here: https://github.com/goharbor/harbor/blob/cb5a145f275464c5044caa951a3ad99a72a10e1d/src/common/utils/registry/repository.go#L347-L351

Basically, Harbor uses the header to get the blob size instead of parsing it from the manifest, so if the upstream registry does not provide said header it doesn't pull the image and the replication job fails.

Again, both the upstream Docker daemon and the podman utility can successfully pull images from the Red Hat registry (they probably don't even bother checking for the Content-Length header since they get the blob size from the manifest), so the absence of Content-Length should not cause an error.

[renmaosheng]

We have not validated the Red Hat registry, would you like to contribute on this? if so, please submit a proposal and work with our maintainers on this. thanks.

[sluetze]

I had exactly the same issue. But also with dockerhub.
Multiple runs of the replication pull images. even tho the error persists.

[sluetze]

Hi,
we have an additional case. Living behind an "enterprise proxy" we have to pull our images through this proxy.
The proxy does remove the content-size header that was provided from dockerhub.
As content-size is not a MUST in the rfc7320 i would like to see a solution where the content-length is no required.

I was able to verify this with a non-proxy replication and a proxy replication

proxy replication:

2019-11-25T15:40:42Z [INFO] [/replication/transfer/image/transfer.go:95]: client for source registry [type: docker-hub, URL: https://hub.docker.com, insecure: true] created
2019-11-25T15:40:42Z [INFO] [/replication/transfer/image/transfer.go:105]: client for destination registry [type: harbor, URL: http://core:8080, insecure: true] created
2019-11-25T15:40:42Z [INFO] [/replication/transfer/image/transfer.go:138]: copying library/alpine:[latest](source registry) to test/alpine:[latest](destination registry)...
2019-11-25T15:40:42Z [INFO] [/replication/transfer/image/transfer.go:157]: copying library/alpine:latest(source registry) to test/alpine:latest(destination registry)...
2019-11-25T15:40:42Z [INFO] [/replication/transfer/image/transfer.go:261]: pulling the manifest of image library/alpine:latest ...
2019-11-25T15:40:44Z [INFO] [/replication/transfer/image/transfer.go:272]: the manifest of image library/alpine:latest pulled
2019-11-25T15:40:44Z [INFO] [/replication/transfer/image/transfer.go:294]: trying abstract a manifest from the manifest list...
2019-11-25T15:40:44Z [INFO] [/replication/transfer/image/transfer.go:306]: a manifest(architecture: amd64, os: linux) found, using this one: sha256:e4355b66995c96b4b468159fc5c7e3540fcef961189ca13fee877798649f531a
2019-11-25T15:40:44Z [INFO] [/replication/transfer/image/transfer.go:261]: pulling the manifest of image library/alpine:sha256:e4355b66995c96b4b468159fc5c7e3540fcef961189ca13fee877798649f531a ...
2019-11-25T15:40:44Z [INFO] [/replication/transfer/image/transfer.go:272]: the manifest of image library/alpine:sha256:e4355b66995c96b4b468159fc5c7e3540fcef961189ca13fee877798649f531a pulled
2019-11-25T15:40:44Z [INFO] [/replication/transfer/image/transfer.go:231]: copying the blob sha256:965ea09ff2ebd2b9eeec88cd822ce156f6674c7e99be082c7efac3c62f3ff652...
2019-11-25T15:40:44Z [ERROR] [/replication/transfer/image/transfer.go:244]: failed to pulling the blob sha256:965ea09ff2ebd2b9eeec88cd822ce156f6674c7e99be082c7efac3c62f3ff652: strconv.ParseInt: parsing "": invalid syntax
2019-11-25T15:40:44Z [ERROR] [/replication/transfer/image/transfer.go:143]: strconv.ParseInt: parsing "": invalid syntax

non proxy:

2019-11-25T15:48:32Z [INFO] [/replication/transfer/image/transfer.go:95]: client for source registry [type: docker-hub, URL: https://hub.docker.com, insecure: false] created
2019-11-25T15:48:32Z [INFO] [/replication/transfer/image/transfer.go:105]: client for destination registry [type: harbor, URL: http://core:8080, insecure: true] created
2019-11-25T15:48:32Z [INFO] [/replication/transfer/image/transfer.go:138]: copying library/alpine:[latest](source registry) to test/alpine:[latest](destination registry)...
2019-11-25T15:48:32Z [INFO] [/replication/transfer/image/transfer.go:157]: copying library/alpine:latest(source registry) to test/alpine:latest(destination registry)...
2019-11-25T15:48:32Z [INFO] [/replication/transfer/image/transfer.go:261]: pulling the manifest of image library/alpine:latest ...
2019-11-25T15:48:33Z [INFO] [/replication/transfer/image/transfer.go:271]: the manifest of image library/alpine:latest pulled
2019-11-25T15:48:33Z [INFO] [/replication/transfer/image/transfer.go:293]: trying abstract a manifest from the manifest list...
2019-11-25T15:48:33Z [INFO] [/replication/transfer/image/transfer.go:305]: a manifest(architecture: amd64, os: linux) found, using this one: sha256:e4355b66995c96b4b468159fc5c7e3540fcef961189ca13fee877798649f531a
2019-11-25T15:48:33Z [INFO] [/replication/transfer/image/transfer.go:261]: pulling the manifest of image library/alpine:sha256:e4355b66995c96b4b468159fc5c7e3540fcef961189ca13fee877798649f531a ...
2019-11-25T15:48:33Z [INFO] [/replication/transfer/image/transfer.go:271]: the manifest of image library/alpine:sha256:e4355b66995c96b4b468159fc5c7e3540fcef961189ca13fee877798649f531a pulled
2019-11-25T15:48:34Z [INFO] [/replication/transfer/image/transfer.go:231]: copying the blob sha256:965ea09ff2ebd2b9eeec88cd822ce156f6674c7e99be082c7efac3c62f3ff652...
2019-11-25T15:48:36Z [INFO] [/replication/transfer/image/transfer.go:252]: copy the blob sha256:965ea09ff2ebd2b9eeec88cd822ce156f6674c7e99be082c7efac3c62f3ff652 completed
2019-11-25T15:48:36Z [INFO] [/replication/transfer/image/transfer.go:231]: copying the blob sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17...
2019-11-25T15:48:40Z [INFO] [/replication/transfer/image/transfer.go:252]: copy the blob sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17 completed
2019-11-25T15:48:40Z [INFO] [/replication/transfer/image/transfer.go:330]: pushing the manifest of image test/alpine:latest ...
2019-11-25T15:48:42Z [INFO] [/replication/transfer/image/transfer.go:342]: the manifest of image test/alpine:latest pushed
2019-11-25T15:48:42Z [INFO] [/replication/transfer/image/transfer.go:200]: copy library/alpine:latest(source registry) to test/alpine:latest(destination registry) completed
2019-11-25T15:48:42Z [INFO] [/replication/transfer/image/transfer.go:151]: copy library/alpine:[latest](source registry) to test/alpine:[latest](destination registry) completed

@renmaosheng
I see this as two topics:

1. support for redhat registry
2. a bug in harbor, which does not allow replication through proxy that remove content-length.

the first one may be solved without additional effort, if the content-length check is gone.

since im interested in both, I am ready to help here... (can't code, but can test)

[finkr]

Red Hat offers three way to clone images: _Authentication with registry tokens_, _Authentication with Red Hat login_ and _Unauthenticated_. The two first use registry.redhat.io, the later one use registry.access.redhat.com. Red Hat is moving away authenticated content from registry.access.redhat.com to registry.redhat.io.

Good news for those with a Red Hat account: I have successfully synchronized images from registry.redhat.io :+1:

Unfortunately, freely available image must still be pulled from registry.access.redhat.com.

(This bug still apply).

_note: this post was edited for clarification._

[rfc1459]

(also, the image _ubi7/ubi-minimal_ is nice to test, because those images are freely available)

Red Hat UBI images are freely available _on registry.access.redhat.com_, hence this issue. Also, I don't think they'll move unauthenticated access to registry.redhat.io soon, that blog post is more than a year old and you still need to use the old registry for freely available images.

[sluetze]

@reasonerjt Hi Daniel, CYTAL ? This content-length check is making image replication unusable for environments with proxy. I don't know if it's just our proxy which causes this problems or also other people. As far as I understand @finkr he has the same issue (content length check)

I would like to have a way forward how this issue can be fixed. Not talking about "support for redhat registry" but for this content length check.

thanks.

[crdnb]

Red Hat offers three way to clone images: _Authentication with registry tokens_, _Authentication with Red Hat login_ and _Unauthenticated_. The two first use registry.redhat.io, the later one use registry.access.redhat.com. Red Hat is moving away authenticated content from registry.access.redhat.com to registry.redhat.io.

Good news for those with a Red Hat account: I have successfully synchronized images from registry.redhat.io 👍

Unfortunately, freely available image must still be pulled from registry.access.redhat.com.

(This bug still apply).

_note: this post was edited for clarification._

Hi @finkr
How did you manage to sync images from the redhat registry (https://registry.redhat.io)? And is it still working for you?

With my configuration, the replication fails with message "failed to fetch image: http error: code 404, message {"errors:" [{"code": "404", "message":"Not Found"}]}.

My configuration (Harbor 1.10):

Registry:
Name: registry.redhat.io
URL: https://registry.redhat.io
Access ID: Valid Access ID
Access Secret: Valid Access Token
Verify Remote Cert: true
Test Connection --> green / all good

Replication:
Source registry: registry.redhat.io
Source resource filter: none
Destination namespace: redhat
Trigger Mode: Manual
Override: true
Enable rule: true

Would be very happy if you coul post me your configuration! Thank you

[finkr]

@crdnb , try to specify a repo to replicate, may be:

Source resource filter:

• Name: ubi8/ubi
• Tag: _leave empty_

[phisco]

Same issue here, even with the filter you suggested @finkr I did not manage to have it working.

[steven-zou]

@rfc1459 @finkr @phisco

We'll take a look at the Content-length checking issue. Any progress, I'll post them here.

[steven-zou]

@chlins

This issue is related to #11858. All are related to Content-length header settings.

[steven-zou]

@rfc1459 @finkr @phisco @sluetze

PR is merged. Who can help to verify if the issue has gone at the RedHat registry side too? Thanks!

[sluetze]

@steven-zou : I was able to replicate the ubi8minimal image using the authenticated endpoint registry.redhat.io.
I was furthermore able to replicate the ubi7minimal image using the unauthenticated endpoint registry.access.redhat.com

The performance was quite low, which might be caused by my testenvironment and unrelated
can't test it through corporate proxy, but am confident (since it was the same error) that this issue is gone too.

ubi8minimal replication:

2020-06-02T13:45:33Z [INFO] [/replication/transfer/image/transfer.go:116]: client for source registry [type: docker-registry, URL: https://registry.redhat.io, insecure: true] created
2020-06-02T13:45:33Z [INFO] [/replication/transfer/image/transfer.go:126]: client for destination registry [type: harbor, URL: http://core:8080, insecure: true] created
2020-06-02T13:45:33Z [INFO] [/replication/transfer/image/transfer.go:159]: copying ubi8/ubi-minimal:[8.2](source registry) to ubi8/ubi-minimal:[8.2](destination registry)...
2020-06-02T13:45:33Z [INFO] [/replication/transfer/image/transfer.go:180]: copying ubi8/ubi-minimal:8.2(source registry) to ubi8/ubi-minimal:8.2(destination registry)...
2020-06-02T13:45:33Z [INFO] [/replication/transfer/image/transfer.go:293]: pulling the manifest of artifact ubi8/ubi-minimal:8.2 ...
2020-06-02T13:45:34Z [INFO] [/replication/transfer/image/transfer.go:299]: the manifest of artifact ubi8/ubi-minimal:8.2 pulled
2020-06-02T13:45:34Z [INFO] [/replication/transfer/image/transfer.go:180]: copying ubi8/ubi-minimal:sha256:fc7fc7bc6b293fd2ec1c1b035acc6b97ba34f0700c5366b699c646d947c39f81(source registry) to ubi8/ubi-minimal:sha256:fc7fc7bc6b293fd2ec1c1b035acc6b97ba34f0700c5366b699c646d947c39f81(destination registry)...
2020-06-02T13:45:34Z [INFO] [/replication/transfer/image/transfer.go:293]: pulling the manifest of artifact ubi8/ubi-minimal:sha256:fc7fc7bc6b293fd2ec1c1b035acc6b97ba34f0700c5366b699c646d947c39f81 ...
2020-06-02T13:45:34Z [INFO] [/replication/transfer/image/transfer.go:299]: the manifest of artifact ubi8/ubi-minimal:sha256:fc7fc7bc6b293fd2ec1c1b035acc6b97ba34f0700c5366b699c646d947c39f81 pulled
2020-06-02T13:45:34Z [INFO] [/replication/transfer/image/transfer.go:257]: copying the blob sha256:401e359e0f45bfdcf004e258b72e253fd07fba8cc5c6f2ed4f4608fb119ecc2c...
2020-06-02T13:45:34Z [INFO] [/replication/transfer/image/transfer.go:284]: copy the blob sha256:401e359e0f45bfdcf004e258b72e253fd07fba8cc5c6f2ed4f4608fb119ecc2c completed
2020-06-02T13:45:34Z [INFO] [/replication/transfer/image/transfer.go:257]: copying the blob sha256:e96e3a1df3b2b1e01f9614725b50ea4d1d8e480980e456815ade3c7afca978d7...
2020-06-02T13:45:40Z [INFO] [/replication/transfer/image/transfer.go:284]: copy the blob sha256:e96e3a1df3b2b1e01f9614725b50ea4d1d8e480980e456815ade3c7afca978d7 completed
2020-06-02T13:45:40Z [INFO] [/replication/transfer/image/transfer.go:257]: copying the blob sha256:1b99828eddf5297ca28fa7eac7f47a40d36a693c628311406788a14dfe75e076...
2020-06-02T13:45:41Z [INFO] [/replication/transfer/image/transfer.go:284]: copy the blob sha256:1b99828eddf5297ca28fa7eac7f47a40d36a693c628311406788a14dfe75e076 completed
2020-06-02T13:45:41Z [INFO] [/replication/transfer/image/transfer.go:318]: pushing the manifest of artifact ubi8/ubi-minimal:sha256:fc7fc7bc6b293fd2ec1c1b035acc6b97ba34f0700c5366b699c646d947c39f81 ...
2020-06-02T13:45:41Z [INFO] [/replication/transfer/image/transfer.go:330]: the manifest of artifact ubi8/ubi-minimal:sha256:fc7fc7bc6b293fd2ec1c1b035acc6b97ba34f0700c5366b699c646d947c39f81 pushed
2020-06-02T13:45:41Z [INFO] [/replication/transfer/image/transfer.go:223]: copy ubi8/ubi-minimal:sha256:fc7fc7bc6b293fd2ec1c1b035acc6b97ba34f0700c5366b699c646d947c39f81(source registry) to ubi8/ubi-minimal:sha256:fc7fc7bc6b293fd2ec1c1b035acc6b97ba34f0700c5366b699c646d947c39f81(destination registry) completed
2020-06-02T13:45:41Z [INFO] [/replication/transfer/image/transfer.go:180]: copying ubi8/ubi-minimal:sha256:00308645ad3f0ee6384bffaab24d39b639c02742a536afa9aed97178e9654258(source registry) to ubi8/ubi-minimal:sha256:00308645ad3f0ee6384bffaab24d39b639c02742a536afa9aed97178e9654258(destination registry)...
2020-06-02T13:45:41Z [INFO] [/replication/transfer/image/transfer.go:293]: pulling the manifest of artifact ubi8/ubi-minimal:sha256:00308645ad3f0ee6384bffaab24d39b639c02742a536afa9aed97178e9654258 ...
2020-06-02T13:45:41Z [INFO] [/replication/transfer/image/transfer.go:299]: the manifest of artifact ubi8/ubi-minimal:sha256:00308645ad3f0ee6384bffaab24d39b639c02742a536afa9aed97178e9654258 pulled
2020-06-02T13:45:41Z [INFO] [/replication/transfer/image/transfer.go:257]: copying the blob sha256:3877646a67218a2db19b629e454a2370e8589d343d91252f9673f84ac5f9430f...
2020-06-02T13:45:43Z [INFO] [/replication/transfer/image/transfer.go:284]: copy the blob sha256:3877646a67218a2db19b629e454a2370e8589d343d91252f9673f84ac5f9430f completed
2020-06-02T13:45:43Z [INFO] [/replication/transfer/image/transfer.go:257]: copying the blob sha256:89eba1daa7492f02d8f3559086eab072f9e4a3002936dd5aa93c2e4517be4ea4...
2020-06-02T13:55:49Z [INFO] [/replication/transfer/image/transfer.go:284]: copy the blob sha256:89eba1daa7492f02d8f3559086eab072f9e4a3002936dd5aa93c2e4517be4ea4 completed
2020-06-02T13:55:49Z [INFO] [/replication/transfer/image/transfer.go:257]: copying the blob sha256:a3d418bf6534815fd6cc1398c2c38f4f1cde9ec472a96a604a9514e53587b8a5...
2020-06-02T13:55:51Z [INFO] [/replication/transfer/image/transfer.go:284]: copy the blob sha256:a3d418bf6534815fd6cc1398c2c38f4f1cde9ec472a96a604a9514e53587b8a5 completed
2020-06-02T13:55:51Z [INFO] [/replication/transfer/image/transfer.go:318]: pushing the manifest of artifact ubi8/ubi-minimal:sha256:00308645ad3f0ee6384bffaab24d39b639c02742a536afa9aed97178e9654258 ...
2020-06-02T13:55:51Z [INFO] [/replication/transfer/image/transfer.go:330]: the manifest of artifact ubi8/ubi-minimal:sha256:00308645ad3f0ee6384bffaab24d39b639c02742a536afa9aed97178e9654258 pushed
2020-06-02T13:55:51Z [INFO] [/replication/transfer/image/transfer.go:223]: copy ubi8/ubi-minimal:sha256:00308645ad3f0ee6384bffaab24d39b639c02742a536afa9aed97178e9654258(source registry) to ubi8/ubi-minimal:sha256:00308645ad3f0ee6384bffaab24d39b639c02742a536afa9aed97178e9654258(destination registry) completed
2020-06-02T13:55:51Z [INFO] [/replication/transfer/image/transfer.go:180]: copying ubi8/ubi-minimal:sha256:7e432c89f285392c7d09343a3100e97158121bd5f73b89c852eba9609e19f9f4(source registry) to ubi8/ubi-minimal:sha256:7e432c89f285392c7d09343a3100e97158121bd5f73b89c852eba9609e19f9f4(destination registry)...
2020-06-02T13:55:51Z [INFO] [/replication/transfer/image/transfer.go:293]: pulling the manifest of artifact ubi8/ubi-minimal:sha256:7e432c89f285392c7d09343a3100e97158121bd5f73b89c852eba9609e19f9f4 ...
2020-06-02T13:55:52Z [INFO] [/replication/transfer/image/transfer.go:299]: the manifest of artifact ubi8/ubi-minimal:sha256:7e432c89f285392c7d09343a3100e97158121bd5f73b89c852eba9609e19f9f4 pulled
2020-06-02T13:55:52Z [INFO] [/replication/transfer/image/transfer.go:257]: copying the blob sha256:0b093ba45a000d0018d70381553cdc8814bdf3b05b47f5a9976c610e335b8765...
2020-06-02T13:55:53Z [INFO] [/replication/transfer/image/transfer.go:284]: copy the blob sha256:0b093ba45a000d0018d70381553cdc8814bdf3b05b47f5a9976c610e335b8765 completed
2020-06-02T13:55:53Z [INFO] [/replication/transfer/image/transfer.go:257]: copying the blob sha256:84ede19b99598002e23de4c70698bb7631ee724fd537215476878141ce4c18c9...
2020-06-02T13:56:17Z [INFO] [/replication/transfer/image/transfer.go:284]: copy the blob sha256:84ede19b99598002e23de4c70698bb7631ee724fd537215476878141ce4c18c9 completed
2020-06-02T13:56:17Z [INFO] [/replication/transfer/image/transfer.go:257]: copying the blob sha256:b85e355b182585f0e146af4173b67fafaaea379ce5ac76aabe7462fffc374d2f...
2020-06-02T13:56:18Z [INFO] [/replication/transfer/image/transfer.go:284]: copy the blob sha256:b85e355b182585f0e146af4173b67fafaaea379ce5ac76aabe7462fffc374d2f completed
2020-06-02T13:56:18Z [INFO] [/replication/transfer/image/transfer.go:318]: pushing the manifest of artifact ubi8/ubi-minimal:sha256:7e432c89f285392c7d09343a3100e97158121bd5f73b89c852eba9609e19f9f4 ...
2020-06-02T13:56:18Z [INFO] [/replication/transfer/image/transfer.go:330]: the manifest of artifact ubi8/ubi-minimal:sha256:7e432c89f285392c7d09343a3100e97158121bd5f73b89c852eba9609e19f9f4 pushed
2020-06-02T13:56:18Z [INFO] [/replication/transfer/image/transfer.go:223]: copy ubi8/ubi-minimal:sha256:7e432c89f285392c7d09343a3100e97158121bd5f73b89c852eba9609e19f9f4(source registry) to ubi8/ubi-minimal:sha256:7e432c89f285392c7d09343a3100e97158121bd5f73b89c852eba9609e19f9f4(destination registry) completed
2020-06-02T13:56:18Z [INFO] [/replication/transfer/image/transfer.go:180]: copying ubi8/ubi-minimal:sha256:7bb4dacb1e5fb5ad0742578d616982f836778fd0c5a0a00f843e9a7b1dabf487(source registry) to ubi8/ubi-minimal:sha256:7bb4dacb1e5fb5ad0742578d616982f836778fd0c5a0a00f843e9a7b1dabf487(destination registry)...
2020-06-02T13:56:18Z [INFO] [/replication/transfer/image/transfer.go:293]: pulling the manifest of artifact ubi8/ubi-minimal:sha256:7bb4dacb1e5fb5ad0742578d616982f836778fd0c5a0a00f843e9a7b1dabf487 ...
2020-06-02T13:56:19Z [INFO] [/replication/transfer/image/transfer.go:299]: the manifest of artifact ubi8/ubi-minimal:sha256:7bb4dacb1e5fb5ad0742578d616982f836778fd0c5a0a00f843e9a7b1dabf487 pulled
2020-06-02T13:56:19Z [INFO] [/replication/transfer/image/transfer.go:257]: copying the blob sha256:625ee01cf1b59c96ce9d52bb94bb6f53b6839ae407a640149397928fff130794...
2020-06-02T13:56:20Z [INFO] [/replication/transfer/image/transfer.go:284]: copy the blob sha256:625ee01cf1b59c96ce9d52bb94bb6f53b6839ae407a640149397928fff130794 completed
2020-06-02T13:56:20Z [INFO] [/replication/transfer/image/transfer.go:257]: copying the blob sha256:f519bc5bf3ce7084efc43646b12e7351c08df4681f58a7c309e65805bfc15899...
2020-06-02T13:56:39Z [INFO] [/replication/transfer/image/transfer.go:284]: copy the blob sha256:f519bc5bf3ce7084efc43646b12e7351c08df4681f58a7c309e65805bfc15899 completed
2020-06-02T13:56:39Z [INFO] [/replication/transfer/image/transfer.go:257]: copying the blob sha256:1e374ff45d724f97c9640ee55195d91ef4fa84375ca75e3a3aeef8374fd5f012...
2020-06-02T13:56:41Z [INFO] [/replication/transfer/image/transfer.go:284]: copy the blob sha256:1e374ff45d724f97c9640ee55195d91ef4fa84375ca75e3a3aeef8374fd5f012 completed
2020-06-02T13:56:41Z [INFO] [/replication/transfer/image/transfer.go:318]: pushing the manifest of artifact ubi8/ubi-minimal:sha256:7bb4dacb1e5fb5ad0742578d616982f836778fd0c5a0a00f843e9a7b1dabf487 ...
2020-06-02T13:56:41Z [INFO] [/replication/transfer/image/transfer.go:330]: the manifest of artifact ubi8/ubi-minimal:sha256:7bb4dacb1e5fb5ad0742578d616982f836778fd0c5a0a00f843e9a7b1dabf487 pushed
2020-06-02T13:56:41Z [INFO] [/replication/transfer/image/transfer.go:223]: copy ubi8/ubi-minimal:sha256:7bb4dacb1e5fb5ad0742578d616982f836778fd0c5a0a00f843e9a7b1dabf487(source registry) to ubi8/ubi-minimal:sha256:7bb4dacb1e5fb5ad0742578d616982f836778fd0c5a0a00f843e9a7b1dabf487(destination registry) completed
2020-06-02T13:56:41Z [INFO] [/replication/transfer/image/transfer.go:318]: pushing the manifest of artifact ubi8/ubi-minimal:8.2 ...
2020-06-02T13:56:41Z [INFO] [/replication/transfer/image/transfer.go:330]: the manifest of artifact ubi8/ubi-minimal:8.2 pushed
2020-06-02T13:56:41Z [INFO] [/replication/transfer/image/transfer.go:223]: copy ubi8/ubi-minimal:8.2(source registry) to ubi8/ubi-minimal:8.2(destination registry) completed
2020-06-02T13:56:41Z [INFO] [/replication/transfer/image/transfer.go:174]: copy ubi8/ubi-minimal:[8.2](source registry) to ubi8/ubi-minimal:[8.2](destination registry) completed

ubi7minimal replication

2020-06-02T14:02:41Z [INFO] [/replication/transfer/image/transfer.go:116]: client for source registry [type: docker-registry, URL: http://registry.access.redhat.com, insecure: true] created
2020-06-02T14:02:41Z [INFO] [/replication/transfer/image/transfer.go:126]: client for destination registry [type: harbor, URL: http://core:8080, insecure: true] created
2020-06-02T14:02:41Z [INFO] [/replication/transfer/image/transfer.go:159]: copying ubi7/ubi-minimal:[7.8](source registry) to test/ubi-minimal:[7.8](destination registry)...
2020-06-02T14:02:41Z [INFO] [/replication/transfer/image/transfer.go:180]: copying ubi7/ubi-minimal:7.8(source registry) to test/ubi-minimal:7.8(destination registry)...
2020-06-02T14:02:41Z [INFO] [/replication/transfer/image/transfer.go:293]: pulling the manifest of artifact ubi7/ubi-minimal:7.8 ...
2020-06-02T14:02:42Z [INFO] [/replication/transfer/image/transfer.go:299]: the manifest of artifact ubi7/ubi-minimal:7.8 pulled
2020-06-02T14:02:42Z [INFO] [/replication/transfer/image/transfer.go:180]: copying ubi7/ubi-minimal:sha256:93888b8d94483afb3c9a142c7e32c9f673150da028c9ae663eb54b1f7170f5b6(source registry) to test/ubi-minimal:sha256:93888b8d94483afb3c9a142c7e32c9f673150da028c9ae663eb54b1f7170f5b6(destination registry)...
2020-06-02T14:02:42Z [INFO] [/replication/transfer/image/transfer.go:293]: pulling the manifest of artifact ubi7/ubi-minimal:sha256:93888b8d94483afb3c9a142c7e32c9f673150da028c9ae663eb54b1f7170f5b6 ...
2020-06-02T14:02:43Z [INFO] [/replication/transfer/image/transfer.go:299]: the manifest of artifact ubi7/ubi-minimal:sha256:93888b8d94483afb3c9a142c7e32c9f673150da028c9ae663eb54b1f7170f5b6 pulled
2020-06-02T14:02:43Z [INFO] [/replication/transfer/image/transfer.go:257]: copying the blob sha256:7d397bf59678872fa4f8074057fb96723eaba24629b915997576e9addde17844...
2020-06-02T14:02:43Z [INFO] [/replication/transfer/image/transfer.go:284]: copy the blob sha256:7d397bf59678872fa4f8074057fb96723eaba24629b915997576e9addde17844 completed
2020-06-02T14:02:43Z [INFO] [/replication/transfer/image/transfer.go:257]: copying the blob sha256:3aa8b87b7f88ed441a44c1fcafdcc80e48dd5b7e0f3d24961941c95b5470c5fd...
2020-06-02T14:02:47Z [INFO] [/replication/transfer/image/transfer.go:284]: copy the blob sha256:3aa8b87b7f88ed441a44c1fcafdcc80e48dd5b7e0f3d24961941c95b5470c5fd completed
2020-06-02T14:02:47Z [INFO] [/replication/transfer/image/transfer.go:257]: copying the blob sha256:57584b59d88badb977c5077949af3d65f9b9f864d1c9f295a97aed0ab555e865...
2020-06-02T14:02:48Z [INFO] [/replication/transfer/image/transfer.go:284]: copy the blob sha256:57584b59d88badb977c5077949af3d65f9b9f864d1c9f295a97aed0ab555e865 completed
2020-06-02T14:02:48Z [INFO] [/replication/transfer/image/transfer.go:318]: pushing the manifest of artifact test/ubi-minimal:sha256:93888b8d94483afb3c9a142c7e32c9f673150da028c9ae663eb54b1f7170f5b6 ...
2020-06-02T14:02:48Z [INFO] [/replication/transfer/image/transfer.go:330]: the manifest of artifact test/ubi-minimal:sha256:93888b8d94483afb3c9a142c7e32c9f673150da028c9ae663eb54b1f7170f5b6 pushed
2020-06-02T14:02:48Z [INFO] [/replication/transfer/image/transfer.go:223]: copy ubi7/ubi-minimal:sha256:93888b8d94483afb3c9a142c7e32c9f673150da028c9ae663eb54b1f7170f5b6(source registry) to test/ubi-minimal:sha256:93888b8d94483afb3c9a142c7e32c9f673150da028c9ae663eb54b1f7170f5b6(destination registry) completed
2020-06-02T14:02:48Z [INFO] [/replication/transfer/image/transfer.go:180]: copying ubi7/ubi-minimal:sha256:447f34ef29bdf72edb267c014843baaacaee80746eeecb4f0b0ff1a47d4ce1ad(source registry) to test/ubi-minimal:sha256:447f34ef29bdf72edb267c014843baaacaee80746eeecb4f0b0ff1a47d4ce1ad(destination registry)...
2020-06-02T14:02:48Z [INFO] [/replication/transfer/image/transfer.go:293]: pulling the manifest of artifact ubi7/ubi-minimal:sha256:447f34ef29bdf72edb267c014843baaacaee80746eeecb4f0b0ff1a47d4ce1ad ...
2020-06-02T14:02:48Z [INFO] [/replication/transfer/image/transfer.go:299]: the manifest of artifact ubi7/ubi-minimal:sha256:447f34ef29bdf72edb267c014843baaacaee80746eeecb4f0b0ff1a47d4ce1ad pulled
2020-06-02T14:02:48Z [INFO] [/replication/transfer/image/transfer.go:257]: copying the blob sha256:9c4ea4ecee66d85b06fa1ebd01caffeb4475c08dd30b9968a3fc77c73e7faec1...
2020-06-02T14:02:50Z [INFO] [/replication/transfer/image/transfer.go:284]: copy the blob sha256:9c4ea4ecee66d85b06fa1ebd01caffeb4475c08dd30b9968a3fc77c73e7faec1 completed
2020-06-02T14:02:50Z [INFO] [/replication/transfer/image/transfer.go:257]: copying the blob sha256:f74522a6e184211db4aa0b9321497a47c75255b84b0cfaacaa08c4548b1c7a0a...
2020-06-02T14:03:13Z [INFO] [/replication/transfer/image/transfer.go:284]: copy the blob sha256:f74522a6e184211db4aa0b9321497a47c75255b84b0cfaacaa08c4548b1c7a0a completed
2020-06-02T14:03:13Z [INFO] [/replication/transfer/image/transfer.go:257]: copying the blob sha256:964e2589db09868b1c21b6b3546266bdd0bf59275c7275be0215dc5a86d55d1e...
2020-06-02T14:03:14Z [INFO] [/replication/transfer/image/transfer.go:284]: copy the blob sha256:964e2589db09868b1c21b6b3546266bdd0bf59275c7275be0215dc5a86d55d1e completed
2020-06-02T14:03:14Z [INFO] [/replication/transfer/image/transfer.go:318]: pushing the manifest of artifact test/ubi-minimal:sha256:447f34ef29bdf72edb267c014843baaacaee80746eeecb4f0b0ff1a47d4ce1ad ...
2020-06-02T14:03:14Z [INFO] [/replication/transfer/image/transfer.go:330]: the manifest of artifact test/ubi-minimal:sha256:447f34ef29bdf72edb267c014843baaacaee80746eeecb4f0b0ff1a47d4ce1ad pushed
2020-06-02T14:03:14Z [INFO] [/replication/transfer/image/transfer.go:223]: copy ubi7/ubi-minimal:sha256:447f34ef29bdf72edb267c014843baaacaee80746eeecb4f0b0ff1a47d4ce1ad(source registry) to test/ubi-minimal:sha256:447f34ef29bdf72edb267c014843baaacaee80746eeecb4f0b0ff1a47d4ce1ad(destination registry) completed
2020-06-02T14:03:14Z [INFO] [/replication/transfer/image/transfer.go:180]: copying ubi7/ubi-minimal:sha256:0c9dc5c44e4dd74c72694abf61cc50a8113a149d9428464b3d4897572ab1447f(source registry) to test/ubi-minimal:sha256:0c9dc5c44e4dd74c72694abf61cc50a8113a149d9428464b3d4897572ab1447f(destination registry)...
2020-06-02T14:03:14Z [INFO] [/replication/transfer/image/transfer.go:293]: pulling the manifest of artifact ubi7/ubi-minimal:sha256:0c9dc5c44e4dd74c72694abf61cc50a8113a149d9428464b3d4897572ab1447f ...
2020-06-02T14:03:14Z [INFO] [/replication/transfer/image/transfer.go:299]: the manifest of artifact ubi7/ubi-minimal:sha256:0c9dc5c44e4dd74c72694abf61cc50a8113a149d9428464b3d4897572ab1447f pulled
2020-06-02T14:03:14Z [INFO] [/replication/transfer/image/transfer.go:257]: copying the blob sha256:521b2689e58ef0eff00c23c7b005ba08c3bda2af69ccb34139d986e8601a0133...
2020-06-02T14:03:15Z [INFO] [/replication/transfer/image/transfer.go:284]: copy the blob sha256:521b2689e58ef0eff00c23c7b005ba08c3bda2af69ccb34139d986e8601a0133 completed
2020-06-02T14:03:15Z [INFO] [/replication/transfer/image/transfer.go:257]: copying the blob sha256:f871b6fadcaf13b16fcdcf479ec6957efdd2cfab5045ffc6e672bda56cd66beb...
2020-06-02T14:03:29Z [INFO] [/replication/transfer/image/transfer.go:284]: copy the blob sha256:f871b6fadcaf13b16fcdcf479ec6957efdd2cfab5045ffc6e672bda56cd66beb completed
2020-06-02T14:03:29Z [INFO] [/replication/transfer/image/transfer.go:257]: copying the blob sha256:0561557f14573b22d9fd9273f8fb98e73bcdfaa39901e8c601510c417b224a88...
2020-06-02T14:03:30Z [INFO] [/replication/transfer/image/transfer.go:284]: copy the blob sha256:0561557f14573b22d9fd9273f8fb98e73bcdfaa39901e8c601510c417b224a88 completed
2020-06-02T14:03:30Z [INFO] [/replication/transfer/image/transfer.go:318]: pushing the manifest of artifact test/ubi-minimal:sha256:0c9dc5c44e4dd74c72694abf61cc50a8113a149d9428464b3d4897572ab1447f ...
2020-06-02T14:03:30Z [INFO] [/replication/transfer/image/transfer.go:330]: the manifest of artifact test/ubi-minimal:sha256:0c9dc5c44e4dd74c72694abf61cc50a8113a149d9428464b3d4897572ab1447f pushed
2020-06-02T14:03:30Z [INFO] [/replication/transfer/image/transfer.go:223]: copy ubi7/ubi-minimal:sha256:0c9dc5c44e4dd74c72694abf61cc50a8113a149d9428464b3d4897572ab1447f(source registry) to test/ubi-minimal:sha256:0c9dc5c44e4dd74c72694abf61cc50a8113a149d9428464b3d4897572ab1447f(destination registry) completed
2020-06-02T14:03:30Z [INFO] [/replication/transfer/image/transfer.go:318]: pushing the manifest of artifact test/ubi-minimal:7.8 ...
2020-06-02T14:03:30Z [INFO] [/replication/transfer/image/transfer.go:330]: the manifest of artifact test/ubi-minimal:7.8 pushed
2020-06-02T14:03:30Z [INFO] [/replication/transfer/image/transfer.go:223]: copy ubi7/ubi-minimal:7.8(source registry) to test/ubi-minimal:7.8(destination registry) completed
2020-06-02T14:03:30Z [INFO] [/replication/transfer/image/transfer.go:174]: copy ubi7/ubi-minimal:[7.8](source registry) to test/ubi-minimal:[7.8](destination registry) completed

[steven-zou]

@sluetze

Thanks for helping to do the verification. About the perf problem, do you happen to find any clues about it?

[sluetze]

No, it also did only happen with some of the images, but at ~ the same time. I guess it was something in my network. The log (see above) seems to show a 10Minute timeslot, where nothing was going on:

2020-06-02T13:45:43Z [INFO] [/replication/transfer/image/transfer.go:257]: copying the blob sha256:89eba1daa7492f02d8f3559086eab072f9e4a3002936dd5aa93c2e4517be4ea4...
2020-06-02T13:55:49Z [INFO] [/replication/transfer/image/transfer.go:284]:

Since I setup the environment only for the test (Create VM, Compile Harbor, Test, Delete VM) and there were some additional changes ongoing during that time, I am quite sure it was some networking stuff in my environment and nothing in Harbor. Also the nearly exact 10 Minutes seems to be something timeout related.

Edit: Misstyped network as needwork... what a great typo...

[steven-zou]

@sluetze

Thanks a lot for your confirmation and clarification. As this issue has been fixed, I'll close it now.