Harbor: Manifest for image <IMAGE> is not ready, retry after... inifinite loop

[jkjell]

If you are reporting a problem, please make sure the following information are provided:

Expected behavior and actual behavior:
Expected: An image pushed with no tag is received by Harbor (may not be visible in the UI due to existing bug). No additional retries or error messages in the logs.
Actual: An image pushed with no tag shows the logs continually retry a push event even after the image has been received by the registry.

Steps to reproduce the problem:
We found this issue by performing a duffle relocate command of a thick CNAB bundle that had an invocation image without an image tag. The duffle command is using image-relocation which uses go-containerregistry to push the images to Harbor. The image was built with https://github.com/vito/oci-build-task

I'm still working on building an individual image that can reproduce the issue but, raising now to see if it's a known issue or can be determined without that image.

Versions:
Please specify the versions of following systems.

• harbor version: [1.9.3]
• docker engine version: [19.03.5]
• helm chart version: [1.2.2]

Additional context:
While this error is occurring, no additional image pushes received will show in the Harbor UI. Upon restart of the registry pod the issue seems to stop.

Attaching Harbor Core and Registry log files.
core.log
registry.log

[MaGaoju]

I have this issue on my harbor instance.

1. harbor version: install harbor via Helm chart dev tag in master branch: Version v1.10.0-eba1a01a .
2. core and registry logs:
   corelog.txt
   registrylog.txt

[MaGaoju]

one addition thing on my instance, once we have this error. When user push image to harbor, it works with command, but pushed images will not show up on harbor UI... it happened twice on my side already, I need to clean those infinity loops.
UI log keep update as below: there are a lot logs generated by this already.

[cafeliker]

It's a very annoying issue, customers complain that they pushed the docker images but couldn't view them in the portal UI. It should be a critical issue.

When we are checking the project summary, it shows there are 5 artifacts and use 600MB disk space, but the repository shows none.

[cafeliker]

@jkjell do you have any workaround on this issue?

[jkjell]

@cafeliker My current workaround is to just restart the registry container.

[cafeliker]

@reasonerjt could you pls help check this issue? thanks

[reasonerjt]

@cafeliker Are you seeing this error also when handling CNAB?
What command did your user use to push the image?
I don't think it's the same problem as @jkjell described.

[MaGaoju]

@cafeliker Are you seeing this error also when handling CNAB?
What command did your user use to push the image?
I don't think it's the same problem as @jkjell described.

user using Jenkins job pulls a base "riak" docker image from harbor, adds in some config, then pushes this new image back to harbor with a different label.
I don't know if this image handing CNAB or not , but I can see exactly the same error in our log below... if you need more information about image it self , let me know.

2019-12-18T05:47:51Z [DEBUG] [/core/service/notifications/registry/handler.go:252]: receive an event:
----ID: 068d5ac6-7860-4faf-8204-d34067c9df6e
----target: rnd-webpress-dfe/riak:6.0.0.1.ae387eb
----digest: sha256:1293b6b87631af861c08f7b11f86ce84ed5731705db3a165db3f84d7109f9bf5
----action: push
----mediatype: application/vnd.docker.distribution.manifest.v2+json
----user-agent: docker/18.06.1-ce go/go1.10.3 kernel/4.14.146-119.123.amzn2.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/18.06.1-ce (linux))
2019-12-18T05:47:51Z [DEBUG] [/core/service/notifications/registry/handler.go:267]: add event to collection: 068d5ac6-7860-4faf-8204-d34067c9df6e
2019-12-18T05:47:51Z [DEBUG] [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: repository:rnd-webpress-dfe/riak:pull
2019-12-18T05:47:51Z [DEBUG] [/common/utils/registry/transport.go:51]: 404 | HEAD http://harbor-pro-1215-harbor-registry:5000/v2/rnd-webpress-dfe/riak/manifests/6.0.0.1.ae387eb
2019-12-18T05:47:51Z [WARNING] [/core/utils/utils.go:74]: manifest for image rnd-webpress-dfe/riak:6.0.0.1.ae387eb is not ready, retry after 80ms
2019-12-18T05:47:51Z [DEBUG] [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: repository:rnd-webpress-dfe/riak:pull
2019-12-18T05:47:51Z [DEBUG] [/common/utils/registry/auth/tokenauthorizer.go:239]: get token for scope repository:rnd-webpress-dfe/riak:pull from cache
2019-12-18T05:47:51Z [DEBUG] [/common/utils/registry/transport.go:51]: 404 | HEAD http://harbor-pro-1215-harbor-registry:5000/v2/rnd-webpress-dfe/riak/manifests/6.0.0.1.ae387eb
2019-12-18T05:47:51Z [WARNING] [/core/utils/utils.go:74]: manifest for image rnd-webpress-dfe/riak:6.0.0.1.ae387eb is not ready, retry after 200ms
2019-12-18T05:47:51Z [DEBUG] [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: repository:rnd-webpress-dfe/riak:pull
2019-12-18T05:47:51Z [DEBUG] [/common/utils/registry/auth/tokenauthorizer.go:239]: get token for scope repository:rnd-webpress-dfe/riak:pull from cache
2019-12-18T05:47:51Z [DEBUG] [/common/utils/registry/transport.go:51]: 404 | HEAD http://harbor-pro-1215-harbor-registry:5000/v2/rnd-webpress-dfe/riak/manifests/6.0.0.1.ae387eb
2019-12-18T05:47:51Z [WARNING] [/core/utils/utils.go:74]: manifest for image rnd-webpress-dfe/riak:6.0.0.1.ae387eb is not ready, retry after 500ms
2019-12-18T05:47:51Z [DEBUG] [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: repository:rnd-webpress-dfe/riak:pull
2019-12-18T05:47:51Z [DEBUG] [/common/utils/registry/auth/tokenauthorizer.go:239]: get token for scope repository:rnd-webpress-dfe/riak:pull from cache
2019-12-18T05:47:51Z [DEBUG] [/common/utils/registry/transport.go:51]: 404 | HEAD http://harbor-pro-1215-harbor-registry:5000/v2/rnd-webpress-dfe/riak/manifests/6.0.0.1.ae387eb
2019-12-18T05:47:51Z [WARNING] [/core/utils/utils.go:74]: manifest for image rnd-webpress-dfe/riak:6.0.0.1.ae387eb is not ready, retry after 1.25s
2019-12-18T05:47:51Z [ERROR] [/common/config/manager.go:192]: failed to get key oidc_groups_claim, error: the configure value is not set
2019-12-18T05:47:52Z [DEBUG] [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: repository:rnd-webpress-dfe/riak:pull
2019-12-18T05:47:52Z [DEBUG] [/common/utils/registry/auth/tokenauthorizer.go:239]: get token for scope repository:rnd-webpress-dfe/riak:pull from cache
2019-12-18T05:47:52Z [DEBUG] [/common/utils/registry/transport.go:51]: 404 | HEAD http://harbor-pro-1215-harbor-registry:5000/v2/rnd-webpress-dfe/riak/manifests/6.0.0.1.ae387eb
2019-12-18T05:47:52Z [ERROR] [/core/service/notifications/registry/handler.go:119]: Manifest for image rnd-webpress-dfe/riak:6.0.0.1.ae387eb is not ready, skip the follow up actions.
2019/12/18 05:47:52 [D] [server.go:2774] | 172.20.234.70| 200 | 5.265910789s|

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[tianon]

This is definitely still relevant on Harbor v1.10.1 (I've got an instance that's been looping all night).

It was looping with a variation of [WARNING] [/core/utils/utils.go:74]: manifest for image IMAGE: is not ready, retry after X.XXs (with varying values of wait time at the end), then when I restarted the registry (as suggested above) it stopped looping with [ERROR] [/core/service/notifications/registry/handler.go:119]: Manifest for image IMAGE: is not ready, skip the follow up actions..

I was then able to trivially reproduce by using skopeo (https://github.com/containers/skopeo) to do a skopeo copy --all to try copying a manifest list from Docker Hub into the instance and it immediately started looping again (skopeo copy --all --dest-creds harbor-user:harbor-password docker://docker.io/library/bash:latest docker://harbor-instance/library/bash:latest for example).

[jkjell]

From what we've seen in our experimentation with Harbor 2.0 the issue should be fixed then. However, we were mostly seeing this issue with untagged images. I can give the skopeo command a shot and see if that's fixed on Harbor 2.0 as well. I'll admit that doesn't help out on Harbor 1.10 though.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.