Harbor: Fatal error after upgrade 1.8.2 to 1.9.0

[CarTerr]

After upgrade from 1.8.2 to 1.9.0 I get fatal error in harbor-core
mesage error:

Sep 29 04:09:28 172.20.0.1 core[6836]: 2019-09-29T04:09:28Z [ERROR] [/core/api/quota/registry/registry.go:388]: http error: code 404, message {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Name":"mybg/web","Revision":"sha256:0e4943fc36e4fc6b29561d46f326827ce1da42844fa1ff686322a3300abb8990"}}]}
Sep 29 04:09:28 172.20.0.1 core[6836]: 2019-09-29T04:09:28Z [ERROR] [/core/api/quota/registry/registry.go:388]: http error: code 404, message {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Name":"shb/shb_patches","Revision":"sha256:ef0dfc36fe1f0e25eb5260d12c57e8b441f0c6387ecba520aae619203a5f2699"}}]}
Sep 29 04:09:31 172.20.0.1 core[6836]: 2019-09-29T04:09:31Z [INFO] [/core/api/quota/migrator.go:101]: [Quota-Sync]:: fail to dump data from server ... [registry], quit sync ...
Sep 29 04:09:31 172.20.0.1 core[6836]: 2019-09-29T04:09:31Z [ERROR] [/core/main.go:104]: Fail to sync quota data, quota sync error on getting info of project: quota sync error on getting info of repo: http error: code 404, message {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Name":"mybg/web","Revision":"sha256:0e4943fc36e4fc6b29561d46f326827ce1da42844fa1ff686322a3300abb8990"}}]}
Sep 29 04:09:31 172.20.0.1 core[6836]: : quota sync error on getting info of repo: http error: code 404, message {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Name":"shb/shb_patches","Revision":"sha256:ef0dfc36fe1f0e25eb5260d12c57e8b441f0c6387ecba520aae619203a5f2699"}}]}
Sep 29 04:09:31 172.20.0.1 core[6836]: 2019-09-29T04:09:31Z [FATAL] [/core/main.go:252]: quota migration error, quota sync error on getting info of project: quota sync error on getting info of repo: http error: code 404, message {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Name":"mybg/web","Revision":"sha256:0e4943fc36e4fc6b29561d46f326827ce1da42844fa1ff686322a3300abb8990"}}]}
Sep 29 04:09:31 172.20.0.1 core[6836]: : quota sync error on getting info of repo: http error: code 404, message {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Name":"shb/shb_patches","Revision":"sha256:ef0dfc36fe1f0e25eb5260d12c57e8b441f0c6387ecba520aae619203a5f2699"}}]}

logs.zip

[BertrandGouny]

hello,
i get ride of the error going in the registry container and deleting the tag directory in
/storage/docker/registry/v2/repositories/xxxx/yyyyyy/_manifests/tags/zzzzzz

where the content of /storage/docker/registry/v2/repositories/xxxx/yyyyyy/_manifests/tags/zzzzzz/current/link
match the sha256 in the error message

[Mr-iX]

hello,
i get ride of the error going in the registry container and deleting the tag directory in
/storage/docker/registry/v2/repositories/xxxx/yyyyyy/_manifests/tags/zzzzzz

where the content of /storage/docker/registry/v2/repositories/xxxx/yyyyyy/_manifests/tags/zzzzzz/current/link
match the sha256 in the error message

I have already deleted them, but still the same error

[CarTerr]

Hello,
Thanks to Bertrand Guney, it helped.
The question remains, why this problem arose only in 1.9.0

Mr-iX, in the log we see only fist error of problem repository tag.
There may be several such errors. I had about 30 spoiled repository tag.
At each start, the sha256 number changed

[CarTerr]

Sep 29 04:09:31 172.20.0.1 core[6836]: 2019-09-29T04:09:31Z [FATAL] [/core/main.go:252]: quota migration error, quota sync error on getting info of project: quota sync error on getting info of repo: http error: code 404, message {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Name":"mybg/web","Revision":"sha256:0e4943fc36e4fc6b29561d46f326827ce1da42844fa1ff686322a3300abb8990"}}]}

Repository mybg/web

cd /data/registry/docker/registry/v2/repositories/mybg/web/_manifests/tags/
grep -rl 'sha256:0e4943fc36e4fc6b29561d46f326827ce1da42844fa1ff686322a3300abb8990' ./*
rm -R ./number_tag/

[Mr-iX]

The log doesn’t show sha256 numbers, because I deleted all of these tags.
After changing the log level to debug, I saw which get requests the quota migrator initiates. After deleting some more tags it still fails, but there shouldn’t be any OCI Image anymore.

But it’s not good that harbor accepts OCI images and then only fails when trying to access them. The API shouldn’t allow OCI images or we need an possibility to delete all OCI images from our harbor

[xaleeks]

'But it’s not good that harbor accepts OCI images and then only fails when trying to access them. The API shouldn’t allow OCI images or we need an possibility to delete all OCI images from our harbor'.

we're working on a mechanism to gate this properly. This is also being tracked for the next 1.9.2 branch release.

[zhiqiangchine2x]

Deleting the tags mentioned by @BertrandGouny solved this for me, i had initially tested uploading podman images to Harbor during 1.8.2 , and these images do not appear in harbor, however you could still pull them.

Therefore i went and deleted the podman pushed images.

[wy65701436]

@CarTerr @Mr-iX I have patched a image with the fix, please let me know if you'd like to have a try.

[wy65701436]

Close it as it's already fixed.