Harbor: Strange behaviour when deleting an repo after GC

[wy65701436]

2, Delete the repository.
3, Run GC with cmd: /usr/bin/registry garbage-collect /etc/registry/config.yml
4, Re-push the deleted images into harbor.

After that, I get the repo size with 0. And, I tried to delete it, UI shows as success and 200 from API, but actually the repo cannot be deleted.

[Root Cause]
After investigation, the root cause should be the GC bug of docker registry v2.6.2(cache invalidation). See issues:
https://github.com/docker/distribution/issues/2270
https://github.com/docker/distribution/issues/2094
https://github.com/docker/distribution/issues/2434
From the issue response, they could provide fix on 2.7, but no mention the schedule.

[Workaround]
Re-start registry and re-push the images.

[Issue on harbor]
For harbor, we eat the 404 error from docker registry, that's why it can get 200 from API when to delete the repository and cause the strange behaviour. https://github.com/vmware/harbor/blob/master/src/ui/api/repository.go#L295

[Log]
Jun 1 08:38:00 172.18.0.1 ui[1463]: 2018-06-01T15:38:00Z [ERROR] [repository.go:538]: failed to get v2 manifest of library1/tomcat:latest: 404 {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Name":"library1/tomcat","Revision":"sha256:8238fb3b0f8e15219399069a4aa78e88cbbce28834de5f1e55749067b82d14dc"}}]}

[ghost]

@reasonerjt – I know we talked about this but can't recall if it was completely fixed in #5273. Thoughts?

[wy65701436]

from v1.6.1, harbor has upgraded to use redis as the cache of registry, and the GC job will delete all the keys from cache, which could workaround this issue.

Close it as fix.

[lmdkfs]

i also encountered this problem, so i upgrade harbor to 1.6.1, but this problem still exists, how can to use gc, thanks @wy65701436

[louyiping]

Also meet this problem with 1.6.0 version. Any idea?

[wy65701436]

The workaround on v1.6.1 is to clean the keys of the registry in redis, this could workaround the cache invalidation issue on registry v2.6.2.

Note:
Before doing the deletion, please make sure you're on the right DB. The prefix for the registry are "blobs::" and "repository::", just run "keys *" to double check.

Examples:

 16) "repository::library/redis::blobs::sha256:d950f81a04072e2ca4502cfb6d47170c88f95fd6fa8661af2493929053107968"
 17) "repository::project011547413033/haproxy::blobs::sha256:ec718aacd304d20d3dc4c5d118ce0fc555633ab5fd97c08d5e789a6034138713"
 18) "repository::project011547412030/busybox::blobs::sha256:571469a4529d10ff26e2d93dcc788c7787901656f0af6236d6b3b98f02f75d01"
 19) "repository::project011547412781/tomcat::blobs::sha256:da938b2e339022cd8fb5785dcdd0f6a0f88d9f9eef8d9301497a0cf2bb65b1d5"
 20) "repository::project011547412030/hello-world::blobs"
 21) "blobs::sha256:e6faa08065edd437a989f5afa3efec256305ca8d83dd87a0b30ad55b31c276a3"
 22) "blobs::sha256:55dccdb0333e28a0165f711f9d101f01102a5f474353e1e003db77931a8d80b0"

1, Go into the container -- goharbor/redis-photon
2, Run 'redis-cli'
3, Run 'select 1'  (1 is the default DB index of registry)
4, Run 'flushdb'
5, User 'keys *' to validate all keys have been removed.