Harbor: Wired tag creation time, CAN NOT be deleted!

[moooofly]

Version of docker engine and docker-compose.

# docker --version
Docker version 17.09.0-ce, build afdb6d4
# docker-compose --version
docker-compose version 1.16.1, build 6d1ac21

Version of Harbor

1.2.2

---

one day, I found some tags in different repos with wired creation time 0001-01-01T00:00:00Z. From harbor UI, you can see:

I'd tried deleting this tag from UI, but failed, although "Deleted tag successfully." showing above.

I'd tried deleting this tag from console, but failed either. This tag is still there.

so, how can I deal with this problem? Any suggestion?

[moooofly]

FYI, not just this repo, some others have the same problem.

[steven-zou]

@ywk253100 please work together with @ninjadq to locate the root cause of this issue. Thanks

[reasonerjt]

@moooofly have you run GC?
This may be related to: https://github.com/vmware/harbor/issues/5078

[moooofly]

I did not run GC.
I'll checkout this out tomorrow.

[moooofly]

After running GC last night, I find tags with wired timestamp is still there. @reasonerjt

root@harbor-service-new-stag:/opt/apps/harbor# docker-compose stop
Stopping nginx              ... done
Stopping harbor-jobservice  ... done
Stopping harbor-ui          ... done
Stopping harbor-adminserver ... done
Stopping registry           ... done
Stopping harbor-db          ... done
Stopping harbor-log         ... done
root@harbor-service-new-stag:/opt/apps/harbor#
root@harbor-service-new-stag:/opt/apps/harbor# docker-compose ps
       Name                     Command                State     Ports
----------------------------------------------------------------------
harbor-adminserver   /harbor/harbor_adminserver       Exit 2
harbor-db            docker-entrypoint.sh mysqld      Exit 0
harbor-jobservice    /harbor/harbor_jobservice        Exit 2
harbor-log           /bin/sh -c crond && rm -f  ...   Exit 137
harbor-ui            /harbor/harbor_ui                Exit 2
nginx                nginx -g daemon off;             Exit 137
registry             /entrypoint.sh serve /etc/ ...   Exit 2
root@harbor-service-new-stag:/opt/apps/harbor#

root@harbor-service-new-stag:/opt/apps/harbor#
root@harbor-service-new-stag:/opt/apps/harbor# docker run -it --name gc --rm --volumes-from registry vmware/registry:2.6.2-photon garbage-collect  /etc/registry/config.yml
DEBU[0000] s3aws.List("/docker/registry/v2/repositories")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=27.493644ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).List trace.id=a23d3cdc-825c-41c3-99aa-355038d85783 trace.line=150
DEBU[0000] s3aws.Stat("/docker/registry/v2/repositories/algorithm")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=15.316684ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=520fb6fc-30bd-49d3-b673-25db7e04bc21 trace.line=137
DEBU[0000] s3aws.List("/docker/registry/v2/repositories/algorithm")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=12.225268ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).List trace.id=fa6e5957-e239-42a0-8c44-7382b6600246 trace.line=150
DEBU[0000] s3aws.Stat("/docker/registry/v2/repositories/algorithm/amplex-anno")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=11.354298ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=4c735843-80b7-4fc3-a4ce-02844f43a8c9 trace.line=137
DEBU[0000] s3aws.List("/docker/registry/v2/repositories/algorithm/amplex-anno")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=12.76186ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).List trace.id=9e76b086-d2e2-4805-b9f8-3f2ea380729f trace.line=150
DEBU[0000] s3aws.Stat("/docker/registry/v2/repositories/algorithm/amplex-anno/_layers")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=12.110443ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=bd99cb80-656e-4033-b251-f4eddae6860a trace.line=137
algorithm/amplex-anno
DEBU[0000] s3aws.List("/docker/registry/v2/repositories/algorithm/amplex-anno/_manifests/revisions")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=11.148381ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).List trace.id=1d5c8336-0841-4a58-8672-63f1c80d116b trace.line=150
DEBU[0000] s3aws.Stat("/docker/registry/v2/repositories/algorithm/amplex-anno/_manifests/revisions/sha256")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=10.12448ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=d41ce209-6223-4033-875e-5e0170ad2cdc trace.line=137
DEBU[0000] s3aws.List("/docker/registry/v2/repositories/algorithm/amplex-anno/_manifests/revisions/sha256")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=12.05107ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).List trace.id=f49ed1ad-76d8-4023-9465-14ac5ed68cce trace.line=150
DEBU[0000] s3aws.Stat("/docker/registry/v2/repositories/algorithm/amplex-anno/_manifests/revisions/sha256/227889fd2a25d6f7a4022b4f84e3033b4cf67fbf0f349aa375b490b8681f5f86")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=10.608116ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=059f5617-2077-45ba-8453-af2a648ce02d trace.line=137
DEBU[0000] s3aws.List("/docker/registry/v2/repositories/algorithm/amplex-anno/_manifests/revisions/sha256/227889fd2a25d6f7a4022b4f84e3033b4cf67fbf0f349aa375b490b8681f5f86")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=10.266212ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).List trace.id=849af0d0-a3ef-4c69-9b37-88d742f9b499 trace.line=150
DEBU[0000] s3aws.Stat("/docker/registry/v2/repositories/algorithm/amplex-anno/_manifests/revisions/sha256/227889fd2a25d6f7a4022b4f84e3033b4cf67fbf0f349aa375b490b8681f5f86/link")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=10.892151ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=17836e19-bb74-430d-9b5d-b5c1c307a3be trace.line=137
DEBU[0000] s3aws.GetContent("/docker/registry/v2/repositories/algorithm/amplex-anno/_manifests/revisions/sha256/227889fd2a25d6f7a4022b4f84e3033b4cf67fbf0f349aa375b490b8681f5f86/link")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=22.91899ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent trace.id=d9d9fcd0-1fd6-4bf8-add4-28bf95fbeb87 trace.line=82
DEBU[0000] s3aws.GetContent("/docker/registry/v2/repositories/algorithm/amplex-anno/_manifests/revisions/sha256/227889fd2a25d6f7a4022b4f84e3033b4cf67fbf0f349aa375b490b8681f5f86/link")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=11.921751ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent trace.id=cfd1e79f-87ca-430d-90bc-6d38978ba1dc trace.line=82
DEBU[0000] s3aws.Stat("/docker/registry/v2/blobs/sha256/22/227889fd2a25d6f7a4022b4f84e3033b4cf67fbf0f349aa375b490b8681f5f86/data")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=12.562627ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=99bc572a-aa84-4395-a240-f415747dee12 trace.line=137
algorithm/amplex-anno: marking manifest sha256:227889fd2a25d6f7a4022b4f84e3033b4cf67fbf0f349aa375b490b8681f5f86
DEBU[0000] (*manifestStore).Get                          go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry
DEBU[0000] s3aws.GetContent("/docker/registry/v2/repositories/algorithm/amplex-anno/_manifests/revisions/sha256/227889fd2a25d6f7a4022b4f84e3033b4cf67fbf0f349aa375b490b8681f5f86/link")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=12.093442ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent trace.id=3d48400f-dc23-4669-ac83-39e6c4a933ac trace.line=82
DEBU[0000] s3aws.Stat("/docker/registry/v2/blobs/sha256/22/227889fd2a25d6f7a4022b4f84e3033b4cf67fbf0f349aa375b490b8681f5f86/data")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=12.102964ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=6d19d779-257c-4343-92d5-c77a4c68d569 trace.line=137
DEBU[0000] s3aws.Reader("/docker/registry/v2/blobs/sha256/22/227889fd2a25d6f7a4022b4f84e3033b4cf67fbf0f349aa375b490b8681f5f86/data", 0)  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=30.285923ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Reader trace.id=462767e9-db42-429a-babd-37c356e46ad1 trace.line=107
DEBU[0000] (*schema2ManifestHandler).Unmarshal           go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry
algorithm/amplex-anno: marking blob sha256:55cf7448334b2eea1287276a69732e199de3e70f9926be8babf26f3c933c79a5
algorithm/amplex-anno: marking blob sha256:7095154754192bfc2306f3b2b841ef82771b7ad39526537234adb1e74ae81a93
algorithm/amplex-anno: marking blob sha256:38a1c0aaa6fda9a4f5f940c5c7a0622430f1faac9de367016cd5a0aed8ef4478
algorithm/amplex-anno: marking blob sha256:cd134db5e982d5650745e9479866c856f24b892a92c4bfad3f3de6fd1bb7cc6d
algorithm/amplex-anno: marking blob sha256:f793a727ce319d2e88cf9a2ac03fa27a14858f22ccc82871fa388a4bf943785c
algorithm/amplex-anno: marking blob sha256:21422ba2631c7c32b702e4d21a7e128538bb5ae177acc0c84f8dc19af7a7dc13
algorithm/amplex-anno: marking blob sha256:17171aebbf0662ad91f8ac4d9b5bb0d1f955381b727a74a9d360612929b16b3d
algorithm/amplex-anno: marking blob sha256:a83ab6d6f664149bdebc388ebd9ab1e57643e4a63dc310bf582be7d97d9f7bee
algorithm/amplex-anno: marking blob sha256:1f28af52be671811aa32605a501770411477e791fe7c4670612fd9fc795830fa
algorithm/amplex-anno: marking blob sha256:c7fcbcbd44fc0f542ca701edf257cf4d79b15124eec1936953698656492130b3
algorithm/amplex-anno: marking blob sha256:2904bb0ed013327a632747a2fadcd1215888571d9c41286c0389b7e397b51dd4
algorithm/amplex-anno: marking blob sha256:0b5122e1d9883ca538f0eb72a448c1b1cf3f7a8585d1656b88cf8a7d0021903c
algorithm/amplex-anno: marking blob sha256:7ba0c16516ba05d335d20aeea846205b298f3fdd33f13eb5e0428ed21e63465e
DEBU[0000] s3aws.Stat("/docker/registry/v2/repositories/algorithm/amplex-anno/_manifests/revisions/sha256/a45aeb43cfe96d902709d56dfd672666ceda4c828f2d67167a9f77990e343a33")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=11.081149ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=94ec85d0-603e-4426-8af4-f6da0666c7a6 trace.line=137
DEBU[0000] s3aws.List("/docker/registry/v2/repositories/algorithm/amplex-anno/_manifests/revisions/sha256/a45aeb43cfe96d902709d56dfd672666ceda4c828f2d67167a9f77990e343a33")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=11.681712ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).List trace.id=b86acfd5-c00b-4cc5-978e-21dc8cdfe104 trace.line=150
DEBU[0000] s3aws.Stat("/docker/registry/v2/repositories/algorithm/amplex-anno/_manifests/revisions/sha256/a45aeb43cfe96d902709d56dfd672666ceda4c828f2d67167a9f77990e343a33/link")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=10.381193ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=5fdf4623-c105-4fd3-9f3e-e2705e0196f4 trace.line=137
DEBU[0000] s3aws.GetContent("/docker/registry/v2/repositories/algorithm/amplex-anno/_manifests/revisions/sha256/a45aeb43cfe96d902709d56dfd672666ceda4c828f2d67167a9f77990e343a33/link")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=20.277349ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent trace.id=5d081437-fb27-49d8-81b6-8009f1976ffd trace.line=82
DEBU[0000] s3aws.GetContent("/docker/registry/v2/repositories/algorithm/amplex-anno/_manifests/revisions/sha256/a45aeb43cfe96d902709d56dfd672666ceda4c828f2d67167a9f77990e343a33/link")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=10.827382ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent trace.id=c87dfe6d-6d1f-433c-9722-9dbd54155107 trace.line=82
DEBU[0000] s3aws.Stat("/docker/registry/v2/blobs/sha256/a4/a45aeb43cfe96d902709d56dfd672666ceda4c828f2d67167a9f77990e343a33/data")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=13.961469ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=982e88af-dc2e-4ef7-9078-5eb1847be2a1 trace.line=137
algorithm/amplex-anno: marking manifest sha256:a45aeb43cfe96d902709d56dfd672666ceda4c828f2d67167a9f77990e343a33
DEBU[0000] (*manifestStore).Get                          go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry
DEBU[0000] s3aws.GetContent("/docker/registry/v2/repositories/algorithm/amplex-anno/_manifests/revisions/sha256/a45aeb43cfe96d902709d56dfd672666ceda4c828f2d67167a9f77990e343a33/link")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=11.043227ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent trace.id=b3568a88-1165-4eed-9e49-a85e06a2f0ec trace.line=82
DEBU[0000] s3aws.Stat("/docker/registry/v2/blobs/sha256/a4/a45aeb43cfe96d902709d56dfd672666ceda4c828f2d67167a9f77990e343a33/data")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=11.906437ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=54630901-2414-432b-a0c0-5c07922f259d trace.line=137
DEBU[0000] s3aws.Reader("/docker/registry/v2/blobs/sha256/a4/a45aeb43cfe96d902709d56dfd672666ceda4c828f2d67167a9f77990e343a33/data", 0)  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=33.317712ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Reader trace.id=c36879b4-fdab-4d5f-b34b-7aa38824ab71 trace.line=107
DEBU[0000] (*schema2ManifestHandler).Unmarshal           go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry
algorithm/amplex-anno: marking blob sha256:620cc806e48ea4878ae561d800418694b92bbc2eedf8069c9b1abfaf5fdde716
algorithm/amplex-anno: marking blob sha256:7095154754192bfc2306f3b2b841ef82771b7ad39526537234adb1e74ae81a93
algorithm/amplex-anno: marking blob sha256:38a1c0aaa6fda9a4f5f940c5c7a0622430f1faac9de367016cd5a0aed8ef4478
algorithm/amplex-anno: marking blob sha256:cd134db5e982d5650745e9479866c856f24b892a92c4bfad3f3de6fd1bb7cc6d
algorithm/amplex-anno: marking blob sha256:f793a727ce319d2e88cf9a2ac03fa27a14858f22ccc82871fa388a4bf943785c
algorithm/amplex-anno: marking blob sha256:21422ba2631c7c32b702e4d21a7e128538bb5ae177acc0c84f8dc19af7a7dc13
algorithm/amplex-anno: marking blob sha256:17171aebbf0662ad91f8ac4d9b5bb0d1f955381b727a74a9d360612929b16b3d
algorithm/amplex-anno: marking blob sha256:a83ab6d6f664149bdebc388ebd9ab1e57643e4a63dc310bf582be7d97d9f7bee
algorithm/amplex-anno: marking blob sha256:d95f53b23d31fae1c17ace3596e2866f713e4b1a87c59fd9a916e065372037e8
algorithm/amplex-anno: marking blob sha256:924de7dace930065cc2ce1f85842ce2d3994a243a213e3a9ab2c0fad4dce38cd
algorithm/amplex-anno: marking blob sha256:f558f04fc6e1d20eac35b3a8260088e82906ab9743466afcd03c8f34ee6a64bc
algorithm/amplex-anno: marking blob sha256:f4a8c9cb632fb8cd553af2814c35641b78f570832431b32f3bf01494425db569
algorithm/amplex-anno: marking blob sha256:e84dabf93a27c9bd29c613479843f1127dd3e50dd89aaa3e74337a121c8cbc04
DEBU[0000] s3aws.Stat("/docker/registry/v2/repositories/algorithm/amplex-anno/_manifests")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=12.285528ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=02300b6b-9bc0-42f3-9a56-0f3b2da31d22 trace.line=137
DEBU[0000] s3aws.Stat("/docker/registry/v2/repositories/algorithm/deepscore2")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=12.460276ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=adf0d7d8-4da1-4aa3-9d43-42069902c95b trace.line=137
DEBU[0000] s3aws.List("/docker/registry/v2/repositories/algorithm/deepscore2")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=10.676892ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).List trace.id=8b226fb6-9f15-4e19-88d7-463642e9857d trace.line=150
DEBU[0000] s3aws.Stat("/docker/registry/v2/repositories/algorithm/deepscore2/_layers")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=10.793296ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=093b9a24-9738-416f-94f0-d66e2bc94a0e trace.line=137
algorithm/deepscore2
DEBU[0000] s3aws.List("/docker/registry/v2/repositories/algorithm/deepscore2/_manifests/revisions")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=10.884547ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).List trace.id=98a57e11-ec88-4e5c-a569-9c80d08266f0 trace.line=150
DEBU[0000] s3aws.Stat("/docker/registry/v2/repositories/algorithm/deepscore2/_manifests/revisions/sha256")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=359.286152ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=9d5047ea-d5d8-4118-a543-576185bb55b8 trace.line=137
DEBU[0000] s3aws.List("/docker/registry/v2/repositories/algorithm/deepscore2/_manifests/revisions/sha256")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=16.165647ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).List trace.id=59613a0d-a2b8-4814-b4e8-1eab60703f9e trace.line=150
DEBU[0000] s3aws.Stat("/docker/registry/v2/repositories/algorithm/deepscore2/_manifests/revisions/sha256/1fbc2284ab26fe1b59d321e65bc74bf25e83048af84f036c525f701ff02c0a40")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=12.484532ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=4ba50e20-70b6-4936-9cbf-a78cf7f44f71 trace.line=137
DEBU[0000] s3aws.List("/docker/registry/v2/repositories/algorithm/deepscore2/_manifests/revisions/sha256/1fbc2284ab26fe1b59d321e65bc74bf25e83048af84f036c525f701ff02c0a40")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=11.872922ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).List trace.id=40db8ac4-cbee-4409-9ab6-d09849ba5334 trace.line=150
DEBU[0000] s3aws.Stat("/docker/registry/v2/repositories/algorithm/deepscore2/_manifests/revisions/sha256/1fbc2284ab26fe1b59d321e65bc74bf25e83048af84f036c525f701ff02c0a40/link")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=14.263335ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=0e4aae88-7aa2-458b-b73f-01357fc03575 trace.line=137
DEBU[0000] s3aws.GetContent("/docker/registry/v2/repositories/algorithm/deepscore2/_manifests/revisions/sha256/1fbc2284ab26fe1b59d321e65bc74bf25e83048af84f036c525f701ff02c0a40/link")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=46.928645ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent trace.id=b8b437e5-7d4d-41e3-b423-7523bb20162d trace.line=82
DEBU[0000] s3aws.GetContent("/docker/registry/v2/repositories/algorithm/deepscore2/_manifests/revisions/sha256/1fbc2284ab26fe1b59d321e65bc74bf25e83048af84f036c525f701ff02c0a40/link")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=9.754906ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent trace.id=5c567fab-115d-4594-aff1-a94df8365c0f trace.line=82
DEBU[0000] s3aws.Stat("/docker/registry/v2/blobs/sha256/1f/1fbc2284ab26fe1b59d321e65bc74bf25e83048af84f036c525f701ff02c0a40/data")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=17.348742ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=c1ff0fd4-8c2c-4e4d-94cc-e769abae258e trace.line=137
algorithm/deepscore2: marking manifest sha256:1fbc2284ab26fe1b59d321e65bc74bf25e83048af84f036c525f701ff02c0a40
DEBU[0000] (*manifestStore).Get                          go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry


DEBU[12053] s3aws.List("/docker/registry/v2/blobs/sha256/ff/fffefe818f4d02268a03a57b8b534c19270125b0d04484d9add5996495e73e9b")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=15.141601ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).List trace.id=67568262-d149-4b4b-be01-2f58af1b920c trace.line=150
DEBU[12053] s3aws.Stat("/docker/registry/v2/blobs/sha256/ff/fffefe818f4d02268a03a57b8b534c19270125b0d04484d9add5996495e73e9b/data")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=12.993203ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=e684d389-74a9-4823-a081-d918f85686f8 trace.line=137
DEBU[12053] s3aws.Stat("/docker/registry/v2/blobs/sha256/ff/ffff9ecf9d200ad17f849f168afb0f142976140a2e0adafa08336e4d52628b81")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=12.175515ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=753f1efc-3ad7-45b7-835e-5b762bdbff43 trace.line=137
DEBU[12053] s3aws.List("/docker/registry/v2/blobs/sha256/ff/ffff9ecf9d200ad17f849f168afb0f142976140a2e0adafa08336e4d52628b81")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=13.721042ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).List trace.id=1251d455-441e-43d9-b3b2-f77ce882bd82 trace.line=150
DEBU[12053] s3aws.Stat("/docker/registry/v2/blobs/sha256/ff/ffff9ecf9d200ad17f849f168afb0f142976140a2e0adafa08336e4d52628b81/data")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=10.98607ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=96b053a2-caae-4d7f-89a7-33b73930ba27 trace.line=137
DEBU[12053] s3aws.Stat("/docker/registry/v2/blobs/sha256/ff/ffffdf86f942100cffb6d130d0398fa23be9f50e56c0cb4e3e76face82e9931d")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=12.915017ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=ffba971a-c2fa-455d-a0cf-113800dc85cd trace.line=137
DEBU[12053] s3aws.List("/docker/registry/v2/blobs/sha256/ff/ffffdf86f942100cffb6d130d0398fa23be9f50e56c0cb4e3e76face82e9931d")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=10.69976ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).List trace.id=7e9f5b5e-9f63-4a01-bd23-7d384a964f54 trace.line=150
DEBU[12053] s3aws.Stat("/docker/registry/v2/blobs/sha256/ff/ffffdf86f942100cffb6d130d0398fa23be9f50e56c0cb4e3e76face82e9931d/data")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=13.022303ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=93ae74b8-40e8-4afa-a0ae-4b942f913fa2 trace.line=137
DEBU[12053] s3aws.Stat("/docker/registry/v2/blobs/sha256/ff/fffff2f1a098df5770a70756d051bea715d1a95405ac13880415fdeefe12fd81")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=13.676276ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=6ca12cf5-5a29-41df-8c22-20242757baa9 trace.line=137
DEBU[12053] s3aws.List("/docker/registry/v2/blobs/sha256/ff/fffff2f1a098df5770a70756d051bea715d1a95405ac13880415fdeefe12fd81")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=9.6727ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).List trace.id=69bfcfe3-955e-40e8-a2ba-13724c0aa76e trace.line=150
DEBU[12053] s3aws.Stat("/docker/registry/v2/blobs/sha256/ff/fffff2f1a098df5770a70756d051bea715d1a95405ac13880415fdeefe12fd81/data")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=11.105141ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat trace.id=850602f6-8a77-492e-87f3-63a2b0c4a517 trace.line=137

142758 blobs marked, 54002 blobs eligible for deletion
blob eligible for deletion: sha256:0a5fd62428a4030364c2e5ce5e0ec22295d4b22475e11dee68214b489536c54b
INFO[12053] Deleting blob: /docker/registry/v2/blobs/sha256/0a/0a5fd62428a4030364c2e5ce5e0ec22295d4b22475e11dee68214b489536c54b  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry
DEBU[12053] s3aws.Delete("/docker/registry/v2/blobs/sha256/0a/0a5fd62428a4030364c2e5ce5e0ec22295d4b22475e11dee68214b489536c54b")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=27.987222ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Delete trace.id=726c7569-ee11-4f11-bce7-6708befe9684 trace.line=177
blob eligible for deletion: sha256:3bd956e1c8d75431d8460cc8efe7bbed6282ba8b6331095f0deee9d6a63a71a5
INFO[12053] Deleting blob: /docker/registry/v2/blobs/sha256/3b/3bd956e1c8d75431d8460cc8efe7bbed6282ba8b6331095f0deee9d6a63a71a5  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry
DEBU[12053] s3aws.Delete("/docker/registry/v2/blobs/sha256/3b/3bd956e1c8d75431d8460cc8efe7bbed6282ba8b6331095f0deee9d6a63a71a5")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=26.828158ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Delete trace.id=371d6ae3-0fe3-4fcc-ae7f-69164de08ee1 trace.line=177
blob eligible for deletion: sha256:501f163384b79a0e253bb8b900eade0d889bd6dc0cf38d990c94cfd1b8cbf84b
INFO[12053] Deleting blob: /docker/registry/v2/blobs/sha256/50/501f163384b79a0e253bb8b900eade0d889bd6dc0cf38d990c94cfd1b8cbf84b  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry
DEBU[12053] s3aws.Delete("/docker/registry/v2/blobs/sha256/50/501f163384b79a0e253bb8b900eade0d889bd6dc0cf38d990c94cfd1b8cbf84b")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=34.585376ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Delete trace.id=ad7a8ddf-644a-4935-9b3c-26da46e8e352 trace.line=177
blob eligible for deletion: sha256:bcc8a522e1d4c4647e8d6b7fa34ca2ee9ea030235553ea973ae69eb2f4045382
INFO[12053] Deleting blob: /docker/registry/v2/blobs/sha256/bc/bcc8a522e1d4c4647e8d6b7fa34ca2ee9ea030235553ea973ae69eb2f4045382  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry
DEBU[12053] s3aws.Delete("/docker/registry/v2/blobs/sha256/bc/bcc8a522e1d4c4647e8d6b7fa34ca2ee9ea030235553ea973ae69eb2f4045382")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=38.225731ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Delete trace.id=8df06c44-672c-46dc-994d-8882a22ca4a6 trace.line=177
blob eligible for deletion: sha256:ddc59efaf374bfb1e3c5dd7826bb05d5337970d860218a27607efc364f4e9b95
...
DEBU[14406] s3aws.Delete("/docker/registry/v2/blobs/sha256/46/46214fd44e01666414d4b65560de6844403df66d34ad62ecfe3a163ed5ee46be")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=28.883484ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Delete trace.id=6e37b920-fb01-42b5-b41d-404d3b79a8a8 trace.line=177
blob eligible for deletion: sha256:62bf589f89899a5778327b518c82add74408ec0ac5aa85e0b08db877d1195cc4
INFO[14406] Deleting blob: /docker/registry/v2/blobs/sha256/62/62bf589f89899a5778327b518c82add74408ec0ac5aa85e0b08db877d1195cc4  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry
DEBU[14406] s3aws.Delete("/docker/registry/v2/blobs/sha256/62/62bf589f89899a5778327b518c82add74408ec0ac5aa85e0b08db877d1195cc4")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=31.732591ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Delete trace.id=5b064528-e54c-4507-a0aa-9fd2b00d85ca trace.line=177
blob eligible for deletion: sha256:90a960ef195bc03eeddc7da598e7fd23f2b8c33d45c289b435dcb450cf3e1772
INFO[14406] Deleting blob: /docker/registry/v2/blobs/sha256/90/90a960ef195bc03eeddc7da598e7fd23f2b8c33d45c289b435dcb450cf3e1772  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry
DEBU[14406] s3aws.Delete("/docker/registry/v2/blobs/sha256/90/90a960ef195bc03eeddc7da598e7fd23f2b8c33d45c289b435dcb450cf3e1772")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=28.834279ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Delete trace.id=80b9675d-7f26-4310-ab4e-0fdc03843ef3 trace.line=177
blob eligible for deletion: sha256:c105c9cb3d482a9b43cdae02335080f5f88bae466b5e05a551aab87a86478b6e
INFO[14406] Deleting blob: /docker/registry/v2/blobs/sha256/c1/c105c9cb3d482a9b43cdae02335080f5f88bae466b5e05a551aab87a86478b6e  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry
DEBU[14406] s3aws.Delete("/docker/registry/v2/blobs/sha256/c1/c105c9cb3d482a9b43cdae02335080f5f88bae466b5e05a551aab87a86478b6e")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=31.722932ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Delete trace.id=3af3f412-d04d-4b65-91d1-d31bcca4188d trace.line=177
blob eligible for deletion: sha256:c9c285554c462303ab45656ae1dc50b00d090486554d3fcec07709554a5dad0d
INFO[14406] Deleting blob: /docker/registry/v2/blobs/sha256/c9/c9c285554c462303ab45656ae1dc50b00d090486554d3fcec07709554a5dad0d  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry
DEBU[14406] s3aws.Delete("/docker/registry/v2/blobs/sha256/c9/c9c285554c462303ab45656ae1dc50b00d090486554d3fcec07709554a5dad0d")  go.version=go1.7.3 instance.id=475b27ad-d6ba-4cde-b9f4-d3327923e686 service=registry trace.duration=32.437074ms trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Delete trace.id=779b9335-50ca-44dd-951b-0e29aca3ce33 trace.line=177
root@harbor-service-new-stag:/opt/apps/harbor#

[moooofly]

A similar issue mentioned in #4214.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mweigel]

I had a very similar issue with tags of zero size and a timestamp of 01/01/1970. I was unable to delete these tags and attempting to re-push the tags didn't help as Docker indicated that all layers already existed. I resolved this with the following, NOTE: I can't guarantee that clearing the Redis cache completely is a safe operation for all deployments, and I'm not using HA mode. Maybe one of the Harbor developers can give some guidance here.

1. Cleared Redis cache completely.
2. Restarted Harbor
3. Re-push problematic tag. This now resulted in some layers being pushed rather than Docker indicating that all layers were already present.

At this point the tag's sizes and dates were displayed correctly and I was able to delete them as this is what I was originally trying to do. I believe this is an issue with the Redis cache being out of sync with what's actually on disk in the registry.

[louyiping]

Also meet this problem with version 1.6.0. And the disk size of these images are all 0B.That's why?

[mweigel]

@louyiping The above definitely worked for me and I have not had any issues with Harbor since. As I say, I can't guarantee that clearing the Redis cache completely is a safe operation for all deployments, one of the Harbor devs would have to give some guidance here. I run Harbor in a virtualised environment which makes taking a snapshot of the VM the easiest way to test changes.

[louyiping]

@mweigel Thanks. I don't know the internal principle of the harbor clearly, so I don't dare to operate it in production environment now.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[abal5]

I had a very similar issue with tags of zero size and a timestamp of 01/01/1970. I was unable to delete these tags and attempting to re-push the tags didn't help as Docker indicated that all layers already existed. I resolved this with the following, NOTE: I can't guarantee that clearing the Redis cache completely is a safe operation for all deployments, and I'm not using HA mode. Maybe one of the Harbor developers can give some guidance here.

1. Cleared Redis cache completely.
2. Restarted Harbor
3. Re-push problematic tag. This now resulted in some layers being pushed rather than Docker indicating that all layers were already present.

At this point the tag's sizes and dates were displayed correctly and I was able to delete them as this is what I was originally trying to do. I believe this is an issue with the Redis cache being out of sync with what's actually on disk in the registry.

This helped me out with a similar issue on version 1.6.0 thank you so much. @mweigel

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.