Harbor: quota migration error

[mmack]

If you are reporting a problem, please make sure the following information is provided:

Expected behavior and actual behavior:
Harbor core should come up

Steps to reproduce the problem:
Harbor 1.9.0 was running fine for days, I needed to rebuild the k8s worker nodes and core needed to restart.
I tried to truncate the table project_blob and even tried to update it to 1.9.1-dev like described here: https://github.com/goharbor/harbor/issues/9279 without success.. any help is appreciated.

Versions:
1.9.0 1.9.1 on eks 1.14.7

Additional context:

2019-10-01T11:00:48Z [INFO] [/core/api/quota/migrator.go:111]: [Quota-Sync]:: start to persist data for server ... [registry]
2019-10-01T11:00:48Z [INFO] [/core/api/quota/registry/registry.go:189]: [Quota-Sync]:: start to persist artifact&blob for project: ci-images, progress... [0/5]
2019-10-01T11:00:48Z [INFO] [/core/api/quota/registry/registry.go:201]: [Quota-Sync]:: success to persist artifact&blob for project: ci-images, progress... [0/5]
2019-10-01T11:00:48Z [INFO] [/core/api/quota/registry/registry.go:189]: [Quota-Sync]:: start to persist artifact&blob for project: oracle-databases, progress... [1/5]
2019-10-01T11:00:48Z [INFO] [/core/api/quota/registry/registry.go:201]: [Quota-Sync]:: success to persist artifact&blob for project: oracle-databases, progress... [1/5]
2019-10-01T11:00:48Z [INFO] [/core/api/quota/registry/registry.go:189]: [Quota-Sync]:: start to persist artifact&blob for project: infrastructure, progress... [2/5]
2019-10-01T11:00:48Z [INFO] [/core/api/quota/registry/registry.go:201]: [Quota-Sync]:: success to persist artifact&blob for project: infrastructure, progress... [2/5]
2019-10-01T11:00:48Z [INFO] [/core/api/quota/registry/registry.go:189]: [Quota-Sync]:: start to persist artifact&blob for project: testbench, progress... [3/5]
2019-10-01T11:00:48Z [INFO] [/core/api/quota/registry/registry.go:201]: [Quota-Sync]:: success to persist artifact&blob for project: testbench, progress... [3/5]
2019-10-01T11:00:48Z [INFO] [/core/api/quota/registry/registry.go:189]: [Quota-Sync]:: start to persist artifact&blob for project: library, progress... [4/5]
2019-10-01T11:00:48Z [INFO] [/core/api/quota/registry/registry.go:201]: [Quota-Sync]:: success to persist artifact&blob for project: library, progress... [4/5]
2019-10-01T11:00:48Z [INFO] [/core/api/quota/registry/registry.go:260]: [Quota-Sync]:: start to persist project&blob for project: ci-images, progress... [0/5]
2019-10-01T11:00:48Z [ERROR] [/core/api/quota/registry/registry.go:287]: sql: duplicate row in DB
2019-10-01T11:00:48Z [INFO] [/core/api/quota/migrator.go:113]: [Quota-Sync]:: fail to persist data from server ... [registry], quit sync ...
2019-10-01T11:00:48Z [ERROR] [/core/main.go:104]: Fail to sync quota data, sql: duplicate row in DB
2019-10-01T11:00:48Z [FATAL] [/core/main.go:252]: quota migration error, sql: duplicate row in DB

[wy65701436]

Just try to clean the data in table of project_blob, and restart harbor, just let it to do the migration again.

Note: DO NOT stop or restart harbor before the migration is done, to get the process status, just monitor the core.log

[mmack]

I actually did this. Several times... My best guess right now is that k8s stops and restarts the container b/c it's failed. I'm using the official helm chart.

[wy65701436]

just refer to the https://github.com/goharbor/harbor-helm/issues/371 to extend timeout, and retry https://github.com/goharbor/harbor/issues/9309#issuecomment-538236920

[matlec]

I just ran into the same problem and had to clear the project_blob table manually. I think it would be helpful to clear this table automatically once a duplicate error occurs.

[matlec]

I think this issue deserves a little more attention. I just noticed that the migration is triggered every time the core pod starts and fails on every restart with the following error:

core 2019-10-22T07:39:44Z [INFO [/core/main.go:102]: Start to sync quota data .....
core 2019-10-22T07:39:44Z [INFO [/core/api/quota/migrator.go:92]: [Quota-Sync:: start to ping server ... [chart
core 2019-10-22T07:39:44Z [INFO [/core/api/quota/migrator.go:97]: [Quota-Sync:: success to ping server ... [chart
core 2019-10-22T07:39:44Z [INFO [/core/api/quota/migrator.go:98]: [Quota-Sync:: start to dump data from server ... [chart
core 2019-10-22T07:39:44Z [DEBUG [/common/dao/project.go:149]: sql:=select distinct p.project_id, p.name, p.owner_id, 
core         p.creation_time, p.update_time  from project as p where p.deleted=false order by p.name, param= 
core 2019-10-22T07:39:44Z [INFO [/chartserver/cache.go:184]: Enable redis cache for chart caching
core 2019-10-22T07:39:44Z [DEBUG [/core/api/quota/chart/chart.go:217]: Chart storage server is set to http://harbor-harbor-chartmuseum
core 2019-10-22T07:39:44Z [INFO [/core/api/quota/chart/chart.go:218]: API controller for chart repository server is successfully initialized
core 2019-10-22T07:39:44Z [INFO [/core/api/quota/migrator.go:104]: [Quota-Sync:: success to dump data from server ... [chart
core 2019-10-22T07:39:44Z [INFO [/core/api/quota/migrator.go:111]: [Quota-Sync:: start to persist data for server ... [chart
core 2019-10-22T07:39:44Z [INFO [/core/api/quota/migrator.go:116]: [Quota-Sync:: success to persist data for server ... [chart
core 2019-10-22T07:39:44Z [INFO [/core/api/quota/migrator.go:92]: [Quota-Sync:: start to ping server ... [registry
core 2019-10-22T07:39:44Z [INFO [/core/api/quota/migrator.go:97]: [Quota-Sync:: success to ping server ... [registry
core 2019-10-22T07:39:44Z [INFO [/core/api/quota/migrator.go:98]: [Quota-Sync:: start to dump data from server ... [registry
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: registry:catalog:*
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/_catalog?n=1000
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: repository:mirror/minideb:pull
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: repository:mirror/postgresql:pull
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: repository:mirror/rabbitmq_prometheus:pull
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/mirror/rabbitmq_prometheus/tags/list
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: repository:mirror/rabbitmq_prometheus:pull
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:239]: get token for scope repository:mirror/rabbitmq_prometheus:pull from cache
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/mirror/postgresql/tags/list
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: repository:mirror/postgresql:pull
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:239]: get token for scope repository:mirror/postgresql:pull from cache
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/apps/image4/tags/list
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: repository:apps/image4:pull
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:239]: get token for scope repository:apps/image4:pull from cache
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/mirror/minideb/tags/list
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: repository:mirror/minideb:pull
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:239]: get token for scope repository:mirror/minideb:pull from cache
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/apps/image1/tags/list
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: repository:apps/image1:pull
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:239]: get token for scope repository:apps/image1:pull from cache
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/mirror/postgresql/manifests/10.7.0
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/apps/image2/tags/list
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: repository:apps/image2:pull
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:239]: get token for scope repository:apps/image2:pull from cache
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/mirror/minideb/manifests/latest
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/apps/image4/manifests/release-1.0.0
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: repository:apps/image4:pull
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:239]: get token for scope repository:apps/image4:pull from cache
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/apps/image1/manifests/release-1.0.0
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/apps/image3/tags/list
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: repository:apps/image3:pull
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:239]: get token for scope repository:apps/image3:pull from cache
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/mirror/rabbitmq_prometheus/manifests/latest
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/apps/image2/manifests/release-1.0.0
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/apps/image4/manifests/release-1.0.1
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/apps/image3/manifests/release-1.0.0
core 2019-10-22T07:39:44Z [INFO [/core/api/quota/migrator.go:104]: [Quota-Sync:: success to dump data from server ... [registry
core 2019-10-22T07:39:44Z [INFO [/core/api/quota/migrator.go:111]: [Quota-Sync:: start to persist data for server ... [registry
core 2019-10-22T07:39:44Z [INFO [/core/api/quota/registry/registry.go:189]: [Quota-Sync:: start to persist artifact&blob for project: mirror, progress... [0/2]
core 2019-10-22T07:39:45Z [INFO [/core/api/quota/registry/registry.go:201]: [Quota-Sync:: success to persist artifact&blob for project: mirror, progress... [0/2]
core 2019-10-22T07:39:45Z [INFO [/core/api/quota/registry/registry.go:189]: [Quota-Sync:: start to persist artifact&blob for project: apps, progress... [1/2]
core 2019-10-22T07:39:45Z [INFO [/core/api/quota/registry/registry.go:201]: [Quota-Sync:: success to persist artifact&blob for project: apps, progress... [1/2]
core 2019-10-22T07:39:45Z [INFO [/core/api/quota/registry/registry.go:260]: [Quota-Sync:: start to persist project&blob for project: mirror, progress... [0/2]
core 2019-10-22T07:39:45Z [ERROR [/core/api/quota/registry/registry.go:287]: sql: duplicate row in DB
core 2019-10-22T07:39:45Z [INFO [/core/api/quota/migrator.go:113]: [Quota-Sync:: fail to persist data from server ... [registry, quit sync ...
core 2019-10-22T07:39:45Z [ERROR [/core/main.go:104]: Fail to sync quota data, sql: duplicate row in DB
core 2019-10-22T07:39:45Z [FATAL [/core/main.go:252]: quota migration error, sql: duplicate row in DB

The only work-around is to clear the project_blob table manually.

[LunaticZorr]

I think this issue deserves a little more attention. I just noticed that the migration is triggered every time the core pod starts and fails on every restart with the following error:

core 2019-10-22T07:39:44Z [INFO [/core/main.go:102]: Start to sync quota data .....
core 2019-10-22T07:39:44Z [INFO [/core/api/quota/migrator.go:92]: [Quota-Sync:: start to ping server ... [chart
core 2019-10-22T07:39:44Z [INFO [/core/api/quota/migrator.go:97]: [Quota-Sync:: success to ping server ... [chart
core 2019-10-22T07:39:44Z [INFO [/core/api/quota/migrator.go:98]: [Quota-Sync:: start to dump data from server ... [chart
core 2019-10-22T07:39:44Z [DEBUG [/common/dao/project.go:149]: sql:=select distinct p.project_id, p.name, p.owner_id, 
core         p.creation_time, p.update_time  from project as p where p.deleted=false order by p.name, param= 
core 2019-10-22T07:39:44Z [INFO [/chartserver/cache.go:184]: Enable redis cache for chart caching
core 2019-10-22T07:39:44Z [DEBUG [/core/api/quota/chart/chart.go:217]: Chart storage server is set to http://harbor-harbor-chartmuseum
core 2019-10-22T07:39:44Z [INFO [/core/api/quota/chart/chart.go:218]: API controller for chart repository server is successfully initialized
core 2019-10-22T07:39:44Z [INFO [/core/api/quota/migrator.go:104]: [Quota-Sync:: success to dump data from server ... [chart
core 2019-10-22T07:39:44Z [INFO [/core/api/quota/migrator.go:111]: [Quota-Sync:: start to persist data for server ... [chart
core 2019-10-22T07:39:44Z [INFO [/core/api/quota/migrator.go:116]: [Quota-Sync:: success to persist data for server ... [chart
core 2019-10-22T07:39:44Z [INFO [/core/api/quota/migrator.go:92]: [Quota-Sync:: start to ping server ... [registry
core 2019-10-22T07:39:44Z [INFO [/core/api/quota/migrator.go:97]: [Quota-Sync:: success to ping server ... [registry
core 2019-10-22T07:39:44Z [INFO [/core/api/quota/migrator.go:98]: [Quota-Sync:: start to dump data from server ... [registry
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: registry:catalog:*
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/_catalog?n=1000
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: repository:mirror/minideb:pull
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: repository:mirror/postgresql:pull
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: repository:mirror/rabbitmq_prometheus:pull
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/mirror/rabbitmq_prometheus/tags/list
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: repository:mirror/rabbitmq_prometheus:pull
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:239]: get token for scope repository:mirror/rabbitmq_prometheus:pull from cache
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/mirror/postgresql/tags/list
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: repository:mirror/postgresql:pull
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:239]: get token for scope repository:mirror/postgresql:pull from cache
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/apps/image4/tags/list
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: repository:apps/image4:pull
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:239]: get token for scope repository:apps/image4:pull from cache
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/mirror/minideb/tags/list
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: repository:mirror/minideb:pull
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:239]: get token for scope repository:mirror/minideb:pull from cache
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/apps/image1/tags/list
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: repository:apps/image1:pull
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:239]: get token for scope repository:apps/image1:pull from cache
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/mirror/postgresql/manifests/10.7.0
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/apps/image2/tags/list
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: repository:apps/image2:pull
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:239]: get token for scope repository:apps/image2:pull from cache
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/mirror/minideb/manifests/latest
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/apps/image4/manifests/release-1.0.0
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: repository:apps/image4:pull
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:239]: get token for scope repository:apps/image4:pull from cache
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/apps/image1/manifests/release-1.0.0
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/apps/image3/tags/list
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:214]: scopes parsed from request: repository:apps/image3:pull
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/auth/tokenauthorizer.go:239]: get token for scope repository:apps/image3:pull from cache
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/mirror/rabbitmq_prometheus/manifests/latest
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/apps/image2/manifests/release-1.0.0
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/apps/image4/manifests/release-1.0.1
core 2019-10-22T07:39:44Z [DEBUG [/common/utils/registry/transport.go:51]: 200 | GET http://harbor-harbor-registry:5000/v2/apps/image3/manifests/release-1.0.0
core 2019-10-22T07:39:44Z [INFO [/core/api/quota/migrator.go:104]: [Quota-Sync:: success to dump data from server ... [registry
core 2019-10-22T07:39:44Z [INFO [/core/api/quota/migrator.go:111]: [Quota-Sync:: start to persist data for server ... [registry
core 2019-10-22T07:39:44Z [INFO [/core/api/quota/registry/registry.go:189]: [Quota-Sync:: start to persist artifact&blob for project: mirror, progress... [0/2]
core 2019-10-22T07:39:45Z [INFO [/core/api/quota/registry/registry.go:201]: [Quota-Sync:: success to persist artifact&blob for project: mirror, progress... [0/2]
core 2019-10-22T07:39:45Z [INFO [/core/api/quota/registry/registry.go:189]: [Quota-Sync:: start to persist artifact&blob for project: apps, progress... [1/2]
core 2019-10-22T07:39:45Z [INFO [/core/api/quota/registry/registry.go:201]: [Quota-Sync:: success to persist artifact&blob for project: apps, progress... [1/2]
core 2019-10-22T07:39:45Z [INFO [/core/api/quota/registry/registry.go:260]: [Quota-Sync:: start to persist project&blob for project: mirror, progress... [0/2]
core 2019-10-22T07:39:45Z [ERROR [/core/api/quota/registry/registry.go:287]: sql: duplicate row in DB
core 2019-10-22T07:39:45Z [INFO [/core/api/quota/migrator.go:113]: [Quota-Sync:: fail to persist data from server ... [registry, quit sync ...
core 2019-10-22T07:39:45Z [ERROR [/core/main.go:104]: Fail to sync quota data, sql: duplicate row in DB
core 2019-10-22T07:39:45Z [FATAL [/core/main.go:252]: quota migration error, sql: duplicate row in DB

The only work-around is to clear the project_blob table manually.

Thanks a lot @matlec ! I just ran into the same issue and this fixed it too for me. I agree that this should not require manual intervention every time...

[dungdm93]

I got the same problem.
I run 3 replicas of harbor-core, and it seem like Quota-Sync is re-run in every pods, even if the previous one already done :'(

[matlec]

Seems like https://github.com/goharbor/harbor/commit/368cf225de1a5e85f540fdaffa6eb4bcd796fc78 fixes this issue. Unfortunately this fix obviously hasn't made it into the v1.9.1 Docker image yet (works perfectly with v1.9.1-dev though). Is there any plan to provide an updated v1.9.1 image that includes this fix?

[hvancamp]

I'm experience the same issue after upgrading to v1.9.1 which I thought was suppose to fix the quota-sync problem?

When performing a docker-compose down/up, the core container/process will never come back because the [Quota-Sync] keeps on looping over all the projects.

Only by truncating the project_blob table on the DB, will the core startup correctly again.

[wy65701436]

the quota migration errors will be fixed in v1.9.2.

[wy65701436]

close it as the fix is in v1.9.2-rc1.