Harbor: Migration issue from 0.5.0 to 1.1.1 release

[roldancer]

Hi All,
We have migrated our Harbor 0.5.0 to 1.1.1 release, using the tools recommended in the documentation, for some reason the repositories's tags are not available, I mean the repositories are available but when you click on one repositories the tags are empty.

But if you migrate from 0.5.0 to 1.1.1.RC1 and after that to 1.1.1 release the repositories's tags are available.

Many thanks.

[reasonerjt]

@roldancer thanks for reporting the issue.

Could you attach the latest logs?
Also, could you open your developer tool on your browser to see if there're any errors on UI?
@wknet123 @steven-zou this looks like #2163 ?

[roldancer]

Hi, We will try to get the logs and screeshots, we did a rollback because our registry is in production mode.

Thanks.

[steven-zou]

@reasonerjt

Right, seems the same issue with #2163.

@wknet123 ,

Is the fixed code delivered to RC release and/or master branch?

[steven-zou]

@roldancer

Any updates for this issue? If you upgrade to RC4, the issue might be gone.

[cbarzu]

Hi,

I have the same problem. We found that the problem is present only on replicated repositories. Repositories where the images where uploaded directly are working as expected.

Logs:

docker-current_ui.log
-----------------------
May 16 10:35:44 vmz1registry01.corp docker-current/ui[2541]: 2017-05-16T08:35:44Z [DEBUG] [authutils.go:87]: user: admin, access: &{repository  MYREPO/javase [push * pull]}
May 16 10:35:44 vmz1registry01.corp docker-current/ui[2541]: 2017-05-16T08:35:44Z [DEBUG] [transport.go:50]: 200 | GET http://registry:5000/v2/MYREPO/javase/tags/list
May 16 10:35:44 vmz1registry01.corp docker-current/ui[2541]: 2017-05-16T08:35:44Z [DEBUG] [transport.go:50]: 404 | GET http://registry:5000/v2/MYREPO/javase/manifests/1.1


docker-current_registry.log
---------------------------------
May 16 10:45:30 vmz1registry01.corp docker-current/registry[2541]: time="2017-05-16T08:45:30.496160871Z" level=error msg="response completed with error" auth.user.name=admin err.code="manifest unknown" err.detail="filesystem: Path not found: /docker/registry/v2/repositories/MYREPO/javase/_manifests/revisions/sha256/29596032da96cc5656e9bd1491c44162a96ea65f2bbe783d0643046f0e5f68cf/signatures/sha256" err.message="manifest unknown" go.version=go1.7.3 http.request.host="registry:5000" http.request.id=31e0cf1b-8320-4054-8cfc-069fcd0bc488 http.request.method=GET http.request.remoteaddr="172.18.0.6:43128" http.request.uri="/v2/MYREPO/javase/manifests/1.1" http.request.useragent="Go-http-client/1.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=2.408857ms http.response.status=404 http.response.written=276 instance.id=a23902b9-32e5-43be-82ab-38efbb078294 service=registry vars.name="MYREPO/javase" vars.reference=1.1 version="v2.4.1+unknown" 
May 16 10:45:30 vmz1registry01.corp docker-current/registry[2541]: 172.18.0.6 - - [16/May/2017:08:45:30 +0000] "GET /v2/MYREPO/javase/manifests/1.1 HTTP/1.1" 404 276 "" "Go-http-client/1.1"

Regards,
Claudiu

[roldancer]

Hi @cbarzu , I have the same problem, only with replicated repositories..

@reasonerjt @steven-zou any idea ?

Thanks for your heap.

[reasonerjt]

@ywk253100 please check

[steven-zou]

@ywk253100

Seems the request from page gets 404 response. Please check if api is working well.

[cbarzu]

Checking logs again it seems like it is unable to find this file:

/docker/registry/v2/repositories/myrepo/javase/_manifests/revisions/sha256/50236fd71b3037fcc7c6086e43f1cdba7df796a26e23278b9f0db0a763959980/signatures/sha256
It only exists /docker/registry/v2/repositories/myrepo/javase/_manifests/revisions/sha256/50236fd71b3037fcc7c6086e43f1cdba7df796a26e23278b9f0db0a763959980/link

However in version 0.5.0 I can see everything perfectly.

[roldancer]

Hi @steven-zou API is working properly I least we can see API traces in the logs.

[reasonerjt]

@cbarzu, please let me clarify to understand the problem you are seeing.

As for "/myrepo/javase", do you mean you could see it on 0.5.0 and it disappears after upgraded from 0.5.0 to 1.1.1?
Was it replicated from other Harbor instance before upgrading?

[roldancer]

@reasonerjt

As for "/myrepo/javase", do you mean you could see it on 0.5.0 and it disappears after upgraded from 0.5.0 to 1.1.1?
* YES
Was it replicated from other Harbor instance before upgrading?
* YES, We have 4 Harbor registries, we have one projects that is replicated to 3 Harbor registries, we have updated one of the Harbor replica/slave, the update finish successfully, but from the console we are not able to get the tags from a repo that is replicated, but doing more troubleshooting we have seen that other (NON replicated projects) have the same problem, could be the docker client version ?

thanks.

[reasonerjt]

@roldancer I'm confused.
You mentioned in previous comment that it happens * only with replicated repositories..*
But very last comment you mention "other projects have same problem".

Could you clarify, is it true that all projects are having issues after upgrading? Or only projects that are replicated?

[reasonerjt]

@ywk253100 please check with the log that @cbarzu provided, need to investigate when does registry look for "signature" in manifest.

[roldancer]

@reasonerjt During the first troubleshooting we found that this issue only were affecting "Replicated projects" but we have tried to update a second Harbor registry, in this second try, we made another a deeper troubleshooting , in this case we found that some projects were affected by this issue (replicated and non replicated project) other projects are working properly.

Could be the docker's manifest version ?

Thanks.

[roldancer]

Hi, Looking at the registry's logs I see that Harbor 0.5.0 is using Docker registry image 2.5.0 but Harbor 1.1.1 release is using Docker Registry 2.4.1, it sounds very strange, Is this right ?

Harbor 1.1.1 trace

May 16 10:45:30 vmz1registry01.corp docker-current/registry[2541]: time="2017-05-16T08:45:30.496160871Z" level=error msg="response completed with error" auth.user.name=admin err.code="manifest unknown" err.detail="filesystem: Path not found: /docker/registry/v2/repositories/MYREPO/javase/_manifests/revisions/sha256/29596032da96cc5656e9bd1491c44162a96ea65f2bbe783d0643046f0e5f68cf/signatures/sha256" err.message="manifest unknown" go.version=go1.7.3 http.request.host="registry:5000" http.request.id=31e0cf1b-8320-4054-8cfc-069fcd0bc488 http.request.method=GET http.request.remoteaddr="172.18.0.6:43128" http.request.uri="/v2/MYREPO/javase/manifests/1.1" http.request.useragent="Go-http-client/1.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=2.408857ms http.response.status=404 http.response.written=276 instance.id=a23902b9-32e5-43be-82ab-38efbb078294 service=registry vars.name="MYREPO/javase" vars.reference=1.1 version="v2.4.1+unknown"

Harbor 0.5.0

May 16 02:31:17 vmz1registry01.corp docker-current/registry[2508]: time="2017-05-16T00:31:17.643637708Z" level=debug msg="authorizing request" go.version=go1.6.
3 http.request.host=registry.global.paas.gsnetcloud.corp http.request.id=71cf1a6c-95d5-4b35-bf9b-d7208c5d4152 http.request.method=GET http.request.remoteaddr=180.214.23.124 http.requ
est.uri="/v2/MYPRO/javase/blobs/sha256:6f8976cdc6f041e4b463a0b6d9c86ed1190b6b23b71f9ba15052edef75afca7a" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/5206701-unsup
ported kernel/3.10.0-327.36.2.el7.x86_64 os/linux arch/amd64" instance.id=cce3dd54-499f-4c76-b17c-46078e27fed6 service=registry vars.digest="sha256:6f8976cdc6f041e4b463a0b6d9c86ed11
90b6b23b71f9ba15052edef75afca7a" vars.name="MYPRO/javase" version=v2.5.0

[reasonerjt]

Yep this is an issue in customizing the img, and will be fixed, could you pull the registry:2.6.0 image and update docker-compose.yml, then restart Harbor to see if the problem remains?

[cbarzu]

Ok, I try and I will let you know. Thanks.

UPDATE: Oh yeah, it works !!!! Changing registry image from vmware/registry:photon-2.6.0
to registry:2.6.0 solved the problem !!

Now all the tags are dispplayed and I can pull images !!

[reasonerjt]

@cbarzu thanks for update. We'll respin the image, however, we tried to install 0.5.0 do some replication, upgraded to 1.1.1 but unable to reproduce the issue you mentioned, will keep this issue open and may need your help to understand how to reproduce from scratch.

[roldancer]

@reasonerjt it is not easy to reproduce, we think it related to the docker client used during the docker push command, we could not found a pattern.

[ywk253100]

@roldancer We are trying to reproduce the issue and need some help from you if it is possible:

1. What is the earliest version of Harbor you use ?
2. Was it clean when you first used Harbor or was there already some image data?
3. What was the earliest version of docker you use to push image to Harbor?
   Thanks.

[cbarzu]

Hi @ywk253100.

1. Version 0.5.0
2. It was clean. We pulled images from our old registry and pushed to the new one.
3. 1.7.1
   

[reasonerjt]

@cbarzu , thanks a lot for answering it's helpful, and by "old registry" could you recall its version?
And were these images in the screenshot built by your own Dockerfiles or you pulled it from somewhere else?

And let me confirm, out of curiosity, @roldancer @cbarzu you guys are working on the same environment, right?

[cbarzu]

Hi @reasonerjt, the previous registry was version registry:0.9.1
In the attached image on the previous comment, last 3 images (4/10/2017 and 4/21/2017) were build and pushed directly to Harbor. The others were pulled from the other registry to my local and pushed to Harbor.

[cbarzu]

And yes, we are working together :)

[strahe]

Hi, i use the latest release(1.1.1), and have the same problem(can login on web), and i checked the vmware/registry default version is photon-2.6.0,
any idea ?

[ywk253100]

@strahe You can workaround it by replacing vmware/registry:photon-2.6.0 with vmware/registry:2.6.1-photon in docker-compose.yml.

[reasonerjt]

Set priority to medium as the fix is merged to master branch, will keep it open until the 1.1.2 is released.

[goodspd]

Hi,

Had this problem with vmware/registry:photon-2.6.0:

Project XXXX in repo YYYY

As sugested by @ywk253100 replaced vmware/registry:photon-2.6.0 with vmware/registry:2.6.1-photon

Now project XXXX in repo YYYY it's working but the problem "passed" to another project :(

Any ideias ?

NOTE: I don't have replicated repos!

Thanks!

[reasonerjt]

Closing it as the original issue is fixed. @goodspd If the problem remains could you please open an new issue?