Harbor: "failed to fetch artifacts: unexpected end of JSON input" JFrog Artifactory replication

Created on 9 Jul 2020 · 23Comments · Source: goharbor/harbor

I have setup an endpoint that connects successfully:

And I created a replication rule, that appears correct (I want to pull all images in):

However, when triggering, the replication rule it fails with:

the execution X failed: failed to fetch artifacts: unexpected end of JSON input

If I look at the core logs I see:

2020-07-08T21:20:46Z [ERROR] [/replication/operation/controller.go:108]: the execution X failed: failed to fetch artifacts: unexpected end of JSON input

The fact that it fails SO FAST without logs, it makes me think that the execution fails before even starting.

I dug into the controller code and nothing jumped out, I also looked at the Jfrog adapter and saw nothing to work with.

I validated that the main API does respond correctly (ex: https://artifactory.example.com/artifactory/api/repositories?type=local&packageType=docker).

It seems tentatively related to @renjithkm86's issue https://github.com/goharbor/harbor/issues/12363

Artifactory Version: 6.15.1
Harbor Version: v2.0.1
Kubernetes Version: 1.17.5

arereplication kinbug priorithigh targe2.0.2 targe2.1.0

Source

jimangel

All 23 comments

Some additional details about the Artifcatory setup:

It is HA / load balanced (F5) behind a tomcat process. JFrog Artifactory provided the following rewrites which may be the cause of some issues:

AllowEncodedSlashes On
RewriteEngine on

RewriteCond %{REQUEST_SCHEME} (.*)
RewriteRule (.*) - [E=my_scheme:%1]

# For docker requests
RewriteRule "^/(v2)/(.*)$" "/artifactory/$1/$2" [P]
RewriteRule ^/$                /artifactory/webapp/ [R,L]
RewriteRule ^/artifactory(/)?$      /artifactory/webapp/ [R,L]
RewriteRule ^/artifactory/webapp$   /artifactory/webapp/ [R,L]

RequestHeader set X-Forwarded-Port 4443
## NOTE: {my_scheme} requires a module which is supported only from apache version 2.4 and above
RequestHeader set X-Forwarded-Proto %{my_scheme}e
RequestHeader set X-Artifactory-Override-Base-Url %{my_scheme}e://artifactory.example.com/artifactory
ProxyPassReverseCookiePath /artifactory /artifactory

The actual docker HTTP settings in the Artifactory GUI are using the simple direct access settings, so nothing fancy.

jimangel on 9 Jul 2020

i get the same error when I try to pull from artifactory too. We have ours behind traefik.

claflico on 10 Jul 2020

👍1

Additional testing:

Tried circumventing the F5 by hitting Apache (:4443) directly and had the same results.
Tried circumventing both the F5 and Apache by hitting Tomcat directly (:8443) and had the same results.

jimangel on 10 Jul 2020

Can you try docker pull from jfrog first to check if it works ?

bitsf on 16 Jul 2020

Yes, pulling works:

# docker pull artifactory.example.com/docker-anonymous/gi/quay.io/coreos/configmap-reload:v0.0.1
v0.0.1: Pulling from docker-anonymous/gi/quay.io/coreos/configmap-reload
e6ff3299c145: Pull complete
a3ed95caeb02: Pull complete
Digest: sha256:f58cb3b14aea0fc5b5f1d8f978218036b48207a2eb53079e3af9ce14db5f0676
Status: Downloaded newer image for artifactory.example.com/docker-anonymous/gi/quay.io/coreos/configmap-reload:v0.0.1
artifactory.example.com/docker-anonymous/gi/quay.io/coreos/configmap-reload:v0.0.1

There are many virtual repositories but the endpoint will only take the root URL (not a specific remote repository).

jimangel on 16 Jul 2020

@jimangel you can try curl following endpoint url, check which can success
https://artifactory.example.com/docker-anonymous/v2/
https://artifactory.example.com/docker-anonymous/gi/v2/
https://artifactory.example.com/docker-anonymous/gi/quay.io/v2/

And use that url without v2/ as the endpoint url

@chlins please also help confirm how you test this before.

bitsf on 17 Jul 2020

@jimangel Can you provide your jfrog HTTP setting - Docker Access Method? and the type of your repo, local、remote or virtual？

chlins on 17 Jul 2020

The v2 endpoints do not work for Artifactory in a traditional docker way, the v2 endpoints are under their own API endpoints. For example, it's not https://artifactory.example.com/docker-anonymous/v2/ it's https://artifactory.example.com/artifactory/api/docker/docker-anonymous/v2/

Artifactory docs: https://jfrog.com/knowledge-base/how-to-use-docker-registry-api-with-artifactory-docker-repository-when-not-using-docker-client/

Also, are you testing with Artifactory SaaS or on-prem? Also do you know what version it is? I understand there is a big change from 6.x -> 7.x

@chlins HTTP setting is commented above here (nothing fancy)

jimangel on 18 Jul 2020

I tested it on my machine. it worked well. Below is my environment information.

jfrog artifactory:

harbor registry configuration:

replication configuration:

execution:

after pushed:

@jimangel

chlins on 19 Jul 2020

@chlins does it work for pull too?

jimangel on 19 Jul 2020

Looks like worked, @jimangel

chlins on 19 Jul 2020

Interesting! How many repositories do you have set up and are they local, remote, or virtual?

jimangel on 19 Jul 2020

The first time i created a push rule, so this rule synced devops/redis:6.0 image to jfrog, which created a local type docker regsitry named 'devops', and contains a redis:6.0 image, the second time, i deleted the devops/redis:6.0 in harbor, and pull it by from jfrog by replication rule, it also pulled success.

chlins on 19 Jul 2020

By the way，my jfrog artifactory deployed by a very simple way: 'docker run -d -p 8081:8081 docker.bintray.io/jfrog/artifactory-pro:6.19.1'

chlins on 19 Jul 2020

👍1

@chlins is replicate one specific image, it will work.
but @jimangel you want to replicate all images in the jfrog artifactory, this maybe the difference.

Just like i got a repo in jfrog AF like /docker/test/ and in this repo have two images: /docker/test/image1:latest and /docker/test/image2:latest

And we set pullbased replication policy in harbor, repo name: /docker/test, tag is none by default, and we will get this error: failed to fetch artifacts: unexpected end of JSON input

So this problem is how to replicate all images in the repositeries from jfrog artifactory to harbor, not just one image? @reasonerjt

cslanjian on 19 Jul 2020

@cslanjian When i try to pull image with repo filter like xxx/*, got error: failed to fetch artifacts: unexpected end of JSON input, i also tested one image but multi tags with tag filter *, it worked well. Maybe some error when replicate multi images, thank you your experiment, i will take some time to fix it.

chlins on 20 Jul 2020

@jimangel Can you try to replicate one specific image, to verify @cslanjian 's explanation? or we can find other new problems.

chlins on 20 Jul 2020

Just to chime in, unless something has changed with Artifactory then I believe that the official deployment instructions are to use either subdomains or different ports for each local docker repo but then put them behind some sort of load-balancer with re-write rules so the LB handles the SSL offload....which is how we have ours setup. We are using the subdomain configuration and then have traefik re-write rules directing the calls to the correct path in artifactory....we are able to pull/push using docker to multiple local repos and then pull from remote/virtual repos....just not with harbor.

Happy to help test tomorrow if there is something particular you'd like me to try.

claflico on 20 Jul 2020

@chlins You should write a new FetchArtifacts() method but not used the base one from native.adapter, just like gitlab adapter's implement.
In order to implement the jfrog API like
curl -X GET "http://:8081/artifactory/api/docker//v2//tags/list" -H "X-JFrog-Art-Api:AKCp2UNCt2ENCPMX2LUQn2kkYfpDm2E4LgE6EKR3JEsWDXGbJxY18LsEvkYAGWmnKLddV88Hw"

bitsf on 20 Jul 2020

@chlins chiming in for @jimangel

We were able to test syncing a single image like your post demonstrated and it did in fact work -- thanks for that.

Example working replication rule:
Capture

The issue is that 'docker-anonymous' is an entire repo within our Artif instance that contains multiple images:
artif

If I try to create a replication rule that looks for everything underneath that repo:
Capture2

It then fails with the 'unexpected end of JSON' error message.

Like @bitsf commented above, I believe it's just that every registry V2 api call that is made for pull replication needs to be updated to use the artif specific pathing. In this specific case, it would need to grab the name of the repository from this replication rule setting ('docker-anonymous') and add it to the artif V2 call: http://artifactory.example:8081/artifactory/api/docker-anonymous/v2/_catalog, for example.

benjaques on 20 Jul 2020

👍1

@bitsf @chlins Could you clarify if there is a PR to fix this problem? We wanna clarify if this can be fixed in v2.0.2

reasonerjt on 21 Jul 2020

I have successfully replicated in version 2.0.2, but it stopped working in version 2.1.0
2020-10-12T07:14:37Z [INFO] [/replication/transfer/image/transfer.go:127]: client for source registry [type: jfrog-artifactory, URL: http://artifactory.localdomen, insecure: true] created 2020-10-12T07:14:37Z [INFO] [/replication/transfer/image/transfer.go:137]: client for destination registry [type: harbor, URL: http://core:8080, insecure: true] created 2020-10-12T07:14:37Z [INFO] [/replication/transfer/image/transfer.go:170]: copying docker/autostand/dnsmasq:[1.0,latest](source registry) to temp/dnsmasq:[1.0,latest](destination registry)... 2020-10-12T07:14:37Z [INFO] [/replication/transfer/image/transfer.go:191]: copying docker/autostand/dnsmasq:1.0(source registry) to temp/dnsmasq:1.0(destination registry)... 2020-10-12T07:14:37Z [INFO] [/replication/transfer/image/transfer.go:319]: pulling the manifest of artifact docker/autostand/dnsmasq:1.0 ... 2020-10-12T07:14:37Z [ERROR] [/replication/transfer/image/transfer.go:322]: failed to pull the manifest of artifact docker/autostand/dnsmasq:1.0: http status code: 404, body: {"errors":[{"code":"MANIFEST_UNKNOWN","message":"The named manifest is not known to the registry.","detail":{"manifest":"docker/autostand/dnsmasq/1.0/manifest.json"}}]} 2020-10-12T07:14:37Z [ERROR] [/replication/transfer/image/transfer.go:175]: http status code: 404, body: {"errors":[{"code":"MANIFEST_UNKNOWN","message":"The named manifest is not known to the registry.","detail":{"manifest":"docker/autostand/dnsmasq/1.0/manifest.json"}}]} 2020-10-12T07:14:37Z [INFO] [/replication/transfer/image/transfer.go:191]: copying docker/autostand/dnsmasq:latest(source registry) to temp/dnsmasq:latest(destination registry)... 2020-10-12T07:14:37Z [INFO] [/replication/transfer/image/transfer.go:319]: pulling the manifest of artifact docker/autostand/dnsmasq:latest ... 2020-10-12T07:14:37Z [ERROR] [/replication/transfer/image/transfer.go:322]: failed to pull the manifest of artifact docker/autostand/dnsmasq:latest: http status code: 404, body: {"errors":[{"code":"MANIFEST_UNKNOWN","message":"The named manifest is not known to the registry.","detail":{"manifest":"docker/autostand/dnsmasq/latest/manifest.json"}}]} 2020-10-12T07:14:37Z [ERROR] [/replication/transfer/image/transfer.go:175]: http status code: 404, body: {"errors":[{"code":"MANIFEST_UNKNOWN","message":"The named manifest is not known to the registry.","detail":{"manifest":"docker/autostand/dnsmasq/latest/manifest.json"}}]} 2020-10-12T07:14:37Z [ERROR] [/replication/transfer/image/transfer.go:181]: got error during the whole transfer period, mark the job failure

rgadeev on 12 Oct 2020

in 2.1.1 replication didnt work. In Version v2.0.0-87602132 replication works well
Harbor 2.1.1
Artifactory 7.7.3
Nov 5 11:00:44 172.31.0.1 core[1039]: 2020-11-05T11:00:44Z [ERROR] [/replication/operation/controller.go:103]: the execution 5 failed: failed to fetch artifacts: json: cannot unmarshal object into Go value of type []*jfrog.repository