rkt fetch: failure when pulling from artifactory docker repository

[ravibhure]

Environment

rkt Version: 1.18.0
appc Version: 0.8.8
Go Version: go1.7.3
Go OS/Arch: linux/amd64
Features: -TPM +SDJOURNAL
--
Linux 4.7.3-coreos-r2 x86_64
--
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1235.6.0
VERSION_ID=1235.6.0
BUILD_ID=2017-01-10-0545
PRETTY_NAME="Container Linux by CoreOS 1235.6.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"
--
systemd 231
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT -GNUTLS -ACL +XZ -LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD -IDN

What did you do?

core@ip-172-31-12-214 ~ $  sudo rkt fetch --insecure-options=all --debug=true --no-store docker://docker-local.ravibhure.com/coreos/flannel:v0.6.2

What did you expect to see?
Rkt will fetch image from docker private registry

What did you see instead?

core@ip-172-31-12-214 ~ $  sudo rkt fetch --insecure-options=all --debug=true --no-store docker://docker-local.ravibhure.com/coreos/flannel:v0.6.2
image: remote fetching from URL "docker://docker-local.ravibhure.com/coreos/flannel:v0.6.2"
image: fetching image from docker://docker-local.ravibhure.com/coreos/flannel:v0.6.2
fetch:
  └─error converting docker image to ACI
    └─registry doesn't support API v2 nor v1

[ravibhure]

This is similar to #3211 but it looks not fixed yet

[lucab]

I'm sorry but this report does not contain enough actionable information.

docker://docker-local.ravibhure.com/

Which registry software is this? Which API version does it implement? How is it private? Did you properly configure auth? What does a curl -v https://docker-local.ravibhure.com/v2/ say?

rkt Version: 1.18.0

Also, this rkt version is quite old to be relevant for debugging. Please try with something more recent (e.g. >= 1.24.0) and with --debug.

[ravibhure]

@lucab

This is private docker registry on Enterprise Artifactory 5.0.1 rev 50008

core@ip-172-31-12-214 ~ $ curl -v https://docker-local.ravibhure.com/v2/
* Hostname was NOT found in DNS cache
*   Trying xx.xx.xx.xxx...
* Connected to docker-local.ravibhure.com (xx.xx.xx.xxx) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using ECDHE-RSA-AES128-GCM-SHA256
* Server certificate:
*        subject: CN=*.ravibhure.com
*        start date: 2017-02-02 00:00:00 GMT
*        expire date: 2018-03-02 12:00:00 GMT
*        subjectAltName: docker-local.ravibhure.com matched
*        issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*        SSL certificate verify ok.
> GET /v2/ HTTP/1.1
> User-Agent: curl/7.35.0
> Host: docker-local.ravibhure.com
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Date: Wed, 12 Apr 2017 09:39:23 GMT
< Content-Type: application/json;charset=ISO-8859-1
< Content-Length: 77
< Connection: keep-alive
* Server Artifactory/5.0.1 is not blacklisted
< Server: Artifactory/5.0.1
< X-Artifactory-Id: ed4e3cb192b6b9ca:3b47d104:15a26e91d31:-8000
< Docker-Distribution-Api-Version: registry/2.0
< WWW-Authenticate: Bearer realm="https://docker-local.ravibhure.com:443/artifactory/api/docker/docker-local/v2/token",service="docker-local.ravibhure.com:443",scope="repository:docker-local:pull,push"
<
{
  "errors" : [ {
    "status" : 401,
    "message" : "Unauthorized"
  } ]
* Connection #0 to host docker-local.ravibhure.com left intact

I have updated the version to 1.25.0

core@ip-172-31-12-214 ~ $ rkt version
rkt Version: 1.25.0
appc Version: 0.8.10
Go Version: go1.7.4
Go OS/Arch: linux/amd64
Features: -TPM +SDJOURNAL

Still getting issue

core@ip-172-31-12-214 ~ $ sudo rkt fetch --insecure-options=image --debug=true  docker://docker-local.ravibhure.com/coreos/flannel:v0.6.2
image: remote fetching from URL "docker://docker-local.ravibhure.com/coreos/flannel:v0.6.2"
image: fetching image from docker://docker-local.ravibhure.com/coreos/flannel:v0.6.2
fetch:
  └─error converting docker image to ACI
    └─registry doesn't support API v2 nor v1

[ravibhure]

This work very well with docker

core@ip-172-31-12-214 ~ $ docker pull docker-local.ravibhure.com/coreos/flannel:v0.6.2
v0.6.2: Pulling from coreos/flannel

9495713bb019: Pull complete
835ac7243dcb: Pull complete
974fe91d03ee: Pull complete
d83fa8fedcec: Pull complete
cfaba0bcdd4d: Pull complete
Digest: sha256:2adcf7a7c21474dd6122303f5a1a9f3fa8196f051f51e0fab79bf341fbd158bd
Status: Downloaded newer image for docker-local.ravibhure.com/coreos/flannel:v0.6.2

core@ip-172-31-12-214 ~ $ docker images
docker-local.ravibhure.com/coreos/flannel                                   v0.6.2              b556ce4987f6        6 months ago        27.89 MB

[ravibhure]

Here is my rkt auth

core@ip-172-31-12-214 ~ $ cat /etc/rkt/auth.d/docker.json
{
      "rktKind": "dockerAuth",
      "rktVersion": "v1",
      "registries": ["docker-local.ravibhure.com"],
      "credentials":
       {
              "user": "admin",
              "password": "adminpassword"
       }
}

[lucab]

The v2 ping endpoint seems fine, so it is probably something down the chain. One worrysome detail is in the authentication header:

WWW-Authenticate: Bearer realm="https://docker-local.ravibhure.com:443/artifactory/api/docker/docker-local/v2/token",service="docker-local.ravibhure.com:443",scope="repository:docker-local:pull,push"

Here the service parameter doesn't match the registry endpoint identity due to the :443 suffix. I'd have to double-check the relevant RFC about the semantic of that field, but this looks like a misconfiguration on the registry side.

[ravibhure]

@lucab
I am not sure, but currently we are able to use the docker pull so are you saying there we need to add any specific configuration on artifactory for rkt or docker2aci ?

[lucab]

I'm saying I'd need to check what is the appropriate specification for that field to know what is the correct behavior/encoding/semantic, but the failure seems to be due to that :443 suffix in service identity and docker-client may be just ignoring the mismatch or trimming it.

In any case, the problem appears to be related to a very custom configuration of this repository (unless further debugging), in which case either removing the suffix from its configured identity (or adding it everywhere on the client side, but I doubt it) should work.

[ravibhure]

Thank you @lucab,

Looking for a solution so could you please look if this is the case or let me know if any further things requires to identify the issue.

[ravibhure]

@lucab
These logs are from from artifactory, may be this help to identify and fix the issue.

2017-04-12 16:07:40,022 [http-nio-8443-exec-11] [ERROR] (o.a.a.d.r.DockerResource:126) - Unsupported docker v1 repository request for 'docker-released'
2017-04-12 16:09:01,038 [http-nio-8443-exec-7] [ERROR] (o.a.a.d.r.DockerResource:126) - Unsupported docker v1 repository request for 'docker-released'
2017-04-12 16:10:43,646 [http-nio-8443-exec-11] [ERROR] (o.a.a.d.r.DockerResource:126) - Unsupported docker v1 repository request for 'docker-released'
2017-04-12 16:11:32,166 [http-nio-8443-exec-7] [ERROR] (o.a.a.d.r.DockerResource:126) - Unsupported docker v1 repository request for 'docker-released'
2017-04-12 16:12:01,522 [http-nio-8443-exec-4] [ERROR] (o.a.a.d.r.DockerResource:126) - Unsupported docker v1 repository request for 'docker-released'
2017-04-12 16:12:29,690 [http-nio-8443-exec-7] [ERROR] (o.a.a.d.r.DockerResource:126) - Unsupported docker v1 repository request for 'docker-released'

looks like rkt is using v1 api instead of v2 and we need to know how to enable rkt to use v2

[lucab]

@ravibhure I actually think it is only failing back to v1 because it can't complete v2 authentication due to the above mismatch in server identity. The problem itself isn't in trying to use v1, but in failing v2 handshake.

As I suggested above, as a workaround to this mismatch in your setup you could try either:

• setting the registry to identify itself as docker-local.ravibhure.com (without port suffix)
• using docker-local.ravibhure.com:443 everywhere on the rkt side (eg. in auth.d and for fetch)

[ravibhure]

After adding 443 the 401 goes away and now I get this:
 

core@ip-10-26-8-167 ~ $ sudo rkt fetch --insecure-options=all --debug=true --no-store docker://docker-local.ravibhure.com:443/coreos/flannel:v0.6.2
image: remote fetching from URL "docker://docker-local.ravibhure.com:443/coreos/flannel:v0.6.2"
image: fetching image from docker://docker-local.ravibhure.com:443/coreos/flannel:v0.6.2
fetch:
  └─error converting docker image to ACI
    └─registry doesn't support API v2 nor v1

[ravibhure]

@lucab
we tried to using docker-local.ravibhure.com:443 everywhere on the rkt side (eg. in auth.d and for fetch) but still not getting succeed to fetch the images.
I wonder if rkt only supporting v1 (just wild guess)

[ravibhure]

@lucab
Any update on this issue ? we are still waiting to hear from you.

[lucab]

As I stated above, this seems to be due a very specific configuration of your registry. There is not much we can do and debug from here, and it doesn't affect general usage. If you have a different guess, feel free to perform more analysis on your side to confirm/debunk my hypotesis, we'll happily take more details and patches.

On a technical side, the service field is defined here as "The name of the service which hosts the resource". So if there is a mismatch it should correctly fail.

The docker fetching code is in https://github.com/rkt/rkt/tree/master/vendor/github.com/appc/docker2aci/lib, and it may need some instrumention and on-field testing to properly debug what's going on. From my side, there is not much else I can add except from more raw speculation.

[ravibhure]

Thank you @lucab

[alekssaul]

Hey @ravibhure , I've seen this in a similar environment with identical symptoms.

I came across https://www.jfrog.com/jira/browse/RTFACT-13305 which seems relevant. This seems to be fixed in Artifactory v5.2.1 . Any chance you can test against that version?

[ravibhure]

Thanks @alekssaul,
We are working on upgrading our current Artifactory version 5.0.1 to the latest one (>=5.2.1) and I will update this thread once its done.

[jumanjiman]

Example of working command

$ sudo rkt fetch --no-store --debug --insecure-options=all docker://docker-dev.artifactory.dv.example.net/alpine:latest 
image: remote fetching from URL "docker://docker-dev.artifactory.dv.example.net/alpine:latest"
image: fetching image from docker://docker-dev.artifactory.dv.example.net/alpine:latest
Downloading sha256:627beaf3eaa [=============================] 1.91 MB / 1.91 MB
sha512-673a9603792ff981bc2b36b96d66edfd

Here's what worked for us.

versions

Container Linux

$ grep -i version /etc/os-release 
VERSION=1235.6.0
VERSION_ID=1235.6.0

rkt

$ rkt version
rkt Version: 1.18.0
appc Version: 0.8.8
Go Version: go1.7.3
Go OS/Arch: linux/amd64
Features: -TPM +SDJOURNAL

Artifactory Version 4.14.3 and 4.16.0

layout

+-------------+            +------------+            +--------------------+
|             |            |            |            |                    |
|  Container  |            |            |            |    Artifactory     |
|  Linux     --------------|    F5      --------------    4.14.3          |
|             |            |            |            |                    |
|             |            |            |            |                    |
+-------------+            +------------+            +--------------------+

rkt auth file

/etc/rkt/auth.d/docker.json

{
        "rktKind": "dockerAuth",
        "rktVersion": "v1",
        "registries": ["docker-dev.artifactory.dv.example.net", "docker-thirdparty.artifactory.example.net"],
        "credentials": {
                "user": "redacted",
                "password": "redacted"
        }
}

F5 config

Our docker rewrite solution uses the host based method, and the only requirement is that docker be in the name. The iRule that controls it all is below:

when HTTP_REQUEST {
    if { [HTTP::host] contains "docker" && [HTTP::uri] starts_with "/v2" } {
        scan [HTTP::host] %\[^.\].%\[^.\].%\[^.\].%\[^.\].%s REPO APP ENV DOMAIN TLD
        HTTP::uri /$APP/api/docker/$REPO[HTTP::uri]
        HTTP::header insert X-Artifactory-Override-Base-Url "https://$APP.$ENV.$DOMAIN.$TLD/$APP"
        HTTP::header insert Docker-Distribution-Api-Version "registry/2.0"
    }
}

:arrow_right: Note that /v2 does NOT have a trailing slash. :eyes:

Here's a snippet from a pcap that shows rkt auth.
It doesn't appear to use the WWW-Authenticate header mentioned elsewhere.

GET /artifactory/api/docker/docker-dev/v2 HTTP/1.1
Host: docker-dev.artifactory.dv.example.net
User-Agent: Go-http-client/1.1
Authorization: Basic cG1vcmdhbjpNM3Nvc3BoM3IzIQ==
Accept-Encoding: gzip
X-Artifactory-Override-Base-Url: https://artifactory.dv.example.net/artifactory
Docker-Distribution-Api-Version: registry/2.0

HTTP/1.1 200 OK
Server: Artifactory/4.16.0
X-Artifactory-Id: e275d87c961d552cd4958927f431d5735d1539f0
X-Artifactory-Node-Id: art1
Docker-Distribution-Api-Version: registry/2.0
Content-Type: application/json
Transfer-Encoding: chunked
Date: Fri, 05 May 2017 18:07:44 GMT

[jcollie]

I seem to be having the same problems pulling docker images that I built using the cloud.docker.com service, but it works just fine with image I built with hub.docker.com.

[ravibhure]

Thank you all,
We have fixed the issue to upgraded the artifactory server to v5.4.5

Finally it works on v5.4.5 and its correctly returning the Docker-Distribution-Api-Version: registry/2.0 header as well.