Cri-o: Pulling images from private registry over http

[outcoldman]

I have added private registry and insecure registry to the crio.server

ExecStart=/usr/bin/crio \
          --registry=10.0.2.2:5000 \
          --insecure-registry=10.0.2.2:5000 \

But when I try to pull the image I get

crictl -D pull 10.0.2.2:5000/myimage:5.0
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:10.0.2.2:5000/myimage:5.0,},Auth:nil,SandboxConfig:nil,}
DEBU[0000] PullImageResponse: nil
FATA[0000] pulling image failed: rpc error: code = Unknown desc = Invalid image name "", expected colon-separated transport:reference

I tried to set the log-level to debug and found that crio tries to pull from https anyway.

# crio -v
crio version 1.11.3-dev
commit: ""

So feels like that the real problem that there is no way to tell crio to use http protocol for registry.

[runcom]

I tried to set the log-level to debug and found that crio tries to pull from http anyway.

can you paste the relevant debug output where you think crio is pulling http anyway? That can help diagnose the issue, the pull image error is different from what you're saying.

[outcoldman]

@runcom sorry, my misprint, it is pulling from https, while my registry is http, I will post a debug output in few days.

[outcoldman]

The list of steps I have performed. Updated crio.service

ExecStart=/usr/bin/crio \
      --log-level=debug \
      --insecure-registry=10.0.2.2:5000 \
      --registry=10.0.2.2:5000 \
          $CRIO_STORAGE_OPTIONS \
          $CRIO_NETWORK_OPTIONS

Trying to pull the image from registry

crictl -D pull collectorforkubernetes:5.0

In debug logs I see that it tries to connect to my insecure registry via https and does not tries with http

Sep 03 19:19:02 ubuntu-xenial crio[3634]: time="2018-09-03 19:19:02.579866179Z" level=debug msg="PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:collectorforkubernetes:5.0,},Auth:nil,SandboxConfig:nil,}"
Sep 03 19:19:02 ubuntu-xenial crio[3634]: time="2018-09-03 19:19:02.580021080Z" level=debug msg="Using registries.d directory /etc/containers/registries.d for sigstore configuration"
Sep 03 19:19:02 ubuntu-xenial crio[3634]: time="2018-09-03 19:19:02.580161720Z" level=debug msg=" Using "default-docker" configuration"
Sep 03 19:19:02 ubuntu-xenial crio[3634]: time="2018-09-03 19:19:02.580171584Z" level=debug msg=" No signature storage configuration found for 10.0.2.2:5000/collectorforkubernetes:5.0"
Sep 03 19:19:02 ubuntu-xenial crio[3634]: time="2018-09-03 19:19:02.580184469Z" level=debug msg="Looking for TLS certificates and private keys in /etc/docker/certs.d/10.0.2.2:5000"
Sep 03 19:19:02 ubuntu-xenial crio[3634]: time="2018-09-03 19:19:02.580201568Z" level=debug msg="GET https://10.0.2.2:5000/v2/"
Sep 03 19:19:02 ubuntu-xenial crio[3634]: time="2018-09-03 19:19:02.582035992Z" level=debug msg="Ping https://10.0.2.2:5000/v2/ err &url.Error{Op:"Get", URL:"https://10.0.2.2:5000/v2/", Err:(*errors.errorString)(0xc4201fec00)}"
Sep 03 19:19:02 ubuntu-xenial crio[3634]: time="2018-09-03 19:19:02.582090612Z" level=debug msg="GET https://10.0.2.2:5000/v1/_ping"
Sep 03 19:19:02 ubuntu-xenial crio[3634]: time="2018-09-03 19:19:02.588261324Z" level=debug msg="Ping https://10.0.2.2:5000/v1/_ping err &url.Error{Op:"Get", URL:"https://10.0.2.2:5000/v1/_ping", Err:(*errors.errorString)(0xc4201fee70)}"
Sep 03 19:19:02 ubuntu-xenial crio[3634]: time="2018-09-03 19:19:02.588328261Z" level=debug msg="error preparing image 10.0.2.2:5000/collectorforkubernetes:5.0: pinging docker registry returned: Get https://10.0.2.2:5000/v2/: http: server gave HTTP response to HTTPS client"

[runcom]

Thanks for the reproducer, I'll try it and see what's going on.

[runcom]

The first error you got is fixed:

FATA[0000] pulling image failed: rpc error: code = Unknown desc = Invalid image name "", expected colon-separated transport:reference

by https://github.com/kubernetes-incubator/cri-o/pull/1772/commits/3c6f266287c49636dfafd88c783c7d5353b829f6

I'm gonna try and fix and reproduce the other one

[runcom]

@outcoldman just saw that you're probably using master, could you test with #1772 (the latest commit) and see if it fixes it for you?

[uablrek]

Works for me.

Version

# crio -v
crio version 1.12.0-dev
commit: "36ef2abc9488ae325f69698fa991c2277e8e7ef0"

Config

insecure_registries = [
 "172.17.0.0/16"
]
registries = [
 "172.17.0.2:5000",
 "docker.io"
]

Image list

# crictl --runtime-endpoint=unix:///var/run/crio/crio.sock images
IMAGE                            TAG                 IMAGE ID            SIZE
172.17.0.2:5000/library/alpine   3.8                 196d12cf6ab19       4.67MB
...

Log

time="2018-10-14 06:19:24.860398704Z" level=debug msg=" No signature storage configuration found for 172.17.0.2:5000/library/alpine:3.8" 
time="2018-10-14 06:19:24.860437748Z" level=debug msg="Looking for TLS certificates and private keys in /etc/docker/certs.d/172.17.0.2:5000" 
time="2018-10-14 06:19:24.860469045Z" level=debug msg="GET https://172.17.0.2:5000/v2/" 
time="2018-10-14 06:19:24.867001625Z" level=debug msg="Ping https://172.17.0.2:5000/v2/ err &url.Error{Op:"Get", URL:"https://172.17.0.2:5000/v2/", Err:(*errors.errorString)(0xc0001f70b0)}" 
time="2018-10-14 06:19:24.867042571Z" level=debug msg="GET http://172.17.0.2:5000/v2/" 
time="2018-10-14 06:19:24.878694601Z" level=debug msg="Ping http://172.17.0.2:5000/v2/ status 200" 
time="2018-10-14 06:19:24.879070430Z" level=debug msg="GET http://172.17.0.2:5000/v2/library/alpine/manifests/3.8" 
time="2018-10-14 06:19:24.885976934Z" level=debug msg="parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]172.17.0.2:5000/library/alpine:3.8"" 
time="2018-10-14 06:19:24.886028737Z" level=debug msg="reference "[overlay@/var/lib/containers/storage+/var/run/containers/storage]172.17.0.2:5000/library/alpine:3.8" does not resolve to an i
mage ID" 
time="2018-10-14 06:19:24.888810271Z" level=debug msg="parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]172.17.0.2:5000/library/alpine:3.8"" 
time="2018-10-14 06:19:24.888996131Z" level=debug msg="Using registries.d directory /etc/containers/registries.d for sigstore configuration" 
time="2018-10-14 06:19:24.889018371Z" level=debug msg=" No signature storage configuration found for 172.17.0.2:5000/library/alpine:3.8" 
time="2018-10-14 06:19:24.889041526Z" level=debug msg="Looking for TLS certificates and private keys in /etc/docker/certs.d/172.17.0.2:5000" 
time="2018-10-14 06:19:24.889072587Z" level=debug msg="GET https://172.17.0.2:5000/v2/" 
time="2018-10-14 06:19:24.891369210Z" level=debug msg="Ping https://172.17.0.2:5000/v2/ err &url.Error{Op:"Get", URL:"https://172.17.0.2:5000/v2/", Err:(*errors.errorString)(0xc0001c71e0)}" 
time="2018-10-14 06:19:24.891411024Z" level=debug msg="GET http://172.17.0.2:5000/v2/" 
time="2018-10-14 06:19:24.900861216Z" level=debug msg="Ping http://172.17.0.2:5000/v2/ status 200" 
time="2018-10-14 06:19:24.901026551Z" level=debug msg="GET http://172.17.0.2:5000/v2/library/alpine/manifests/3.8" 
time="2018-10-14 06:19:24.910872423Z" level=debug msg="IsRunningImageAllowed for image docker:172.17.0.2:5000/library/alpine:3.8" 
time="2018-10-14 06:19:24.910903933Z" level=debug msg=" Using default policy section" 
time="2018-10-14 06:19:24.910915121Z" level=debug msg=" Requirement 0: allowed" 
time="2018-10-14 06:19:24.910923494Z" level=debug msg="Overall: allowed" 
time="2018-10-14 06:19:24.910976736Z" level=debug msg="Downloading /v2/library/alpine/blobs/sha256:196d12cf6ab19273823e700516e98eb1910b03b17840f9d5509f03858484d321" 
time="2018-10-14 06:19:24.910998443Z" level=debug msg="GET http://172.17.0.2:5000/v2/library/alpine/blobs/sha256:196d12cf6ab19273823e700516e98eb1910b03b17840f9d5509f03858484d321" 
time="2018-10-14 06:19:24.917571598Z" level=debug msg="Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution
.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json]" 
time="2018-10-14 06:19:24.917605191Z" level=debug msg="... will first try using the original manifest unmodified" 
time="2018-10-14 06:19:24.917729898Z" level=debug msg="Downloading /v2/library/alpine/blobs/sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde" 
time="2018-10-14 06:19:24.917761034Z" level=debug msg="GET http://172.17.0.2:5000/v2/library/alpine/blobs/sha256:4fe2ade4980c2dda4fc95858ebb981489baec8c1e4bd282ab1c3560be8ff9bde" 
time="2018-10-14 06:19:24.923711848Z" level=debug msg="Detected compression format gzip" 
time="2018-10-14 06:19:24.923802350Z" level=debug msg="Using original blob without modification" 

[uablrek]

Try it yourself in ~10 min if you use Ubuntu? see;
https://github.com/Nordix/xcluster

Or specifically; https://github.com/Nordix/xcluster/tree/master/ovl/private-reg

[rhatdan]

Looks like this is fixed? Reopen if you disagree.