K3s: [Documentation] Private Registry

[davidnuzik]

Private registry is only briefly mentioned on the Airgap install page. This should be it's own page with information on how to set up Private registry via all supported methods that containerd supports.
For example http (with and without user/pass authentication) and https (TLS) (with and without user/pass authentication).

How should the registries.yaml properly be set up for each configuration type?

Docs should explain that insecure TLS is not supported (by containerd). Update: containerd may support insecure TLS now in a newer version but it's not recommended so we currently don't have plans to document this method.

Anything needed for Docker? And what is the priority of this? Needs a separate issue?

[VladoPortos]

Yup I agree, its the last thing I'm stuck on before I start deploying images to cluster...

[j0holo]

There is an issue describe here with a tiny example, have not tested it yet.

[j0holo]

Currently trying to get it to work, created my own docker registry with the following config:

version: 0.1
log:
  fields:
    service: registry
storage:
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /var/lib/registry
http:
  addr: :5000
  net: tcp
  host: https://images.dest.lan
  draintimeout: 60s
auth:
  htpasswd:
    realm: lab-realm
    path: /etc/docker/registry/htpasswd
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3
validation:
  enabled: false

I run an Nginx proxy in front of it which handles TLS, the max_client_body_size needs to be set to 0 (unlimited) otherwise Nginx will throw an error that the body size is to large.

I can push to it without a problem, but k3s doesn't trust the certificate which I added to /usr/local/share/ca-certificates and ran update-ca-certificates.

In /etc/rancher/k3s/registries.yaml I have the following config:

mirrors:
  images.dest.lan:
    endpoint:
    - https://images.dest.lan
configs:
  images.dest.lan:
    auth:
      username: REDACTED
      password: REDACTED

As recommended by this issue.

The errors that k3s throws in the event log:
Failed to pull image "images.dest.lan/nginx:latest": rpc error: code = Unknown desc = failed to pull and unpack image "images.dest.lan/nginx:latest": failed to resolve reference "images.dest.lan/nginx:latest": failed to do request: Head https://images.dest.lan/v2/nginx/manifests/latest: x509: certificate signed by unknown authority

Restarting the k3s master or k3s-agents does not help. Does k3s not support the default certificate store of Linux (Debian in my case)?

If we can solve this issue I can write some documentation on how to use a private registry with k3s.

EDIT: According to this blog post: https://itnext.io/setup-a-private-registry-on-k3s-f30404f8e4d3
adding the certs (in my case CA and intermediate CA) to /usr/local/share/ca-certificates and it should work. Hmmmm, weird.

[j0holo]

Okay, I get the master node working. Stupid but .pem files do not get picked up by update-ca-certificates, so I renamed them to *.crt. The worker nodes are still having authentication issues because they don't have a /etc/rancher/k3s directory.

EDIT: Found this solution for the k3s-agents, but don't have the time to test it today. Will try it out tomorrow.

[rajr0]

Thanks @j0holo for the above clues :)
I'm able make my k3s cluster (running containerd on 4 raspberry-pi 's) pull from my private-registry.
I've been using this private-registry (running on raspberrypi-1.local:5000) for my docker images
I put my ca.crt in /etc/docker/certs.d/raspberrypi-1.local:5000/ca.crt, so docker login raspberrypi-1.local:5000 worked.

Now I added /etc/rancher/k3s/registries.yaml on all my k3s nodes (including master) and k3s is able to pull from my private registry.

pi@raspberrypi-4:~ $  cat /etc/rancher/k3s/registries.yaml
mirrors:
  raspberrypi-1.local:5000:
    endpoint:
            - https://raspberrypi-1.local:5000
configs:
  raspberrypi-1.local:5000:
    auth:
      username: admin
      password: admin
    tls:
      ca_file: "/etc/docker/certs.d/raspberrypi-1.local:5000/ca.crt"

I run my private-registry on raspberrypi-1 using the below script. This script generates self-signed certificates and installs on all my nodes (raspberrypi-[1-6]).

#! /bin/bash
REGISTRY_FQDN=${REGISTRY_FQDN:-raspberrypi-1.local}
echo "Registry FQDN: ${REGISTRY_FQDN}"

REGISTRY_USER=admin
REGISTRY_PASSWD=admin

_PWD=`pwd`
rmdir private-registry
mkdir private-registry
cd private-registry
mkdir -p auth certs data

## Generate your own certificate:
echo "Generating private certificate"
echo "Be sure to use the Common Name == ${REGISTRY_FQDN}"
openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/domain.key -x509 -days 365 -out certs/domain.crt
  #--- Be sure to use the Common Name == ${REGISTRY_FQDN}

## Create a directory to store your htpasswd file, create the credentials, then remove the temporary container:
docker run --name htpasswd --entrypoint htpasswd registry:2 -Bbn ${REGISTRY_USER} ${REGISTRY_PASSWD} >>  auth/htpasswd
docker stop htpasswd
docker rm htpasswd

SSH_CMDS=" \
        echo -n 'Now Inside Host:' && cat /etc/hostname && \
        cd /tmp && \
        sudo mkdir -p /etc/docker/certs.d/${REGISTRY_FQDN}:5000 && \
        sudo cp domain.crt /etc/docker/certs.d/${REGISTRY_FQDN}:5000/ca.crt && \
        echo -n 'Restarting docker on:' && cat /etc/hostname && \
        sudo service docker restart && \
        echo -n 'Leaving Host:' && cat /etc/hostname"

for i in raspberrypi-1  raspberrypi-2  raspberrypi-3 raspberrypi-4 raspberrypi-5 raspberrypi-6 
do
  echo $i
  scp certs/domain.crt pi@${i}.local:/tmp
  ssh -t pi@${i}.local $SSH_CMDS
done

docker run -d \
  --name registry \
  --restart=always \
  -p 5000:5000 \
  -e "REGISTRY_HTTP_TLS_CERTIFICATE=/cert/domain.crt" \
  -e "REGISTRY_HTTP_TLS_KEY=/cert/domain.key" \
  -e "REGISTRY_AUTH=htpasswd" \
  -e "REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd" \
  -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
  -v ${_PWD}/private-registry/data:/var/lib/registry \
  -v ${_PWD}/private-registry/certs:/cert \
  -v ${_PWD}/private-registry/auth:/auth \
  registry:2

## TESTS
### try logging in
docker login ${REGISTRY_FQDN}:5000

### try push an image
docker pull hello-world
docker tag hello-world ${REGISTRY_FQDN}:5000/hello-world
docker push ${REGISTRY_FQDN}:5000/hello-world

[j0holo]

@rajr0 okay, cool. So did you create /etc/rancher/k3s/ directory on you worker nodes, because by default this directory doesn't exist on my 2 worker nodes? If that is the case I think we cracked this issue.

[j0holo]

Just tested it! It works, mkdir -p /etc/rancher/k3s and move the registries.yaml file there. Works really well. Got a nginx pod running on each node!

@davidnuzik is there a way I can commit my finding (written in a more fitting style for documentation ofcourse) to the k3s documentation?

[daxmc99]

@j0holo A PR to https://github.com/rancher/docs/tree/master/content/k3s would likely be a great place to start!

[daxmc99]

Also more docs here https://github.com/rancher/k3d/wiki/Examples%3A-Private-Registry

[j0holo]

@daxmc99 Will look into it tomorrow, thanks.

[davidnuzik]

https://github.com/rancher/docs/pull/2194 is merged in.

Thank you @j0holo for you time and contribution I'm sorry we could not get what you had merged in. If you're interested in contributing to the docs in the future feel free to message me on the rancher users slack https://slack.rancher.io/
This can help to touch base if there is something you want work on to make sure it doesn't conflict with something we have in progress. We are making lot of little docs changes currently so there is some risk of conflicts with the community presently.

[Id2ndR]

There is an error on the documentation: each item of mirrors and configs need an additional colon.
Otherwise, there is an error like _line 3: mapping values are not allowed in this context_ when K3S is restarted.

Example:

mirrors:
  mycustomreg.com:5000
    endpoint:
      - "https://mycustomreg.com:5000"

Should be

mirrors:
  mycustomreg.com:5000:
    endpoint:
      - "https://mycustomreg.com:5000"

[cjellick]

cc @catherineluse @davidnuzik