Rke: Feature Request: Ability to set base directory for private registry

[patrick0057]

RKE version:
0.1.17

Docker version: (docker version,docker info preferred)
[root@rkalvrsiel9 ~]# docker version
Client:
Version: 18.06.2-ce
API version: 1.38
Go version: go1.10.3
Git commit: 6d37f41
Built: Sun Feb 10 03:46:03 2019
OS/Arch: linux/amd64
Experimental: false

Server:
Engine:
Version: 18.06.2-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: 6d37f41
Built: Sun Feb 10 03:48:29 2019
OS/Arch: linux/amd64
Experimental: false

[root@rkalvrsiel9 ~]# docker info
Containers: 27
Running: 17
Paused: 0
Stopped: 10
Images: 10
Server Version: 18.06.2-ce
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-957.1.3.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.701GiB
Name: rkalvrsiel9.kau.roche.com
ID: 7SCW:CJOX:Q346:PB56:CZMP:XGAB:KLDL:7JF6:SFVA:NH3T:P37U:JSHZ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

Operating system and kernel: (cat /etc/os-release, uname -r preferred)

[root@rkalvrsiel9 ~]# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

[root@rkalvrsiel9 ~]# uname -r

3.10.0-957.1.3.el7.x86_64
[root@rkalvrsiel9 ~]#

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
VMWare

cluster.yml file:

nodes:
  - address: node1.company.com
    user: userwithprivileges
    ssh_key_path: pathtosshkey
    role: etcd,controlplane,worker
  - address: node2.company.com
    user: userwithprivileges
    ssh_key_path: pathtosshkey
    role: etcd,controlplane,worker
  - address: node3.company.com
    user: userwithprivileges
    ssh_key_path: pathtosshkey
    role: etcd,controlplane,worker
private_registries:
  - url: ourregistry.company.com/infrafolder/kubefolder
    user: userwithprivileges
    password: passwordofuser
    is_default: true

services:
  etcd:
    snapshot: true
    creation: 6h
    retention: 24h

cluster_name: cluster1
kubernetes_version: v1.13.4-rancher1-1

Steps to Reproduce:
Call rke up with this config

Results:
Calling rke up with this config causes rke to look for credentials for the default registry with ourregistry.company.com in config file. The config in its current state ourregistry.company.com/infrafolder/kubefolder does not work. It works if we use ourregistry.company.com in the config file but then we are forced to place the rancher folder directly in the root of our private registry.

We would like the flexibility to place rancher images deeper in our registry.

[alena1108]

We can do it in 2 ways:

• Allow specifying sub directory in the url param
  or
• Introduce a base_dir filed to the private_registry

[alena1108]

We can do it in 2 ways:
Allow specifying sub directory in the url param
or
Introduce a base_dir filed to the private_registry

Went with 1) for the fix. Now you can supply url like url:ourregistry.company.com/infrafolder/kubefolder

[alena1108]

@sangeethah can be tested from rancher, or using rke cli v1.0.0-rc4

[jiaqiluo]

The is validated with rke-v1.0.0-rc4

Steps:

• follow Rancher's documentation to prepare the private registry https://rancher.com/docs/rke/latest/en/config-options/private-registries/
• push all images to the private registry with prefix my-registry.com/qa, for example:

my-registry.com/qa/test/rancher/metrics-server                            v0.3.1                         61a0c90da56e        14 months ago       40.8MB
my-registry.com/qa/test/rancher/pstauffer-curl                            v1.0.3                         e1ca8ea8aa9c        16 months ago       5.93MB
my-registry.com/qa/test/rancher/coredns-coredns                           1.1.2                          8558f8c47fd7        19 months ago       46.4MB
my-registry.com/qa/test/rancher/coreos-flannel                            v0.10.0                        f0fad859c909        22 months ago       44.6MB
my-registry.com/qa/test/rancher/coreos-flannel                            v0.10.0-rancher1               f0fad859c909        22 months ago       44.6MB
my-registry.com/qa/test/rancher/calico-ctl                                v2.0.0                         dd009afe5809        23 months ago       41.9MB
my-registry.com/qa/test/rancher/pause                                     3.1                            da86e6ba6ca1        23 months ago       742kB
my-registry.com/qa/test/rancher/coreos-configmap-reload                   v0.0.1                         3129a2ca29d7        2 years ago         4.79MB

• use rke to provisoin a cluster with the following cluster.yml file

nodes:
  - address: public-ip
    internal_address: private-ip
    user: ubuntu
    role: [etcd, controlplane, worker]
    ssh_key_path: /path/to/ssh/key

services:
  etcd:
    snapshot: true
    creation: 6h
    retention: 24h

private_registries:
- url: my-registry.com/qa/test
  user: username
  password: "password"
  is_default: true

Results:

• the cluster is provisioned successfully
• the log also shows images are pulled from the private registry

INFO[0001] Pulling image [my-registry.com/qa/test/rancher/rke-tools:v0.1.51] on host [18.144.17.163], try #1
INFO[0004] Image [my-registry.com/qa/test/rancher/rke-tools:v0.1.51] exists on host [18.144.17.163]
INFO[0005] Starting container [rke-etcd-port-listener] on host [18.144.17.163], try #1
INFO[0006] [network] Successfully started [rke-etcd-port-listener] container on host [18.144.17.163]
INFO[0007] Image [my-registry.com/qa/test/rancher/rke-tools:v0.1.51] exists on host [18.144.17.163]
INFO[0007] Starting container [rke-cp-port-listener] on host [18.144.17.163], try #1
INFO[0008] [network] Successfully started [rke-cp-port-listener] container on host [18.144.17.163]
INFO[0008] Image [my-registry.com/qa/test/rancher/rke-tools:v0.1.51] exists on host [18.144.17.163]
INFO[0008] Starting container [rke-worker-port-listener] on host [18.144.17.163], try #1
INFO[0009] [network] Successfully started [rke-worker-port-listener] container on host [18.144.17.163]
INFO[0009] [network] Port listener containers deployed successfully
INFO[0009] [network] Running control plane -> etcd port checksmy-registry.com

[jiaqiluo]

The bug is reproduced with rke v0.2.8

follow the same steps as the above comment provided, it fails to pull images from the private registry with the following error

INFO[0001] [network] Pulling image [my-registry.com/qa/test/rancher/rke-tools:v0.1.42] on host [52.53.229.60]
FATA[0001] [Can't pull Docker image [my-registry.com/qa/test/rancher/rke-tools:v0.1.42] for host [52.53.229.60]: Error response from daemon: Get https://my-registry.com/v2/qa/test/rancher/rke-tools/manifests/v0.1.42: no basic auth credentials]

also confirmed that the image can be pulled if I do the pull inside the target instance.

[jiaqiluo]

The feature is validated in Rancher:v2.3-head 5ac58c6d6

• run Rancher
• set the system default registry to my-registry.com/qa/test
• add a custom cluster via Rancher UI with private registry enabled and set its url to my-registry.com/qa/test
• follow Rancher's documentation to finish the cluster creation process

Results:

• the cluster is provisioned and active
• ssh into the node and confirm that all images are pulled from the private registry