Rke: Allow user to provide internal-use images for air-gapped setups

[moelsayed]

We need to allow users to provide their own cert downloader and nginx proxy images to allow running using private registries and air-gapped deployments.

[galal-hussein]

all images used and statically hardcoded:

• alpine:latest
• rancher/rke-nginx-proxy:0.1.0
• rancher/rke-cert-deployer:0.1.0
• quay.io/calico/kube-controllers:v1.0.0
• quay.io/calico/node:v2.6.2
• quay.io/calico/cni:v1.11.0
• quay.io/coreos/flannel:v0.9.1
• quay.io/coreos/flannel-cni:v0.2.0
• gcr.io/google_containers/cluster-proportional-autoscaler-amd64:1.0.0
• gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.5
• gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.5
• gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.5

[soumyalj]

Tested with rke version v0.0.9-dev.
Pushed the images above to a quay private repository and used them in the cluster.yml file.
Logged into the host with quay.io login.
./rke up --config cluster.yml was executed
When the images are private in the repository, the pull is not successful and rke up fails.

INFO[0002] [reconcile] Reconciling cluster state        
INFO[0002] [reconcile] This is newly generated cluster  
INFO[0002] [certificates] Deploying kubernetes certificates to Cluster nodes 
INFO[0002] [certificates] Checking image [quay.io/soumyalj/rke-cert-deployer-private:latest] on host [159.89.176.30] 
DEBU[0002] Checking if image [quay.io/soumyalj/rke-cert-deployer-private:latest] exists on host [159.89.176.30] 
DEBU[0002] Image [quay.io/soumyalj/rke-cert-deployer-private:latest] does not exist on host [159.89.176.30]: Error: No such image: quay.io/soumyalj/rke-cert-deployer-private:latest 
INFO[0002] [certificates] Pulling image [quay.io/soumyalj/rke-cert-deployer-private:latest] on host [159.89.176.30] 
{"status":"Pulling repository quay.io/soumyalj/rke-cert-deployer-private"}
{"errorDetail":{"code":403,"message":"Error: Status 403 trying to pull repository soumyalj/rke-cert-deployer-private: \"{\\\"error\\\": \\\"Permission Denied\\\"}\""},"error":"Error: Status 403 trying to pull repository soumyalj/rke-cert-deployer-private: \"{\\\"error\\\": \\\"Permission Denied\\\"}\""}
INFO[0003] [certificates] Successfully pulled image [quay.io/soumyalj/rke-cert-deployer-private:latest] on host [159.89.176.30] 
FATA[0003] Failed to create Certificates deployer container on host [159.89.176.30]: Error: No such image: quay.io/soumyalj/rke-cert-deployer-private:latest 

When the images are made public, rke pulls the images on the host and the cluster is created successfully.

[moelsayed]

Previously rke would try to pull an image and it failed if it's a private image. #276 fixes this by adding configuration for private registry authentication.

[stefanvangastel]

Fyi: I can confirm a working rke in an air-gapped on-premise offline environment.

[adalga]

Does it work with insecure registries ? I have an insecure registry and I can pull image from it manually. However,when I define it as a private_registries: in the cluster.yml rke still tries to pull from docker.io and gives an error like
Can't pull Docker image [alpine:latest] for host [192.168.112.146]: Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

[galal-hussein]

RKE Version: master build Feb 9

I was able to verify airgap using private docker registry and the following cluster.yml file:

private_registries:
  - url: rke-registry.rancher:5000
    user: testuser
    password: testpassword
network:
  plugin: canal
nodes:
- address: 10.0.1.142
  user: ubuntu
  role: [controlplane,worker,etcd]
- address: 10.0.1.151
  user: ubuntu
  role: [controlplane,worker,etcd]

system_images:
  etcd: rke-registry.rancher:5000/rancher/etcd:v3.0.17
  kubernetes: rke-registry.rancher:5000/rancher/k8s:v1.8.7-rancher1-1
  alpine: rke-registry.rancher:5000/alpine:latest
  nginx_proxy: rke-registry.rancher:5000/rancher/rke-nginx-proxy:v0.1.1
  cert_downloader: rke-registry.rancher:5000/rancher/rke-cert-deployer:v0.1.1
  kubernetes_services_sidecar: rke-registry.rancher:5000/rancher/rke-service-sidekick:v0.1.0
  kubedns: rke-registry.rancher:5000/rancher/k8s-dns-kube-dns-amd64:1.14.5
  dnsmasq: rke-registry.rancher:5000/rancher/k8s-dns-dnsmasq-nanny-amd64:1.14.5
  kubedns_sidecar: rke-registry.rancher:5000/rancher/k8s-dns-sidecar-amd64:1.14.5
  kubedns_autoscaler: rke-registry.rancher:5000/rancher/cluster-proportional-autoscaler-amd64:1.0.0
  canal_node: rke-registry.rancher:5000/rancher/calico-node:v2.6.2
  canal_cni: rke-registry.rancher:5000/rancher/calico-cni:v1.11.0
  canal_flannel: rke-registry.rancher:5000/rancher/coreos-flannel:v0.9.1

I verified that nodes are airgapped and has no access to the internet, and i was able to pull from this private registry with authentication without docker login on the hosts