Kubespray: Kubespray offline (on-premise) installation support

[sirsikar-lalit]

What would you like to be added:

Offline support for Kubespray needs rpm's to be installed and docker images to be downloaded from internet first and present on cluster cut off from internet access. Other than the generalized steps mentioned in https://github.com/kubernetes-sigs/kubespray/blob/master/docs/downloads.md, the document does not guide the user to define exact variables needed for offline installation. Also, steps needed to be performed for offline installation which according to my research on a high level are:

1. Downloading rpm's (OS dependent)
2. Getting docker images (dependent on tags for corresponding docker images)

These steps can be included as a block in one of the config files defined by variables which user needs to populate and the complete setup will run accordingly, after user has rpm and docker images content present in some repository.

Why is this needed:

This is needed to perform an offline on premise cluster installation for one of our clients.

Please let me know what you think about this use-case and how to proceed forward from here. We actively work for Production on-prem cluster installs.

[Miouge1]

Considering that there are many ways to do offline environment depending the requirements and services available in said environment.

Anyway, I'll flag @EppO who showed some interest in offline environments.

[sirsikar-lalit]

Thank you @Miouge1 .

[EppO]

This is my current list I'm using for offline installation in my inventory:

# Registry overrides
gcr_image_repo: "{{ registry_host }}"
docker_image_repo: "{{ registry_host }}"
quay_image_repo: "{{ registry_host }}"

kubeadm_download_url: "{{ files_repo }}/kubernetes/{{ kube_version }}/kubeadm"
kubectl_download_url: "{{ files_repo }}/kubernetes/{{ kube_version }}/kubectl"
kubelet_download_url: "{{ files_repo }}/kubernetes/{{ kube_version }}/kubelet"
# etcd is optional if you use etcd_deployment != host 
etcd_download_url: "{{ files_repo }}/kubernetes/etcd/etcd-{{ etcd_version }}-linux-amd64.tar.gz"
cni_download_url: "{{ files_repo }}/kubernetes/cni/cni-plugins-linux-{{ image_arch }}-{{ cni_version }}.tgz"
crictl_download_url: "{{ files_repo }}/kubernetes/cri-tools/crictl-{{ crictl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"
sonobuoy_url: "{{ files_repo }}/kubernetes/sonobuoy/sonobuoy_{{ sonobuoy_version }}-{{ ansible_system | lower }}-{{ sonobuoy_arch }}.tar.gz"

# CentOS/RedHat docker-ce repo
docker_rh_repo_base_url: "{{ yum_repo }}/docker-ce/7/x86_64/"
docker_rh_repo_gpgkey: "{{ yum_repo }}/repo-key.gpg"

Then in your inventory, you need to define the following variables:

• registry_host
• files_repo
• yum_repo

Bottom line, the requirements for a full offline installation:

• Container Registry that contains all the required image in roles/download/defaults/main.yml (the exhaustive list depends of your setup)
• YUM repository for your container runtime RPM (CRI-O, docker or containerd)
• A file server (http or https) that can serve the different tarballs and binaries
• a python pypi server that will host all python packages required by kubespray and their dependencies (install by pip install -r requirements.txt -i http://yourinternalpypiserver/simple/

I will update the docs accordingly to make it clear.

[sirsikar-lalit]

I propose defining a variable: offline_install: true followed by:

```
gcr_image_repo: "{% if offline_install %}{{ image_repo }}{% else %}gcr.io{% endif %}"
kube_image_repo: "{% if offline_install %}{{ image_repo }}{% else %}k8s.gcr.io{% endif %}"

docker_image_repo: "{% if offline_install %}{{ image_repo }}{% else %}docker.io{% endif %}"

quay_image_repo: "{% if offline_install %}{{ image_repo }}{% else %}quay.io{% endif %}"
```

Similar proposal for yum_repo as well.

Per my research, found out that for pypi server: jinja templating is the major package needed along with may by couple others if any.

Also, binaries need to be introduced to the offline cluster manually, and mostly consists of kubectl, kubelet and kubeadm only.

Since, binaries and pypi packages are numbered, just carrying those in pre defined dirs may be a more feasible option. We can define a dir like binaries/ and say copy from this folder. For pypi we can similarly just install like pip install *.whl from a predefined dir.

Also, all variables for offline install should be defined at one place only, eg., in roles/downloads/defaults/main.yml

Please suggest.

[sirsikar-lalit]

I am new to contributing to open source. I notice that @EppO opened a new membership request. What is supposedly the next step here? Am I supposed to give a pr directly or what am I supposed to do, to be able to contribute to kubespray project. Please guide.

[floryut]

I am new to contributing to open source. I notice that @EppO opened a new membership request. What is supposedly the next step here? Am I supposed to give a pr directly or what am I supposed to do, to be able to contribute to kubespray project. Please guide.

You don't need a membership to contribute :)
If you have some improvement or fix, please open up a PR even if it's a work in progress, as other member/contributor car help you.

[EppO]

I like the idea of this "offline_install" but the main drawback is that we're losing flexibility to point to different container registries, http servers and yum repositories compared to a simple inventory override. Not sure that's bad but I don't know if everybody using air-gap clusters use it the same way.

[olganicegirl]

Hello, I have been testing offline install a few time and agree with the introduction of the var: offline_install: true
It helps with the following issues:

1. Control where images are being pulled from: internet, local repo or cached locally on disk (default behavior in kubespray which needs improvement)
2. Control other actions of re-pulling docker images or not.
   The challenge with offline images is that docker tries to connect to docker image to re-pull, even though it's already loaded on the node, and if image url is still pointing to online resource because image is not re-tagged. This seems to be a docker issue and kubespray has some hacks in the playbooks.

Example:

Failed to pull image "gcr.io/google_containers/kubernetes-dashboard-amd64:v1.10.1": rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Typically, one would download all kubernetes images into , re-tag them and push to a local repo that is then accessed during kubespray installation. We do that for our application images successfully.

In my case, I wanted to use kubespray native "download_only" method with dedicated "download" host as I wanted to have images available locally on masaters/nodes in case my "registry_host" goes kaput or have some level of control.

__ansible-playbook -i ./inventory/mycluster/$my_inventory_file -b --become-user=root cluster.yml -e download_cache_dir="$my_root/kubespray_cache" -e download_keep_remote_cache=true -e download_localhost=true -e download_run_once=true -e download_force_cache=false -vv -e ignore_assert_errors=yes --flush-cache_

I also agree with introduction of these inventory items as I do have them as well.
Then in your inventory, you need to define the following variables:

registry_host
files_repo
yum_repo

I am not using files_repo, but the other two assist with pointing to my yum_repo with rpms and python files, registry_host (can be many) basically fetches the host for these group_vars:

repo_host_ip: "{{ hostvars[groups['registry_host'][0]]['ansible_host'] }}"
docker_insecure_registries: [ "{{ repo_host_ip }}:5000" ]

I will be happy to contribute my verdicts with offline installer.