Kubespray: it's really really really hard run kubespray in china!

[willzhang]

i have change many things ,but i still failed when change the kubelet kubectl kubeadm url, it need check sum, i really have no idea ,so i must give up, it's too hard !

TASK [download : download_file | Download item] ***************************************************************************************************************************************************************
fatal: [node1 -> 192.168.93.60]: FAILED! => {"attempts": 4, "changed": true, "checksum_dest": null, "checksum_src": "28619090913443987880712a3388fa26ce50bac1", "dest": "/tmp/releases/kubeadm-v1.18.2-amd64", "elapsed": 3, "msg": "The checksum for /tmp/releases/kubeadm-v1.18.2-amd64 did not match 290bb6acb12c844f76affbab1ce374903bd97c4f19ac8cd3e6fdb7208d638ac8; it was a60974e9840e006076d204fd4ddcba96213beba10fb89ff01882095546c9684d.", "src": "/root/.ansible/tmp/ansible-moduletmp-1590761713.14-2YfN66/tmpPfQqLR", "url": "https://kube-binary.oss-cn-shenzhen.aliyuncs.com/V1.18.3/kubeadm"}
fatal: [node2 -> 192.168.93.61]: FAILED! => {"attempts": 4, "changed": true, "checksum_dest": null, "checksum_src": "28619090913443987880712a3388fa26ce50bac1", "dest": "/tmp/releases/kubeadm-v1.18.2-amd64", "elapsed": 3, "msg": "The checksum for /tmp/releases/kubeadm-v1.18.2-amd64 did not match 290bb6acb12c844f76affbab1ce374903bd97c4f19ac8cd3e6fdb7208d638ac8; it was a60974e9840e006076d204fd4ddcba96213beba10fb89ff01882095546c9684d.", "src": "/root/.ansible/tmp/ansible-moduletmp-1590761717.34-iDdOsp/tmp1ZEN9B", "url": "https://kube-binary.oss-cn-shenzhen.aliyuncs.com/V1.18.3/kubeadm"}

修改k8s版本及镜像仓库，使用阿里云镜像仓库

# cat inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml
kube_version: v1.18.2
kube_image_repo: "registry.aliyuncs.com/google_containers"

修改docker镜像加速，使用阿里云镜像加速

# cat inventory/mycluster/group_vars/all/docker.yml
docker_registry_mirrors:
   - https://uyah70su.mirror.aliyuncs.com

修改docker-ce yum源，使用阿里云yum源安装docker-ce

$ vim roles/container-engine/docker/defaults/main.yml

# CentOS/RedHat docker-ce repo
docker_rh_repo_base_url: 'https://mirrors.aliyun.com/docker-ce/linux/centos/7/$basearch/stable'
docker_rh_repo_gpgkey: 'https://mirrors.aliyun.com/docker-ce/linux/centos/gpg'

# CentOS/RedHat Extras repo
extras_rh_repo_base_url: "http://mirrors.aliyun.com/centos/$releasever/extras/$basearch/"
extras_rh_repo_gpgkey: "http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7"

修改kube download url

$ vim roles/download/defaults/main.yml
kubelet_download_url: "https://kube-binary.oss-cn-shenzhen.aliyuncs.com/V1.18.3/kubelet"
kubectl_download_url: "https://kube-binary.oss-cn-shenzhen.aliyuncs.com/V1.18.3/kubectl"
kubeadm_download_url: "https://kube-binary.oss-cn-shenzhen.aliyuncs.com/V1.18.3/kubeadm"

[willzhang]

and the offline docs not clearly, i can't get any help from it
https://github.com/kubernetes-sigs/kubespray/blob/master/docs/downloads.md#offline-environment

i want a step by step offline docs.

[floryut]

@EppO Maybe can give you some hint on a full offline installation

[EppO]

you're mixing 2 kubernetes versions so checksums don't match:

https://kube-binary.oss-cn-shenzhen.aliyuncs.com/V1.18.3/kubeadm
/tmp/releases/kubeadm-v1.18.2-amd64

I guess you're setting the version in the kubeadm_download_url. In order to avoid that checksum mismatch, you should set it that way:

kubeadm_download_url: "http://yourhttpfilesserver/path/{{ kube_version }}/kubeadm"

and use the kube_version of your inventory.

[rahulwagh]

@floryut
May be this offline guide can help you to do the kubespray installation
Guide - https://jhooq.com/kubespray-12-steps-for-installing-a-production-ready-kubernetes-cluster/

I pretty much followed all the 12 steps mentioned in the guide, i hope it helps you.

[floryut]

@floryut
May be this offline guide can help you to do the kubespray installation
Guide - https://jhooq.com/kubespray-12-steps-for-installing-a-production-ready-kubernetes-cluster/

I pretty much followed all the 12 steps mentioned in the guide, i hope it helps you.

You mean @willzhang

[EppO]

@floryut
May be this offline guide can help you to do the kubespray installation
Guide - https://jhooq.com/kubespray-12-steps-for-installing-a-production-ready-kubernetes-cluster/

I pretty much followed all the 12 steps mentioned in the guide, i hope it helps you.

This guide is very detailed but it's not intended to be used for an offline installation:

• git clone the kubespray repo on github
• download pip packages from PyPI
• doesn't modify the image repos for all container images used by kubespray
• download kubectl from google storage

For now the instructions for offline instructions are pretty succinct: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/downloads.md#offline-environment
I commented in that issue the recipe I was using: https://github.com/kubernetes-sigs/kubespray/issues/5973#issuecomment-616893417

If you're struggling with these instructions, let us know we'll improve the docs.

[willzhang]

you're mixing 2 kubernetes versions so checksums don't match:

https://kube-binary.oss-cn-shenzhen.aliyuncs.com/V1.18.3/kubeadm
/tmp/releases/kubeadm-v1.18.2-amd64

I guess you're setting the version in the kubeadm_download_url. In order to avoid that checksum mismatch, you should set it that way:

kubeadm_download_url: "http://yourhttpfilesserver/path/{{ kube_version }}/kubeadm"

and use the kube_version of your inventory.

you are right, the version is wrong, i have change it and it download success.

and there have no kubeadm kubelet kubectl binary mirror in china, i must download and upload it to some place like https://kube-binary.oss-cn-shenzhen.aliyuncs.com/V1.18.3/kubeadm, a object storage of aliyun.it's so troublesome

i can install them with yum install , there have yum repo mirror in china ,so can kubespray support yum install kubelet kubeadm kubectl ?

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg \
https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF


#yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

[willzhang]

other problems:
1.docker images
can i change all the registry url to my local registry ,maybe a local harbor, but the problem is what docker images should i save to my local registry first?

$ vim inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml

kube_image_repo: "registry.aliyuncs.com/google_containers"

$ vim roles/download/defaults/main.yml
# gcr and kubernetes image repo define
gcr_image_repo: "gcr.io"
kube_image_repo: "k8s.gcr.io"

# docker image repo define
docker_image_repo: "docker.io"

# quay image repo define
#quay_image_repo: "quay.io"
quay_image_repo: "quay-mirror.qiniu.com"

# alauda.cn image repo (for kube-ovn...)
alauda_image_repo: "index.alauda.cn"

can kubespray support a comand like

ansible-playbook -i inventory/mycluster/hosts.yaml image list
ansible-playbook -i inventory/mycluster/hosts.yaml image pull

long long ago ,in china ,we can use gcr.azk8s.cn as replice of gcr.io and k8s.gcr.io ,But now it's not available. Microsoft has canceled it。 so it's really hard to find a suitable registry mirror.

so tell me what image i need , i can pull them use other way， and save them to my local harbor registry first.

2.github files
download file from github in china is very slow ,just about 50kb/s， so can i download this file to some local directory like /tmp/kubespray/xxxfile before run kubespray?

[root@node1 kubespray]# cat roles/download/defaults/main.yml | grep github.com
# https://github.com/ansible/ansible/issues/32750. Set this variable to False to disable
etcd_download_url: "https://github.com/coreos/etcd/releases/download/{{ etcd_version }}/etcd-{{ etcd_version }}-linux-{{ image_arch }}.tar.gz"
cni_download_url: "https://github.com/containernetworking/plugins/releases/download/{{ cni_version }}/cni-plugins-linux-{{ image_arch }}-{{ cni_version }}.tgz"
calicoctl_download_url: "https://github.com/projectcalico/calicoctl/releases/download/{{ calico_ctl_version }}/calicoctl-linux-{{ image_arch }}"
crictl_download_url: "https://github.com/kubernetes-sigs/cri-tools/releases/download/{{ crictl_version }}/crictl-{{ crictl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"

[EppO]

i can install them with yum install , there have yum repo mirror in china ,so can kubespray support yum install kubelet kubeadm kubectl ?

No currently kubespray doesn't support this mode of installation for kubelet/kubeadm/kubectl

can kubespray support a comand like

ansible-playbook -i inventory/mycluster/hosts.yaml image list
ansible-playbook -i inventory/mycluster/hosts.yaml image pull

long long ago ,in china ,we can use gcr.azk8s.cn as replice of gcr.io and k8s.gcr.io ,But now it's not available. Microsoft has canceled it。 so it's really hard to find a suitable registry mirror.

so tell me what image i need , i can pull them use other way， and save them to my local harbor registry first.

First, the complete list of container images you need depends on your setup (network plugin, addons, ...), so look at https://github.com/kubernetes-sigs/kubespray/blob/master/roles/download/defaults/main.yml and try to determine which one you'll use.
For the kubernetes related images, you can run kubeadm config images list, it will tell about you kube-apiserver, kube-scheduler, kube-proxy, kube-controller-manager, pause and etcd images.
There is no easier way than trial and error: you run kubespray, it fails because you miss an image in your local registry, you add it, you reset your cluster and try again, until all images needed are finally there.

2.github files
download file from github in china is very slow ,just about 50kb/s， so can i download this file to some local directory like /tmp/kubespray/xxxfile before run kubespray?

[root@node1 kubespray]# cat roles/download/defaults/main.yml | grep github.com
# https://github.com/ansible/ansible/issues/32750. Set this variable to False to disable
etcd_download_url: "https://github.com/coreos/etcd/releases/download/{{ etcd_version }}/etcd-{{ etcd_version }}-linux-{{ image_arch }}.tar.gz"
cni_download_url: "https://github.com/containernetworking/plugins/releases/download/{{ cni_version }}/cni-plugins-linux-{{ image_arch }}-{{ cni_version }}.tgz"
calicoctl_download_url: "https://github.com/projectcalico/calicoctl/releases/download/{{ calico_ctl_version }}/calicoctl-linux-{{ image_arch }}"
crictl_download_url: "https://github.com/kubernetes-sigs/cri-tools/releases/download/{{ crictl_version }}/crictl-{{ crictl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"

I would suggest to put them at a similar spot than kubeadm/kubectl/kubelet if that's faster for you

[DancingFFFire]

https://github.com/DancingFFFire/kubespray-offline-china

I write it today。maybe can help you。

[willzhang]

https://github.com/DancingFFFire/kubespray-offline-china

I write it today。maybe can help you。

thanks, But it's still not easy，hope some day kubespray have a solution。

[EppO]

We improved the docs regarding offline environments: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/offline-environment.md, please have a look and let us know if that helps.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.