Kubespray: Docker installation failing on Centos7

[ServerNinja]

Issue:
Kubespray seems to be configuring a bad repo in /etc/yum.repos.d/docker.repo

Environment:

• Tried on kubespray Vagrant in repo using $os = "centos-bento"
• Also tried on three CentOS VMs running in VirtualBox

OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"):

Linux 3.10.0-1062.9.1.el7.x86_64 x86_64
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Version of Ansible (ansible --version):

$ ansible --version
ansible 2.7.12
  config file = /Users/REDACTED/src/third-party/kubespray/ansible.cfg
  configured module search path = ['/Users/REDACTED/src/third-party/kubespray/library']
  ansible python module location = /Users/REDACTED/.pyenv/versions/3.6.4/lib/python3.6/site-packages/ansible
  executable location = /Users/REDACTED/.pyenv/versions/3.6.4/bin/ansible
  python version = 3.6.4 (default, Jan 14 2020, 15:06:47) [GCC 4.2.1 Compatible Apple LLVM 11.0.0 (clang-1100.0.33.8)]

Kubespray version (commit) (git rev-parse --short HEAD):

370a0635

Network plugin used:
default

Copy of your inventory file:

[all]
master1 ansible_host=192.168.1.61 etcd_member_name=etcd1
node1 ansible_host=192.168.1.62 etcd_member_name=etcd2
node2 ansible_host=192.168.1.63 etcd_member_name=etcd3

[kube-master]
master1

[etcd]
master1
node1
node2

[kube-node]
node1
node2

[calico-rr]

[k8s-cluster:children]
kube-master
kube-node
calico-rr

Command used to invoke ansible:

ansible-playbook -i myhosts.ini -b -v -k -u centos cluster.yml

Output of ansible run:

TASK [container-engine/docker : check if container-selinux is available] *******
Tuesday 14 January 2020  16:07:17 -0500 (0:00:00.950)       0:01:48.172 *******
fatal: [k8s-1]: FAILED! => {"changed": false, "msg": "Error from repoquery: ['/bin/repoquery', '--show-duplicates', '--plugins', '--quiet', '--disablerepo', '', '--enablerepo', '', '--qf', '%{name}|%{epoch}|%{version}|%{release}|%{arch}|%{repoid}', 'container-selinux']: Could not match packages: failure: repodata/repomd.xml from docker-engine: [Errno 256] No more mirrors to try.\nhttps://yum.dockerproject.org/repo/main/centos/7/repodata/repomd.xml: [Errno 14] HTTPS Error 403 - Forbidden\n"}
fatal: [k8s-3]: FAILED! => {"changed": false, "msg": "Error from repoquery: ['/bin/repoquery', '--show-duplicates', '--plugins', '--quiet', '--disablerepo', '', '--enablerepo', '', '--qf', '%{name}|%{epoch}|%{version}|%{release}|%{arch}|%{repoid}', 'container-selinux']: Could not match packages: failure: repodata/repomd.xml from docker-engine: [Errno 256] No more mirrors to try.\nhttps://yum.dockerproject.org/repo/main/centos/7/repodata/repomd.xml: [Errno 14] HTTPS Error 403 - Forbidden\n"}
fatal: [k8s-2]: FAILED! => {"changed": false, "msg": "Error from repoquery: ['/bin/repoquery', '--show-duplicates', '--plugins', '--quiet', '--disablerepo', '', '--enablerepo', '', '--qf', '%{name}|%{epoch}|%{version}|%{release}|%{arch}|%{repoid}', 'container-selinux']: Could not match packages: failure: repodata/repomd.xml from docker-engine: [Errno 256] No more mirrors to try.\nhttps://yum.dockerproject.org/repo/main/centos/7/repodata/repomd.xml: [Errno 14] HTTPS Error 403 - Forbidden\n"}

NO MORE HOSTS LEFT *************************************************************
    to retry, use: --limit @/Users/REDACTED/src/third-party/kubespray/cluster.retry

PLAY RECAP *********************************************************************
k8s-1                      : ok=94   changed=13   unreachable=0    failed=1
k8s-2                      : ok=83   changed=13   unreachable=0    failed=1
k8s-3                      : ok=81   changed=13   unreachable=0    failed=1
localhost                  : ok=1    changed=0    unreachable=0    failed=0

Anything else do we need to know:
The repo that is being added to /etc/yum.repos.d/docker.repo seems to be bad. It even fails with cURL

$ curl 'https://yum.dockerproject.org/repo/main/centos/7'<html>
<head><title>403 Forbidden</title></head>
<body>
<h1>403 Forbidden</h1>
<ul>
<li>Code: AllAccessDisabled</li>
<li>Message: All access to this object has been disabled</li>
<li>RequestId: 83E767B9B2C690D0</li>
<li>HostId: QzvoR3ThjIG9oQpyU2UB/6ypwfRqn6ZWsSFKtMhpL10fTxFF9LnZv4Igdob6/+NXyTDcA+vyxV4=</li>
</ul>
<hr/>
</body>
</html>

[ServerNinja]

So my workaround to get past this error is to remove this block from these two files:

• roles/container-engine/containerd/templates/rh_containerd.repo.j2
• roles/container-engine/docker/templates/rh_docker.repo.j2

[docker-engine]
name=Docker-Engine Repository
baseurl={{ dockerproject_rh_repo_base_url }}
enabled=1
gpgcheck=1
keepcache={{ docker_rpm_keepcache | default('1') }}
gpgkey={{ dockerproject_rh_repo_gpgkey }}
{% if http_proxy is defined %}proxy={{ http_proxy }}{% endif %}
{% if ansible_os_family == "RedHat" and ansible_distribution_major_version|int == 8 %}module_hotfixes=True{% endif %}

[thaJeztah]

The status of the dockerproject.org repository is being looked at (see the discussion on https://github.com/moby/moby/issues/40376), but note that that repository was deprecated in favour of the new repository at download.docker.com and should no longer be used. The dockerproject.org repository only has old (EOL) packages, so should not be used.

The docs how to install current versions of Docker from the download.docker.com repository can be found at https://docs.docker.com/install/linux/docker-ce/ubuntu/ or https://docs.docker.com/install/linux/docker-ce/centos/

[rafal-jan]

My workaround was to override below two parameters:

dockerproject_rh_repo_base_url: https://download.docker.com/linux/centos/7/$basearch/stable
dockerproject_rh_repo_gpgkey: https://download.docker.com/linux/centos/gpg

in the ansible inventory.
I assume that a proper fix to the issue would be to update the parameters in https://github.com/kubernetes-sigs/kubespray/blob/master/roles/container-engine/docker/defaults/main.yml#L42.

[AayushNagpal]

+1