Kubespray: Cannot install under Ubuntu 18.04 LTS - apt.dockerproject.org has no canidates for bionic

[juliohm1978]

Is this a BUG REPORT or FEATURE REQUEST?: bug

Environment:

• Baremetal/VMs clusters

• OS:

root@k8s-master01-dev:~# uname -a
Linux k8s-master01-dev 4.15.0-29-generic #31-Ubuntu SMP Tue Jul 17 15:39:52 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

root@k8s-master01-dev:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04 LTS
Release:    18.04
Codename:   bionic

• Version of Ansible:

ansible 2.5.2
  config file = /kubespray/ansible.cfg
  configured module search path = [u'/kubespray/library']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.15rc1 (default, Apr 15 2018, 21:51:34) [GCC 7.3.0]

Kubespray version: 2.5.0

Output of ansible run (relevant snippet):

TASK [docker : ensure docker-engine repository public key is installed] ********
Tuesday 24 July 2018  16:32:22 +0000 (0:00:02.729)       0:01:34.176 ********** 

changed: [k8s-node04-dev] => (item=58118E89F3A912897C070ADBF76221572C52609D)
changed: [k8s-node03-dev] => (item=58118E89F3A912897C070ADBF76221572C52609D)
changed: [k8s-master01-dev] => (item=58118E89F3A912897C070ADBF76221572C52609D)
changed: [k8s-node01-dev] => (item=58118E89F3A912897C070ADBF76221572C52609D)
changed: [k8s-node02-dev] => (item=58118E89F3A912897C070ADBF76221572C52609D)

TASK [docker : ensure docker-engine repository is enabled] *********************
Tuesday 24 July 2018  16:32:24 +0000 (0:00:01.283)       0:01:35.459 ********** 

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: apt.cache.FetchFailedException: E:Failed to fetch https://apt.dockerproject.org/repo/dists/ubuntu-bionic/InRelease  403  Forbidden [IP: 52.84.170.123 443], E:The repository 'https://apt.dockerproject.org/repo ubuntu-bionic InRelease' is not signed.
failed: [k8s-node04-dev] (item=deb https://apt.dockerproject.org/repo ubuntu-bionic main
) => {"changed": false, "item": "deb https://apt.dockerproject.org/repo ubuntu-bionic main\n", "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_KumodZ/ansible_module_apt_repository.py\", line 551, in <module>\n    main()\n  File \"/tmp/ansible_KumodZ/ansible_module_apt_repository.py\", line 543, in main\n    cache.update()\n  File \"/usr/lib/python2.7/dist-packages/apt/cache.py\", line 543, in update\n    raise FetchFailedException(e)\napt.cache.FetchFailedException: E:Failed to fetch https://apt.dockerproject.org/repo/dists/ubuntu-bionic/InRelease  403  Forbidden [IP: 52.84.170.123 443], E:The repository 'https://apt.dockerproject.org/repo ubuntu-bionic InRelease' is not signed.\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 1}

This was earlier reported in #2957 and promptly dismissed as a third party issue, but I think it deserves a bit more thinking.

The docker-engine dependency is apparently included just in case one wishes to install a really-really-really old version of docker, going back to v1.13. That version of Docker has long been left out as recommended for use with Kubernetes.

I understand that some installations might need it, and it's great that Kubespray supports that. But it feels unecessary to include the repository as a required step for installations that do not have that version available. By looking over the Kubespray tasks, as far as I can tell, the only reason for adding apt.dockerproject.org as a repository is to make sure the docker-engine package is not installed before going ahead and installing the newest supported version of Docker.

Whether Docker devs decide to support current or future versions of their legacy is their choice. But I don't see any reason why Kubespray should fail when that legacy package is not even available :)

Best regards!

[gopinatht]

Hello, is there any progress on this issue? I think it is reasonable to expect that Kubespray will work out of the box with Ubuntu 18.04 LTS.

[asdfsx]

I got the same problem。After docker-engine repo installed, I cann't use "apt update" 。

# apt update 
Hit:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu bionic InRelease
Hit:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu bionic-updates InRelease             
Hit:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu bionic-backports InRelease           
Hit:4 https://download.docker.com/linux/ubuntu bionic InRelease                                                     
Err:5 https://apt.dockerproject.org/repo ubuntu-bionic InRelease                                                                     
  403  Forbidden [IP: 13.32.207.123 443]
Get:6 http://security.ubuntu.com/ubuntu bionic-security InRelease [83.2 kB]                                                          
Reading package lists... Done       
E: Failed to fetch https://apt.dockerproject.org/repo/dists/ubuntu-bionic/InRelease  403  Forbidden [IP: 13.32.207.123 443]
E: The repository 'https://apt.dockerproject.org/repo ubuntu-bionic InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

Then I found that the ubuntu-bionic doesn't exists under https://apt.dockerproject.org/repo/dists

Index of /repo/dists/
../
debian-jessie/                              12-Apr-2017 19:11  -
debian-stretch/                             12-Apr-2017 19:11  -
debian-wheezy/                              12-Apr-2017 19:11  -
raspbian-jessie/                            12-Apr-2017 19:11  -
ubuntu-precise/                             11-Jan-2017 21:23  -
ubuntu-trusty/                              12-Apr-2017 19:11  -
ubuntu-utopic/                              01-Dec-2016 19:42  -
ubuntu-vivid/                               01-Dec-2016 19:42  -
ubuntu-wily/                                08-Dec-2016 19:40  -
ubuntu-xenial/                              13-Apr-2017 23:23  -
ubuntu-yakkety/                             12-Apr-2017 19:11  -
ubuntu-zesty/                               27-Apr-2017 19:42  -

I think may be we should stop using docker-engine

[glitchcrab]

Docker on Bionic works fine, but for whatever reason apt.dockerproject.org doesn't contain the packages, whereas download.docker.com does (and the Docker docs state that the latter is correct: https://docs.docker.com/install/linux/docker-ce/ubuntu/#install-using-the-repository).

[juliohm1978]

I'm trying to patch related tasks in a private fork, but even if the legacy docker-engine tasks are skipped, the repo new Docker repo added for Ubuntu 18.04 does not work.

root@k8s-master01-dev:~# apt update
Hit:1 http://tjpr-mirror.tjpr.net/ubuntu bionic InRelease
Hit:2 http://tjpr-mirror.tjpr.net/ubuntu bionic-updates InRelease
Hit:3 http://tjpr-mirror.tjpr.net/ubuntu bionic-backports InRelease
Hit:4 http://tjpr-mirror.tjpr.net/ubuntu bionic-security InRelease
Hit:5 https://download.docker.com/linux/ubuntu bionic InRelease                 
Reading package lists... Done                                                   
Building dependency tree       
Reading state information... Done
24 packages can be upgraded. Run 'apt list --upgradable' to see them.
N: Skipping acquire of configured file 'stable/binary-i386/Packages' as repository 'https://download.docker.com/linux/ubuntu bionic InRelease' doesn't support architecture 'i386'

Funny enough, mine is a 64bit kernel. So, even if you get past apt.dockerproject.org checks, the new repos need to be fixed to include the system architecture (32bits or 64bits).

In short, it appears this will require more than just skipping a few checks. Kubespray for Ubuntu 18.04 is really messed up.

If anything, an update to README or release notes informing users of this incompatibility would be welcome.

[gopinatht]

I managed to get Kubespray working with Ubuntu 18.04 on Vagrant. Here is the Vagrantfile with the complete set of modifications.

Specifically, to solve this issue, I had the set the docker_version, docker_versioned_pkg ansible variables to the proper values but also set dockerproject_repo_info.pkg_repo and dockerproject_repo_info.repos to nothing so the docker-engine check is skipped. Here are the changes:

  docker_version: '18.03',
  docker_versioned_pkg: {
    'latest': 'docker-ce',
    '18.03': 'docker-ce=18.03.1~ce~3-0~ubuntu'
  },
  dockerproject_repo_info: {
    pkg_repo: '',
    repos: []
  }

The other changes in the vagrant file is to use bento/ubuntu-18.04 VM image, define the above variable changes in extra_vars scope, and use the extra_vars like this: ansible.extra_vars = extra_vars.

One caveat with this workaround is that the docker version I use is 18.03, which is beyond what the kubernetes build tests with (currently 1.11.2 to 1.13.1 and 17.03.x). So far, I have not had problems with it but I suspect we might as we test more.

[juliohm1978]

Thank you, @gopinatht! Noting on kubernetes tested versions is an important point here. Makes it obvious that Kubernetes itself is not ready for Ubuntu 18.04, so it makes sense that Kubespray just won't be right now.

I'm not sure how to use extra_vars that way. Is that an external file passed to the command line? Where do I define that?

[gopinatht]

@juliohm1978 Checkout the Vagrantfile link I provided in my previous comment: https://gist.github.com/gopinatht/0cc3fb89314c44288c8b3ff652d9ec7c That shows how to set the variables and use extra_vars. It's a complete file that you can use as is with Kubespray master today. I tested this on my Mac.

About your other comment about readiness, I would not go so far as to say K8S is not ready for Ubuntu 18.04. It's just that we have not found an easy way to deploy Docker 17.03 on Bionic. This is because docker made a choice to only provide 18.03 and 18.06 in its docker-ce repo for bionic currently.

I really do not see why Docker 17.03 will not work properly in Bionic. If anyone has ideas for installing Docker 17.03 easily on bionic, we might be able to submit a simple patch to kubespray to work with it.

[juliohm1978]

Got it. I'm not using Vagrant, so with a couple of adjustments it works.

Saved the contents to a file extra_vars.json:

{
  "docker_version": "18.03",
  "docker_versioned_pkg": {
    "latest": "docker-ce",
    "18.03": "docker-ce=18.06.0~ce~3-0~ubuntu"
  },
  "dockerproject_repo_info": {
    "pkg_repo": "",
    "repos": []
  }
}

and used the command line:

ansible-playbook cluster.yml -b -i inventory.ini -e "@extra_vars.json"

Installation succeeded.

Thanks again!

[gopinatht]

@juliohm1978 No worries. Thanks for sharing your approach!

[tangentspace]

The extra_vars.json workaround can also be used with kubespray cli by adding the --ansible-opts argument:

kubespray deploy --ansible-opts "-e @extra_vars.json"

[gopinatht]

@tangentspace is the Kubespray CLI still an active project? The last PR Merge was on Nov 19, 2017. I would love to use it if it is an active project but it does not look like it and I remember reading somewhere that it will be deprecated at some point.

[juliohm1978]

Just wanted to drop by for a quick note.

We are working on a k8s installation under 18.04 and I have to admit there are issues with the new version of Ubuntu -- namely, we had particular problems with how Kubespray configures k8s for DNS resolution. Its default resolvconf_mode: docker_dns tries to use the host's /etc/resolv.conf which, now has only one entry and is managed by netplan.

nameserver 127.0.0.53

127.0.0.53 ends up in the pod/container's resolv.conf and nothing resolves anywhere. We had to revert back to the classic host_resolvconf mode to make it work under Ubuntu 18.04.

[gopinatht]

@juliohm1978 Thanks for sharing! Did you find an easy way to install docker 17.03 or earlier in Ubuntu 18.04?

[juliohm1978]

@gopinatht

Without an official repo, you might be able to get away with adding 16.04's repos (replace bionic with xenial). Most dependencies should be in place. But that's a real stretchy workaround, even for staging environments.

[ltupin]

Why kubespray is setup to use https://apt.dockerproject.org/repo/dists/ instead of https://download.docker.com/linux/ubuntu/dists/ please ?

I ask the question because https://apt.dockerproject.org/repo/dists/ seems to be not updated since months.

[ant31]

@ltupin could you propose a PR to change this ?