Minikube: Additional symbol requirements of docker-machine-driver-kvm2 included in v0.31.0 cause regression from v0.30.0 when running on Ubuntu 16.04

[alex-xnor]

Is this a BUG REPORT or FEATURE REQUEST? (choose one): bug

Please provide the following details:

Environment: Ubuntu 16.04.5 LTS with libvirtd

Minikube version (use minikube version): v0.31.0

• OS (e.g. from /etc/os-release): Ubuntu 16.04.5 LTS
• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): kvm2
• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION): minikube-v0.31.0.iso
• Install tools: curl from GitHub releases into ~/bin and chmod +x; libvirt-bin from APT

What happened: After updating minikube and docker-machine-driver-kvm2 from v0.30.0 to v0.31.0:

$ minikube delete
Deleting local Kubernetes cluster...
Machine deleted.
$ minikube start --vm-driver kvm2
Starting local Kubernetes v1.10.0 cluster...
Starting VM...
E1210 13:12:40.228592   30982 start.go:180] Error starting host: Error creating new host: dial tcp: missing address.

 Retrying.
E1210 13:12:40.228836   30982 start.go:186] Error starting host:  Error creating new host: dial tcp: missing address

When I looked at other issues, people had this problem due to libvirt-bin not being installed, or not being in the libvirtd group. It is installed though, and I am in the group. I also saw suggestions to try running docker-machine-driver-kvm2 directly, which appears to point to the root of the problem:

$ docker-machine-driver-kvm2
docker-machine-driver-kvm2: /usr/lib/x86_64-linux-gnu/libvirt-lxc.so.0: version `LIBVIRT_LXC_2.0.0' not found (required by docker-machine-driver-kvm2)
docker-machine-driver-kvm2: /usr/lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_2.2.0' not found (required by docker-machine-driver-kvm2)
docker-machine-driver-kvm2: /usr/lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_3.0.0' not found (required by docker-machine-driver-kvm2)
docker-machine-driver-kvm2: /usr/lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_1.3.3' not found (required by docker-machine-driver-kvm2)
docker-machine-driver-kvm2: /usr/lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_2.0.0' not found (required by docker-machine-driver-kvm2)

When downgrading to the old binary instead, I get this:

$ docker-machine-driver-kvm2
This is a Docker Machine plugin binary.
Plugin binaries are not intended to be invoked directly.
Please use this plugin through the main 'docker-machine' binary.
(API version: 1)

Hence, I think the new docker-machine-driver-kvm2 binary introduces a regression with Ubuntu 16.04 by depending on new symbols that are not present on Ubuntu 16.04. When I use the new minikube binary with the old docker-machine-driver-kvm2 binary, everything works fine.

What you expected to happen:

$ minikube start --vm-driver kvm2
Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Getting VM IP address...
E1210 13:16:45.375024   31024 start.go:210] Error parsing version semver:  Version string empty
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Stopping extra container runtimes...
Starting cluster components...
Verifying kubelet health ...
Verifying apiserver health ...Kubectl is now configured to use the cluster.
Loading cached images from config file.


Everything looks great. Please enjoy minikube!
$ docker-machine-driver-kvm2
This is a Docker Machine plugin binary.
Plugin binaries are not intended to be invoked directly.
Please use this plugin through the main 'docker-machine' binary.
(API version: 1)

How to reproduce it (as minimally and precisely as possible):

• Install Ubuntu 16.04.
• Install libvirt-bin.
• Add yourself to the libvirtd group.
• Log out and back in again.
• Download into your PATH and chmod +x the minikube and docker-machine-driver-kvm2 binaries from the v0.31.0 release on GitHub.
• Run minikube start --vm-driver kvm2

Output of minikube logs (if applicable):

F1210 13:19:41.436786   31890 logs.go:50] Error getting cluster bootstrapper: getting kubeadm bootstrapper: getting api client: Error loading host from store: Docker machine "minikube" does not exist. Use "docker-machine ls" to list machines. Use "docker-machine create" to add a new one.

Anything else do we need to know: No

[brianonn]

I have the same problem exactly on Linux Mint 18.2 (which is Ubuntu 16.04 based). The libvirt.so.0 symbol errors are displayed when starting minikube with the verbose level at 10 (-v 10 option )

Downgrading to docker-machine-driver-kvm2 from minikube 0.30.0 works fine.

[pc-tradelab]

Hi @brianonn , I have the same problem, can you explain :

Downgrading to docker-machine-driver-kvm2 from minikube 0.30.0 works fine.

You downgrade docker-machine-driver-kvm2 or minikube ?

[brianonn]

@pc-tradelab I downgraded the docker-machine-driver-kvm2 to v0.30.0
from here:

https://storage.googleapis.com/minikube/releases/v0.30.0/docker-machine-driver-kvm2

[ANZHONGGEN]

macox
minikube v 0.31.0 machine-driver: virtrualbox same err
E1212 19:15:18.580331 25180 start.go:210] Error parsing version semver: Version string empty

[alex-xnor]

@ANZHONGGEN I think that is a different problem, looks more like #3447. This issue is specifically about missing symbols in docker-machine-driver-kvm2, which would only occur when using the KVM2 driver, not the VirtualBox driver.

[AugustasV]

Thank you, that worked for me!

[kcyeu]

Downgrade to kvm2 driver to v0.30.0 works for me, thank you @brianonn .

[pfuhrmann]

Based on @brianonn comments this fixed the issue

curl -LO https://storage.googleapis.com/minikube/releases/v0.30.0/docker-machine-driver-kvm2
chmod +x docker-machine-driver-kvm2
sudo mv docker-machine-driver-kvm2 /usr/local/bin/

[tstromberg]

It seems like we're getting to a situation where a single docker-machine-driver-kvm2 binary is not appropriate across distros, especially as libvirt has been moving quickly. This is somewhat tangentially related to #1623

Help wanted!

[alex-xnor]

@tstromberg Are you sure that the old binary does not work on new distros? It may be that a single docker-machine-driver-kvm2 binary is sufficient, as long as it is built in a way that is sufficiently compatible (read: built on a Linux distribution at least as old as the other distributions you want to support).

I am having trouble deciphering the release process (it looks like a mixture of Jenkins, shell scripts, Makefile, and Docker) but it looks like docker-machine-driver-hyperkit is built in Docker, but docker-machine-driver-kvm2 isn't, so the operating system of the Jenkins runner was probably upgraded at some point and the produced binaries changed accordingly. Fixing this issue may be as simple as making docker-machine-driver-kvm2 be built in Docker too. Or, maybe not; I don't understand exactly what's going on in the path from Git repository to GitHub release.

[brianonn]

@alex-xnor @tstromberg FYI, the most recent 0.33.1 minikube also ships with a broken docker-machine-driver-kvm2 and the 0.30.0 binary works on this latest release of minikube too. I'm running it like this now.

@mheese Marcus, is this also something you are going to be fixing with the rename to -libvirt? re #3169

[mheese]

@brianonn no, this issue is unrelated, and probably more related to our build system for the driver (how we compile it) than anything else. Afaik, there have been no changes to the kvm2 driver between the releases, so it is perfectly safe to use the kvm2 driver from the 0.30 release.

This raises another point actually. We don't necessarily need to rebuild the kvm2 driver every time. And is another good reason why we should move the driver to -libvirt so that we can independently release it. @tstromberg what do you think?

[afbjorklund]

Unfortunately any libvirt driver is bound to pick up requirements from the libvirt version used...

This is why the kvm driver was built for half a dozen different distributions (unfortunately) :

https://github.com/dhiltgen/docker-machine-kvm/releases

Hopefully there is a way to link it towards an older ABI ?

[tstromberg]

I believe the Makefile now uses an older Ubuntu version - so can this be closed?

[afbjorklund]

Should be fixed (with next release), we are now using old libvirt 1.3.1 again

...
LIBVIRT_1.2.7
LIBVIRT_1.2.8
LIBVIRT_1.2.9
LIBVIRT_LXC_1.0.2
LIBVIRT_LXC_1.0.4
...

https://packages.ubuntu.com/xenial/libvirt-dev

[alex-xnor]

@afbjorklund Is this in v1.0.0? Your comment was on March 24 and v1.0.0 was released March 27, so I would assume it is, but maybe it missed the merge window or something, since v1.0.0 still seems to be using a version of libvirt that is too new:

$ curl -Lo docker-machine-driver-kvm2 https://github.com/kubernetes/minikube/releases/download/v1.0.0/docker-machine-driver-kvm2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   616    0   616    0     0   2207      0 --:--:-- --:--:-- --:--:--  2215
100 36.6M  100 36.6M    0     0  7817k      0  0:00:04  0:00:04 --:--:-- 9550k
$ chmod +x docker-machine-driver-kvm2 
$ ./docker-machine-driver-kvm2 
./docker-machine-driver-kvm2: /usr/lib/x86_64-linux-gnu/libvirt-lxc.so.0: version `LIBVIRT_LXC_2.0.0' not found (required by ./docker-machine-driver-kvm2)
./docker-machine-driver-kvm2: /usr/lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_2.2.0' not found (required by ./docker-machine-driver-kvm2)
./docker-machine-driver-kvm2: /usr/lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_3.0.0' not found (required by ./docker-machine-driver-kvm2)
./docker-machine-driver-kvm2: /usr/lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_1.3.3' not found (required by ./docker-machine-driver-kvm2)
./docker-machine-driver-kvm2: /usr/lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_2.0.0' not found (required by ./docker-machine-driver-kvm2)
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.6 LTS
Release:    16.04
Codename:   xenial

[afbjorklund]

Unfortunately there were race conditions in make, that cause this to not be fixed in 1.0.0. See #4555