Minikube: Unable to start with --vm-driver=none (systemd error)

[cdrage]

Environment:

Debian 7

Minikube version (use minikube version):

v0.22.0

To replicate:

use minikube start --vm-driver=none

Error output:

-- Logs begin at Mon 2017-09-11 09:20:50 EDT, end at Mon 2017-09-11 13:58:06 EDT. --
Sep 11 13:51:49 wikus systemd[1]: Starting Localkube...
Sep 11 13:51:49 wikus localkube[3367]: proto: duplicate proto type registered: google.protobuf.Any
Sep 11 13:51:49 wikus localkube[3367]: proto: duplicate proto type registered: google.protobuf.Duration
Sep 11 13:51:49 wikus localkube[3367]: proto: duplicate proto type registered: google.protobuf.Timestamp
Sep 11 13:51:49 wikus localkube[3367]: listening for peers on http://localhost:2380
Sep 11 13:51:49 wikus localkube[3367]: listening for client requests on localhost:2379
Sep 11 13:51:49 wikus localkube[3367]: name = default
Sep 11 13:51:49 wikus localkube[3367]: data dir = /var/lib/localkube/etcd
Sep 11 13:51:49 wikus localkube[3367]: member dir = /var/lib/localkube/etcd/member
Sep 11 13:51:49 wikus localkube[3367]: heartbeat = 100ms
Sep 11 13:51:49 wikus localkube[3367]: election = 1000ms
Sep 11 13:51:49 wikus localkube[3367]: snapshot count = 10000
Sep 11 13:51:49 wikus localkube[3367]: advertise client URLs = http://localhost:2379
Sep 11 13:51:49 wikus localkube[3367]: initial advertise peer URLs = http://localhost:2380
Sep 11 13:51:49 wikus localkube[3367]: initial cluster = default=http://localhost:2380
Sep 11 13:51:49 wikus localkube[3367]: starting member 8e9e05c52164694d in cluster cdf818194e3a8c32
Sep 11 13:51:49 wikus localkube[3367]: 8e9e05c52164694d became follower at term 0
Sep 11 13:51:49 wikus localkube[3367]: newRaft 8e9e05c52164694d [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
Sep 11 13:51:49 wikus localkube[3367]: 8e9e05c52164694d became follower at term 1
Sep 11 13:51:49 wikus localkube[3367]: starting server... [version: 3.1.5, cluster version: to_be_decided]
Sep 11 13:51:49 wikus localkube[3367]: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
Sep 11 13:51:49 wikus localkube[3367]: 8e9e05c52164694d is starting a new election at term 1
Sep 11 13:51:49 wikus localkube[3367]: 8e9e05c52164694d became candidate at term 2
Sep 11 13:51:49 wikus localkube[3367]: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 2
Sep 11 13:51:49 wikus localkube[3367]: 8e9e05c52164694d became leader at term 2
Sep 11 13:51:49 wikus localkube[3367]: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 2
Sep 11 13:51:49 wikus localkube[3367]: setting up the initial cluster version to 3.1
Sep 11 13:51:49 wikus localkube[3367]: set the initial cluster version to 3.1
Sep 11 13:51:49 wikus localkube[3367]: enabled capabilities for version 3.1
Sep 11 13:51:49 wikus localkube[3367]: published {Name:default ClientURLs:[http://localhost:2379]} to cluster cdf818194e3a8c32
Sep 11 13:51:49 wikus localkube[3367]: ready to serve client requests
Sep 11 13:51:49 wikus localkube[3367]: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
Sep 11 13:51:49 wikus localkube[3367]: I0911 13:51:49.747782    3367 etcd.go:58] Etcd server is ready
Sep 11 13:51:49 wikus localkube[3367]: localkube host ip address: 192.168.1.73
Sep 11 13:51:49 wikus localkube[3367]: Starting apiserver...
Sep 11 13:51:49 wikus localkube[3367]: Waiting for apiserver to be healthy...
Sep 11 13:51:49 wikus localkube[3367]: I0911 13:51:49.748251    3367 server.go:112] Version: v1.7.5
Sep 11 13:51:49 wikus localkube[3367]: W0911 13:51:49.748535    3367 authentication.go:368] AnonymousAuth is not allowed with the AllowAll authorizer.  Resetting AnonymousAuth to false. You should use a different authorizer
Sep 11 13:51:50 wikus localkube[3367]: fatal error: unexpected signal during runtime execution
Sep 11 13:51:50 wikus localkube[3367]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x63 pc=0x7f25699fb60b]
Sep 11 13:51:50 wikus localkube[3367]: runtime stack:
Sep 11 13:51:50 wikus localkube[3367]: runtime.throw(0x4a97def, 0x2a)
Sep 11 13:51:50 wikus localkube[3367]: /usr/local/go/src/runtime/panic.go:596 +0x95
Sep 11 13:51:50 wikus localkube[3367]: runtime.sigpanic()
Sep 11 13:51:50 wikus localkube[3367]: /usr/local/go/src/runtime/signal_unix.go:274 +0x2db
Sep 11 13:51:50 wikus localkube[3367]: goroutine 229 [syscall, locked to thread]:
Sep 11 13:51:50 wikus localkube[3367]: runtime.cgocall(0x3b5ad90, 0xc420d43dd8, 0x4a9047f)
Sep 11 13:51:50 wikus localkube[3367]: /usr/local/go/src/runtime/cgocall.go:131 +0xe2 fp=0xc420d43d98 sp=0xc420d43d58
Sep 11 13:51:50 wikus localkube[3367]: net._C2func_getaddrinfo(0x7f256c0008c0, 0x0, 0xc420dc9020, 0xc42000e2c0, 0x0, 0x0, 0x0)
Sep 11 13:51:50 wikus localkube[3367]: net/_obj/_cgo_gotypes.go:86 +0x68 fp=0xc420d43dd8 sp=0xc420d43d98
Sep 11 13:51:50 wikus localkube[3367]: net.cgoLookupIPCNAME.func2(0x7f256c0008c0, 0x0, 0xc420dc9020, 0xc42000e2c0, 0x4a286b0, 0x9, 0xc420d43f90)
Sep 11 13:51:50 wikus localkube[3367]: /usr/local/go/src/net/cgo_unix.go:151 +0x171 fp=0xc420d43e38 sp=0xc420d43dd8
Sep 11 13:51:50 wikus localkube[3367]: net.cgoLookupIPCNAME(0x4a286b0, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Sep 11 13:51:50 wikus localkube[3367]: /usr/local/go/src/net/cgo_unix.go:151 +0x1bd fp=0xc420d43f38 sp=0xc420d43e38
Sep 11 13:51:50 wikus localkube[3367]: net.cgoIPLookup(0xc420df4000, 0x4a286b0, 0x9)
Sep 11 13:51:50 wikus localkube[3367]: /usr/local/go/src/net/cgo_unix.go:203 +0x4d fp=0xc420d43fc8 sp=0xc420d43f38
Sep 11 13:51:50 wikus localkube[3367]: runtime.goexit()
Sep 11 13:51:50 wikus localkube[3367]: /usr/local/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420d43fd0 sp=0xc420d43fc8
Sep 11 13:51:50 wikus localkube[3367]: created by net.cgoLookupIP
Sep 11 13:51:50 wikus localkube[3367]: /usr/local/go/src/net/cgo_unix.go:213 +0xb4
Sep 11 13:51:50 wikus localkube[3367]: goroutine 1 [chan receive]:
Sep 11 13:51:50 wikus localkube[3367]: k8s.io/minikube/vendor/k8s.io/apimachinery/pkg/util/wait.WaitFor(0xc42044a880, 0xc42108da90, 0xc42032a540, 0xc420aff990, 0xc420410440)
Sep 11 13:51:50 wikus localkube[3367]: /go/src/k8s.io/minikube/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:274 +0x5d
Sep 11 13:51:50 wikus localkube[3367]: k8s.io/minikube/vendor/k8s.io/apimachinery/pkg/util/wait.PollUntil(0x3b9aca00, 0xc420affa90, 0xc42032a540, 0x27, 0x0)
Sep 11 13:51:50 wikus localkube[3367]: /go/src/k8s.io/minikube/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:254 +0x56
Sep 11 13:51:50 wikus localkube[3367]: k8s.io/minikube/vendor/k8s.io/apimachinery/pkg/util/wait.PollInfinite(0x3b9aca00, 0xc420affa90, 0x0, 0x0)
Sep 11 13:51:50 wikus localkube[3367]: /go/src/k8s.io/minikube/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:228 +0x99
Sep 11 13:51:50 wikus localkube[3367]: k8s.io/minikube/pkg/localkube.Servers.StartAll(0xc420357d80, 0x6, 0x8)
Sep 11 13:51:50 wikus localkube[3367]: /go/src/k8s.io/minikube/pkg/localkube/servers.go:47 +0x249
Sep 11 13:51:50 wikus localkube[3367]: k8s.io/minikube/cmd/localkube/cmd.StartLocalkube()
Sep 11 13:51:50 wikus localkube[3367]: /go/src/k8s.io/minikube/cmd/localkube/cmd/start.go:52 +0x93

[ernstp]

I'm also seeing this on Ubuntu 17.04.

[ernstp]

A better description could be "localkube sigsegv when trying to call getaddrinfo"

[jtlisi]

I am also seeing this issue on Ubuntu 17.04

[cdrage]

ping @r2d4 @luxas

[ernstp]

This is also the same as #1680 , which I think was a valid issue.

[czras]

I have the same issue on Arch Linux.

The following patch solves this problem:

      1 diff --git a/pkg/minikube/bootstrapper/localkube/commands.go b/pkg/minikube/bootstrapper/localkube/commands.go
      2 index 69f81a87..18cf3c7a 100644
      3 --- a/pkg/minikube/bootstrapper/localkube/commands.go
      4 +++ b/pkg/minikube/bootstrapper/localkube/commands.go
      5 @@ -46,6 +46,7 @@ Type=notify
      6  Restart=always
      7  RestartSec=3
      8  
      9 +Environment='GODEBUG=netdns=go'
     10  ExecStart={{.LocalkubeStartCmd}}
     11  
     12  ExecReload=/bin/kill -s HUP $MAINPID

[simonswine]

@czras this solves it for me.

If someone wants a linux-amd64 build: https://github.com/jetstack-experimental/minikube/releases/tag/v0.22.2-netgo

[r2d4]

We tried to enable this by default, but it seems that it needs to be enabled for some platforms and disabled for others.

https://github.com/kubernetes/minikube/pull/1829
https://github.com/kubernetes/minikube/pull/1848

I need to investigate a little more on exactly why this is before we enable it by default

[czras]

It seems to be a glibc issue. I would definitely compare the glibc available on the platforms in question on either side of this issue.

On the proposed solution: as a workaround the additional line Environment='GODEBUG=netdns=go' can be applied directly to the localkube.service on the system followed by a systemctl daemon-reload and systemctl restart localkube.

The downside is that a minikube start will overwrite the service file.

[ernstp]

There is a lot written about the exact same crash in other go programs on the net. Here's an example

"When using glibc, statically linking calls to getpwuid only works if the system has the exact shared libraries available when the program is run as were used when the program was built. If you build your program with go build "-ldflags=-extldflags=-static -v" you should see, along with other debug output, a warning from the C linker. On my system I see this:
/home/iant/go/src/net/cgo_unix.go:57: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking

If there is any discrepancy there--if, for example, you are building on one system and running on a different one--that could be the cause of your problem."

[simonswine]

I think another way might be to compile localkube with an older gcc, which made cgo applications work for me across older and newer glibc versions

[iancward]

For the record, I'm seeing the same issues with Fedora 25 and Docker Engine 1.12.
minikube start --vm-driver=none exits with a zero return code; however, the localkube systemd unit has failed.

[rodcloutier]

Any thoughts on the proposed solution to fix this? Could @czras workaround be implemented as a parameter to avoid having to recompile the binary for each platform?

[Rizbe]

Curious about the same thing, sucks having to recompile :(

[davidnewman]

The solution that @czras provides works for me (Archlinux), but there's actually a slightly better way to do it that survives minikube restarts.

# Place the following in /etc/systemd/system/localkube.service.d/override.conf
[Service]
Environment='GODEBUG=netdns=go'

This will use systemd's builtin override capability and make sure that localkube starts properly.

[dlorenc]

Yeah, we should be able to add an environment variable like that to the systemd unit here:
https://github.com/kubernetes/minikube/blob/master/pkg/minikube/bootstrapper/localkube/commands.go#L41