Microk8s: failed to access from outside VM: Unable to connect to the server: x509: certificate is valid for 127.0.0.1...

[iwanbk]

Please run microk8s.inspect and attach the generated tarball to this issue.

inspection-report-20190405_143043.tar.gz

Hi, as the issue subject said, i failed to access microk8s installed on my VM, the host OS is Mac.

I tried to install it by getting the config using microk8s.config and copy it to the host machine.

and i got this

ibk:~
$ KUBECONFIG=~/microk8s_kubeconfig kubectl get pods
Unable to connect to the server: x509: certificate is valid for 127.0.0.1, 10.152.183.1, 10.0.2.15, not 192.168.33.10

I'm aware about #110, but it is my local network, so it should be safe.

The solution seems to be to regenerate the certificate, but i'm not sure how to do it on microk8s.

[ktsakalozos]

Hi @iwanbk, could you try the microk8s.config -l (note the -l flag) to produce the config. If this is no good for you, could you try the --edge channel (sudo snap install microk8s --classic --edge)? The issue you are reporting is already addressed and is released to edge, we will soon release it to the rest of the channels.

[iwanbk]

microk8s.config -l

didn't help

sudo snap install microk8s --classic --edge

didn't work as well.

FYI, i created my ubuntu 18.04 with vagrant, and it has two eth* interfaces.
I disabled the local one, so the only one left is the "public" interfaces.

And then i installed the --edge channel and it worked!

[iwanbk]

and it worked!

note that i only did small test:

• create pod
• kubectl get pods

[ktsakalozos]

@iwanbk the code that figures out the ip is here: https://github.com/ubuntu/microk8s/blob/master/microk8s-resources/actions/common/utils.sh#L98

Could you share the output of ip -o -4 addr list and netstat -rn? I wonder if we are figuring out the wrong IP.

[karlmutch]

I am seeing something very similar when using VirtualBox on an OSX machine to deploy the Linux host and then install microk8s from there. When using AWS EC2 instances from a release of microk8s from maybe a month ago it was working. Now I have upgraded the microk8s past 1.13/stable I have had the problem occur. In my case using client-go calls to send commands to a running pod causes the issue but it comes and goes at random seemingly without reinstalling.

error sending request: Post https://127.0.0.1:16443/api/v1/namespaces/gw-0-11-0/pods/copy-pod/exec?command=%2Fbin%2Fcp&command=%2Fdev%2Fstdin&command=%2Fdata%2Ftmp.gz&container=alpine&stderr=true&stdin=true&stdout=true: x509: certificate signed by unknown authority

Even if the call is successful it seems that using the streaming support inside the go-client libraries to copy files on to the pod locks up anyway and never returns.

$ ip -4 addr list
174.62.97.25
$ netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.0.2.2        0.0.0.0         UG        0 0          0 eth0
10.0.2.0        0.0.0.0         255.255.255.0   U         0 0          0 eth0
10.1.1.0        0.0.0.0         255.255.255.0   U         0 0          0 cbr0
172.17.0.0      0.0.0.0         255.255.0.0     U         0 0          0 docker0
192.168.58.0    0.0.0.0         255.255.255.0   U         0 0          0 eth1

[karlmutch]

Sorry needed to use the $SNAP version of ip to make the -o option work. Also it looks a bit dodgy to rely on the command shell to do this ?

/sbin/ip -o -4 addr list
1: lo    inet 127.0.0.1/8 scope host lo\       valid_lft forever preferred_lft forever
2: eth0    inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0\       valid_lft forever preferred_lft forever
3: eth1    inet 192.168.58.5/24 brd 192.168.58.255 scope global eth1\       valid_lft forever preferred_lft forever
4: docker0    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0\       valid_lft forever preferred_lft forever
19: cbr0    inet 10.1.1.1/24 scope global cbr0\       valid_lft forever preferred_lft forever

```
$ export $SNAP=
$ DEFAULT_INTERFACE="$($SNAP/bin/netstat -rn | $SNAP/bin/grep '^0.0.0.0' | $SNAP/usr/bin/gawk '{print $NF}' | head -1)"
$ IP_ADDR="$($SNAP/sbin/ip -o -4 addr list "$DEFAULT_INTERFACE" | $SNAP/usr/bin/gawk '{print $4}' | $SNAP/usr/bin/cut -d/ -f1)"
$ echo $IP_ADDR
10.0.2.15

[iwanbk]

@ktsakalozos
This is the result of the command before installing microk8s

root@vagrant:~# ip -o -4 addr list
1: lo    inet 127.0.0.1/8 scope host lo\       valid_lft forever preferred_lft forever
2: eth0    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic eth0\       valid_lft 86301sec preferred_lft 86301sec
3: eth1    inet 192.168.1.11/24 brd 192.168.1.255 scope global dynamic eth1\       valid_lft 7103sec preferred_lft 7103sec
root@vagrant:~# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.0.2.2        0.0.0.0         UG        0 0          0 eth0
0.0.0.0         192.168.1.1     0.0.0.0         UG        0 0          0 eth1
10.0.2.0        0.0.0.0         255.255.255.0   U         0 0          0 eth0
10.0.2.2        0.0.0.0         255.255.255.255 UH        0 0          0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 eth1
192.168.1.1     0.0.0.0         255.255.255.255 UH        0 0          0 eth1

the expected "public" IP is 192.168.1.11, your script gave me 10.0.2.15

[niklasholm]

I also have this issue, I'm running micro8ks inside a VM configured with two interfaces, one NATed and one host-only, to give host->guest access while also giving the VM internet access without exposing it to the world (which bridging would).

It seems to me that the problem is that the code assumes only the default interface is external, and includes only that in the certificate instead of including both external interfaces as it should. Apparently it still binds to both interfaces so why shouldn't it include both addresses in the certificate?

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Unable to connect to the server: x509: certificate is valid for 127.0.0.1, 10.152.183.1, 10.0.2.15, not 192.168.56.101

$ /snap/core/current/bin/ip -o -4 addr list
1: lo    inet 127.0.0.1/8 scope host lo\       valid_lft forever preferred_lft forever
2: enp0s3    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3\       valid_lft 74568sec preferred_lft 74568sec
3: enp0s8    inet 192.168.56.101/24 brd 192.168.56.255 scope global dynamic enp0s8\       valid_lft 761sec preferred_lft 761sec
4: docker0    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0\       valid_lft forever preferred_lft forever
5: cbr0    inet 10.1.1.1/24 scope global cbr0\       valid_lft forever preferred_lft forever

$ /snap/core/current/bin/netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.0.2.2        0.0.0.0         UG        0 0          0 enp0s3
10.0.2.0        0.0.0.0         255.255.255.0   U         0 0          0 enp0s3
10.0.2.2        0.0.0.0         255.255.255.255 UH        0 0          0 enp0s3
10.1.1.0        0.0.0.0         255.255.255.0   U         0 0          0 cbr0
172.17.0.0      0.0.0.0         255.255.0.0     U         0 0          0 docker0
192.168.56.0    0.0.0.0         255.255.255.0   U         0 0          0 enp0s8