Minikube: Node InternalIP not matching Host-only address

[ReDucTor]

Environment:

Minikube version v0.30.0

• OS Windows 10 64-bit
• VM Driver: VirtualBox
• ISO version: v0.30.0

• minikube ip: 192.168.99.100
• minikube ssh ifconfig:

docker0   Link encap:Ethernet  HWaddr 02:42:43:CF:20:AF
          inet addr:172.17.0.1  Bcast:172.17.255.255  Mask:255.255.0.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth0      Link encap:Ethernet  HWaddr 08:00:27:A4:13:0D
          inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fea4:130d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:217809 errors:0 dropped:0 overruns:0 frame:0
          TX packets:14653 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:321672994 (306.7 MiB)  TX bytes:1092520 (1.0 MiB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:84:82:B1
          inet addr:192.168.99.100  Bcast:192.168.99.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe84:82b1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:953 errors:0 dropped:0 overruns:0 frame:0
          TX packets:912 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:63500 (62.0 KiB)  TX bytes:49816 (48.6 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:106 errors:0 dropped:0 overruns:0 frame:0
          TX packets:106 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:7956 (7.7 KiB)  TX bytes:7956 (7.7 KiB)

• kubectl describe node | grep InternalIP: InternalIP: 10.0.2.15

What happened:
Minikube uses the NAT adapter as the Internal IP instead of the Host-only adapter

What you expected to happen:
kubectl describe node | grep InternalIP should show the same IP as minikube ip

How to reproduce it (as minimally and precisely as possible):

minikube start
kubectl describe node | grep InternalIP

Output of minikube logs (if applicable):
Nothing seemed related to this

Anything else do we need to know:

[ReDucTor]

As a workaround the following appears to resolve the issue

minikube start --extra-config kubelet.node-ip=192.168.99.100

[tstromberg]

Do you mind describing the impact of this bug? I'd like to understand it a bit better.

[ReDucTor]

When working with agones it attempts to use ExternalIP and falls back to InternalIP, ideally Minikube would have ExternalIP set instead, but overwriting InternalIP is a workaround

Relevant agones code:
https://github.com/GoogleCloudPlatform/agones/blob/37635982c2699fac76bc5528db7d71797e37b44f/pkg/gameservers/controller.go#L695

[afbjorklund]

Current installation is a bit confused as to where it "lives", if you asks /etc/hosts it's even 127.0.1.1 (!).

We do get the AdvertiseAddress correct, but apparently not the InternalIP of the master node itself...

apiEndpoint:
  advertiseAddress: 192.168.99.100
  bindPort: 8443

https://github.com/kubernetes/minikube/blob/master/pkg/minikube/bootstrapper/kubeadm/kubeadm.go#L460

Adding kubelet.node-ip seems to be the way to go, there is no kubeadm --node-ip option it seems:

https://github.com/kubernetes/kubeadm/issues/203

---

Probably we _should_ refer to this address as the ExternalIP:

https://kubernetes.io/docs/concepts/architecture/nodes/#addresses

• HostName: The hostname as reported by the node’s kernel. Can be overridden via the kubelet --hostname-override parameter.
• ExternalIP: Typically the IP address of the node that is externally routable (available from outside the cluster).
• InternalIP: Typically the IP address of the node that is routable only within the cluster.

The host name is OK, it is the addresses that are "wrong":

$ kubectl describe node | grep Name:
Name:               minikube
$ minikube ssh
                         _             _            
            _         _ ( )           ( )           
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __  
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ uname -n
minikube
$ hostname
minikube
$ hostname -i
127.0.1.1

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[roberthbailey]

This came up again recently with another bug report against the Agones documentation.

[roberthbailey]

In contrast to the initial bug report, I don't think minikube needs to change what it assigns to the internal ip. Instead, it should set the external ip to the value of minikube ip since that is the address that is routable from outside of the minikube cluster.

[roberthbailey]

@tstromberg - what's the likelihood of this getting addressed soon-ish? My plan is to just update the agones docs with a workaround if this isn't fixed before we get to our 1.0 milestone.

[alexlapa]

@tstromberg,

Do you mind describing the impact of this bug? I'd like to understand it a bit better.

My case:

1. running pod in network=host (there are reasons :) )
2. pod listens 192.168.99.100: {{ bind_port }} (minikube ip)
3. add livenessProbe.tcpSocket.port: {{ bind_port }}
4. liveness probes fails since it tries 10.0.2.15 ( node.InternalIP)

minikube start --extra-config kubelet.node-ip=192.168.99.100

Fixes this issue.

[afbjorklund]

The InternalIP is _not_ routable within the cluster either, since each node will have a 10.0.2.15...
And currently each node will have a 127.0.1.1 as well, but we should probably stop using that.

[tstromberg]

At it's current state, this issue risks not being addressed for v1.4.0 - anyone up for taking this issue on?

[afbjorklund]

I think we can move it to v1.5.0 instead, just wanted it to not be forgotten.

[woodcockjosh]

/assign

[afbjorklund]

Thanks for fixing this! Bonus points will be awarded for fixing /etc/hosts

[LoseyourseIf]

I found a problem that when I use minikube start without --vm-driver=hyperkit, the minikube ip dosen't match kubectl describe node | grep InternalIP. After I Add --vm-driver=hyperkit that's fine.
OS macOS Catalina
minikube version v1.15.1