Kubeadm: discuss the meaning of kubeadm init --api-advertise-addresses

[mikedanese]

_From @errordeveloper on October 18, 2016 14:55_

As of #34607, we have the following code:

if len(s.API.AdvertiseAddresses) > 0 {
    command = append(command, fmt.Sprintf("--advertise-address=%s", s.API.AdvertiseAddresses[0]))
}

It turns out that the condition will always be true, which was not what I desired when I wrote #34607, however it may be what we want, let's discuss here.

When I wrote this, I have overlooked the fact that when kubeadm auto-detects the primary IP , it just sets it in s.API.AdvertiseAddresses and doesn't indicate that it's an auto-detected IP.

I may be desirable to leave API server's --advertise-address flag unset in the case when kubeadm auto-detects the IP and only set it when user has explicitly specified --api-advertise-addresses (and document clearly that only first value will be used for advertising internally, i.e. what kubernetes service IP will get pointed at). This way we can leave API server to us it's own logic to set address it should advertise.

Alternatively, we can keep having strong preference here, which may be a good thing. But it will probably conflict with --bind-address, whenever we decide to add support for it.

Initially, --api-advertise-addresses flag was only mean for certificates and completely independent of --bind-address, but that was simply about picking less preferences. We also still need to have the discussion about --bind-address as well, I think the main requirement there is to allow user to have any kind of external load-balancer, DNS or whatever NAT situation they might have...

ATTN: @kubernetes/sig-cluster-lifecycle

_Copied from original issue: kubernetes/kubernetes#35043_

[mikedanese]

_From @Crazee on October 25, 2016 20:54_

What I need is a way to specify the only "public" IP address to be used for communication between nodes. Not something in addition to the "I'm gonna use eth0 because I like it" auto-detection business.

[mikedanese]

_From @errordeveloper on October 26, 2016 15:45_

Point taken, we definitely want to allow this to be done easily. Do you
always have public IP on one of the interfaces on the machine? Many cloud
providers use NAT, or force you to use a load balancer. Also, in an HA
setting, restricting to single IP is undesirable.

On Tue, 25 Oct 2016, 21:54 Guy Riddle, [e-mail] wrote:

What I need is a way to specify the only "public" IP address to be used
for communication between nodes. Not something in addition to the "I'm
gonna use eth0 because I like it" auto-detection business.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/kubernetes/issues/35043#issuecomment-256172645,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPWSxwQnrs_f4erArilmJB29vpsu0iXks5q3myBgaJpZM4KZ5Mz
.

[mikedanese]

_From @luxas on October 26, 2016 15:58_

Eventually we want to be able to use componentconfig to set this via the config file...

[mikedanese]

_From @Crazee on October 27, 2016 15:42_

In my case I have set up an overlay network (on an interface not eth0) connecting the nodes. It's cloud based and does HA by placing a new outbound connection to the cloud (load balancer, etc.) if the current connection fails. I need no open inbound ports for running the infrastructure.

I'd be happy if all the bits of each Kubernetes node used only this address to talk to other Kubernetes nodes.

From: Ilya Dmitrichenko <[email protected][e-mail]>
Reply-To: kubernetes/kubernetes <[email protected][e-mail]>
Date: Wednesday, October 26, 2016 at 8:46 AM
To: kubernetes/kubernetes <[email protected][e-mail]>
Cc: Guy Riddle, Comment <[email protected][e-mail]>
Subject: Re: [kubernetes/kubernetes] discuss the meaning of kubeadm init --api-advertise-addresses (#35043)

Point taken, we definitely want to allow this to be done easily. Do you
always have public IP on one of the interfaces on the machine? Many cloud
providers use NAT, or force you to use a load balancer. Also, in an HA
setting, restricting to single IP is undesirable.

On Tue, 25 Oct 2016, 21:54 Guy Riddle, <[email protected][e-mail]> wrote:

What I need is a way to specify the only "public" IP address to be used
for communication between nodes. Not something in addition to the "I'm
gonna use eth0 because I like it" auto-detection business.

-

[luxas]

Ping @errordeveloper, what should we do about this issue?
Any ideas?

[bronger]

In our setup, all nodes and the master are directly connected through say eth1, and have a connection to the internet through eth0. I must be able to connect to the master using the IP of eth0, but I also want to have all cluster-internal communication through the eth1 interfaces.

I have the impression that --api-advertise-addresses covers both kinds of communication, namely, “node wants to connect to master” as well as “I want to connect to master”. Is this correct?

[etoews]

I discovered this issue with a setup similar to @bronger's but mine was on AWS. I got the same impression that @bronger did from the help text.

--api-advertise-addresses stringSlice The IP addresses to advertise, in case autodetection fails

However, from this code in manifests.go it looks like kube-apiserver is only capable of listening on one address.

However however! From looking at that code now I see the commit https://github.com/kubernetes/kubernetes/commit/5cbefbcbcaaebdba8ad70e818cceb2d1bdfe4010 merged a day ago which clears this up.

@luxas Does https://github.com/kubernetes/kubernetes/commit/5cbefbcbcaaebdba8ad70e818cceb2d1bdfe4010 close this issue?

[luxas]

@luxas Does kubernetes/kubernetes@5cbefbc close this issue?

@everett-toews Yes, I think so. As pointed out, the apiserver supports _advertising_ only one address, but it can _listen_ on multiple. So I would say you should set --apiserver-advertise-address to the eth1 address, which will be the cluster-internal address. The apiserver listens by default on all interfaces, so it should be reachable on eth0 as well in the example above.

Does that solve your issue(s)?

[luxas]

And note that in v1.6, --apiserver-advertise-address is mostly decoupled from the API Server serving cert. What you specify in --apiserver-advertise-address will indeed show up in the cert, but if you want it to be valid for other addrs as well, use --apiserver-cert-extra-sans where you can set both DNS names and IP addresses.

[etoews]

@luxas From your description I'd say so.

I'm working on building the binary from master and testing it out to confirm it. It's my first crack at doing this so it might take a while to get all of the bits sorted out.

I'd say it's okay to close this and I'll open another issue if I run into problems.