Rke: add ability to force set node-ip argument on kubelet

[iljaweis]

RKE version:
latest from master

Docker version: (docker version,docker info preferred)
Containers: 16 Running: 10 Paused: 0 Stopped: 6 Images: 14 Server Version: 17.03.2-ce Storage Driver: overlay2 Backing Filesystem: xfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe init version: 949e6fa Security Options: seccomp Profile: default Kernel Version: 3.10.0-862.9.1.el7.x86_64 Operating System: CentOS Linux 7 (Core) OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 3.701 GiB Name: ilja-worker-03 ID: I7SV:G5IB:QNS3:GV4J:CWWQ:2GGX:MUIY:SOY5:YTWX:XXIF:DB6I:UOC2 Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

Operating system and kernel: (cat /etc/os-release, uname -r preferred)
3.10.0-862.9.1.el7.x86_64

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
vmware

We would like to set the kubelet argument node-ip explicitly to the internal_address. RKE will set node-ip only if internal_address is different from the host address (at https://github.com/rancher/rke/blob/67448c38c672c81d20004d5532fc0db464189d30/cluster/plan.go#L348), but in our case they're the same but we still need to set node-ip.

The reason we need to do that is that we add additional addresses to our main network interface but then we run into this: https://github.com/kubernetes/contrib/issues/2761 / https://github.com/kubernetes/kubernetes/issues/62830 . Setting node-ip works around that, but RKE does not support it.

I'd be willing to work on a patch for RKE but I'm not sure what the best way would be.
Would it be correct to always set CommandArgs["node-ip"] = host.InternalAddress? Or introduce a configuration option? Or perhaps there is another way?

[estevao90]

Hi.

Any update? I have the same problem when deploy a cluster in Vagrant environment...

[finchr]

I found an ugly workaround, add a third interface and make the address and internal_address different.

I'm using test kitchen and found that the node IP was defaulting to the IP of interface eth0 (10.0.2.15) for all nodes regardless of what I put in the address field.

[pasikarkkainen]

I just hit this same issue with rke.

Basicly I need to be able to set the kubelet "--node-ip=address" parameter for each node, to get the generated kubernetes service IP DNAT rules to use the correct interface IP on each node, but currently rke does NOT do that.

Basicly there are multiple network interfaces on the kubernetes nodes, and I've configured cluster.yml to force usage of single internal interface:

nodes:

• address: 10.0.0.11
  port: "22"
  internal_address: "10.0.0.11"
  role:
  
  
  
  • controlplane
  
  
  • worker
  
  
  • etcd
    
    hostname_override: node01
  
  

.. but that means rke won't configure the kubelet node-ip parameter because both the address and internal_address are the same, and kubelet ends up using wrong interface IP for generated DNAT rules.

Maybe add an option "force_kubelet_internal_address" or something like that to rke?

[estevao90]

I just hit this same issue with rke.

Basicly I need to be able to set the kubelet "--node-ip=address" parameter for each node, to get the generated kubernetes service IP DNAT rules to use the correct interface IP on each node, but currently rke does NOT do that.

Basicly there are multiple network interfaces on the kubernetes nodes, and I've configured cluster.yml to force usage of single internal interface:

nodes:

* address: 10.0.0.11
  port: "22"
  internal_address: "10.0.0.11"
  role:

  * controlplane
  * worker
  * etcd
    hostname_override: node01

.. but that means rke won't configure the kubelet node-ip parameter because both the address and internal_address are the same, and kubelet ends up using wrong interface IP for generated DNAT rules.

Maybe add an option "force_kubelet_internal_address" or something like that to rke?

Yes! I think that an additional configuration its a good solution for this problem. Today the workaround is add another network interface for the internal cluster communication.

[pasikarkkainen]

@alena1108 @vincent99 any thoughts about this one?

[pasikarkkainen]

Adding @superseb and @galal-hussein here because they were involved in the original issue(s) where the "internal_address" option/feature was added.

What do you think of this issue, what's the best way to get this fixed?

As described above currently rke "node address" / "internal_address" configuration is not flexible enough because kubelet auto-detects the wrong interface IP in some cases, and rke has the check in place which makes rke skip "internal_address" configuration for kubelet when it actually is needed for correct behaviour.. the proposed "force_kubelet_internal_address" option (or something similar) would solve this issue, allowing us to force rke to pass the needed "internal_address" to kubelet as node IP.

Thanks!

[finchr]

@pasikarkkainen it seems like the entire purpose of internal_address is to override what the kubelet thinks the node-ip should be. Shouldn't the mere existence of internal_address be enough to force a node-ip setting?

[pasikarkkainen]

@finchr well, that works for me, and solves the issue :) how do others feel about that? if "internal_address" is configured, always use it (and pass it to kubelet), even if it's the same as "node address" ?

[pasikarkkainen]

I think we just need to create a PR, proposing this change, eg. if internal_address is given in rke config then always pass it to kubelet, even if it's the same as node_address.

[pasikarkkainen]

So I spent some time looking at the rke code, and it seems in rke cluster/defaults.go there is:

            if len(host.InternalAddress) == 0 {
                    c.Nodes[i].InternalAddress = c.Nodes[i].Address
            }

So if there's no InternalAddress given in cluster.yml, InternalAddress is set to same as the node Address.
This means we can't do only the most obvious change "if InternalAddress is specified, always pass it to kubelet --node-ip", because InternalAddress is always defined in the code, no matter if it's in rke cluster.yml or not.. which would then mean a change of behaviour for many other use cases.

I guess we need a new flag, which tells if InternalAddress was configured in cluster.yml or not, and use that for the logic.. or a new cluster.yml config option.

[pasikarkkainen]

What about introducing new "kubelet_node_ip" cluster.yml config option for nodes? In addition to existing "address" and "internal_address" options. Introducing new config option wouldn't mess up existing logic..

[finchr]

Hi @pasikarkkainen,
Looking at the code, that does seem like the easiest solution. It seems a little redundant as we would need to set the address, internal_address, and kubelet_node_ip to the same IP for vagrant.

[pasikarkkainen]

Before I start coding and creating a PR for this feature, I'd appreciate comments/feedback about the approach.. ping @superseb , @alena1108 , @galal-hussein . Thanks!

[rgl]

ignoring the cluster yaml internal_address property was a sad surprise for me... it should always set the kubelet node-ip. if there are other use cases that break because of this, its because they are not correctly handled and should be fixed with something more explicit.

while we are at this, maybe, as a last resort measure, let us pass extra arguments for the kubelet on a per-node basis (with the same semantics of the global ones)?

[alwinmark]

Ok this Topic is quite old and still not fixed. What workaround did you guys apply?
I think the internal_address was introduced because of this issue: https://github.com/rancher/rke/issues/363

But it still does not work. I would fix it by myself if I wouldn't know that contributions are not welcomed, but I would suggest moving the logic to set the node-ip to the place where the

if len(host.InternalAddress) == 0

check is done, or leave that check completely away

[mkjpryor-stfc]

Will setting node-ip=address in the case where internal_address is not given actually do any harm?

I.e. can we throw away the if host.Address != host.InternalAddress check?

[think01]

We have exactly this problem. Is there any timeline/work in progress on the proposed solution (that would be ideal for us)?

[marvinwankersteen]

I had the same problem and solved it with iptables. But I am not sure if it's really save to use.
On my client machine where I run rke, I used the following iptables rule:
iptables -t nat -A OUTPUT -d 10.8.1.100 -j DNAT --to-destination 10.8.1.10

$ cat cluster.yml
nodes:
- address: 10.8.1.100
  port: "22"
  internal_address: 10.8.1.10

$ kubectl describe node <node>
Roles:              worker
Addresses:
  InternalIP:  10.8.1.10

Now, the internal ip is not any more the external address of the server and it's not necessary to add a second interface or ip address.
The only thing I noticed is, that in the rancher-gui at the workloads the wrong ip is displayed for e.g. 3306/tcp. (10.8.1.100:3306)
Maybe it'll help somebody.

[alwinmark]

This does not work IMO when you are using flannel (canal). It will still pick up default gateway interface. Clusterwide its possible to set the flannel interface with network options but then every node must have that device.

Didn't find a per node setting yet.

[harshalk91]

Any updates on this issue? I am facing issue with openstack where i have multiple network interfaces. Kubelet fails to update pod status.