Consul-helm: Does not work with single-node Kubernetes Cluster using kind

[weitzjdevk]

Overview of the Issue

The consul-helm chart fails to be used on single-node Kubernetes Clusters, which are bootstrapped with kind (Kubernetes in Docker), when you want to install the consul-agent (in clientmode) and consul-agent (in servermode) on the same node-instance.

Reason being is:

kind uses the same host-node to install multiple pods, and the consul-helm chart uses the host-node-ip for a stable agent IP using the flag: -disable-host-node-id=false

=> therefore using a client and server on the same node, results in conflicting host-node-ids.

The error one is experiencing is similar to the related issue #791

    2021-01-27T00:12:51.732Z [WARN]  agent.fsm: EnsureRegistration failed: error="failed inserting node: Error while renaming Node ID: "afa5d55b-ea34-cd37-176f-b24447c850b5": Node name gke-shustava-default-pool-bad0f39a-f890 is reserved by node 7df16f21-b493-a066-f0cb-261a81cba524 with name gke-shustava-default-pool-bad0f39a-f890 (10.24.2.32)"

which fixed this behaviour by setting -disable-host-node-id=false (explicitly to false)

Reproduction Steps

Install a client-server Consul-Helm setup using kind on a single-node Kubernetes cluster and try to join the client with the server.

Logs

similar to #791

    2021-01-27T00:12:51.732Z [WARN]  agent.fsm: EnsureRegistration failed: error="failed inserting node: Error while renaming Node ID: "afa5d55b-ea34-cd37-176f-b24447c850b5": Node name gke-shustava-default-pool-bad0f39a-f890 is reserved by node 7df16f21-b493-a066-f0cb-261a81cba524 with name gke-shustava-default-pool-bad0f39a-f890 (10.24.2.32)"

Expected behavior

I should be able to run client and server on the same Kubernetes single-node cluster

Environment details

If not already included, please provide the following:

• consul version: 1.9.3
• consul-k8s version: 0.24
• consul-helm version: 0.30.0
• values.yaml used to deploy the helm chart:

Any other information you can provide about the environment/deployment.

single kind Kubernetes cluster.

Sketched solution to the problem

Make the flag:

-disable-host-node-id=false 

configurable in the Helm chart, with a default-value of false so it stays backwards compatible with the current setup.
Allow the users of the Helm-chart to override this value to true so one can bring up single-node clusters using kind (mainly for consul-Demo purposes)

[lkysow]

Hi, I'm unable to reproduce this:

helm install consul hashicorp/consul --set server.replicas=1

kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
consul-consul-m7bfj      1/1     Running   0          70s
consul-consul-server-0   1/1     Running   0          70s

kubectl get nodes
NAME                 STATUS   ROLES    AGE   VERSION
kind-control-plane   Ready    master   17d   v1.19.1

kubectl exec consul-consul-server-0 -- consul members -detailed
Node                    Address           Status  Tags
consul-consul-server-0  10.244.0.17:8301  alive   acls=0,bootstrap=1,build=1.9.3:f55da930,dc=dc1,ft_fs=1,ft_si=1,id=84215ab9-0f23-04b6-1718-8732bf5b7482,port=8300,raft_vsn=3,role=consul,segment=<all>,vsn=2,vsn_max=3,vsn_min=2,wan_join_port=8302
kind-control-plane      10.244.0.15:8301  alive   acls=0,build=1.9.3:f55da930,dc=dc1,id=b5c07ce0-be48-8bea-66ce-796f9d3db46f,role=node,segment=<default>,vsn=2,vsn_max=3,vsn_min=2

Here you can see the two node IDs are: 84215ab9-0f23-04b6-1718-8732bf5b7482, b5c07ce0-be48-8bea-66ce-796f9d3db46f. And I see no errors in the logs.

In regards to -disable-host-node-id, this flag is only set on the client pod so the server pod will use a different ID.

[ilhaan]

I have a PR submitted here to enable setting -disable-host-node-id via values.yaml

[lkysow]

Thanks @ilhaan however I was unable to reproduce this issue so I'm not yet confident that disable host node id is needed. This flag was added on purpose to stop errors that would occur if a consul client pod was force killed.

@weitzjdevk were able to reproduce this issue? @ilhaan if you can provide a reproduction then that would help us understand the usefulness of the PR.

[ilhaan]

@lkysow I am able to consistently reproduce this when -disable-host-node-id is set to false. I see errors similar to the following on consul clients:

* Failed to join 10.8.8.140: Member 'k8snode-22951' has conflicting node ID '43ba14f8-030c-8369-20c0-553ac9699995' with member 'k8snode-34137'

I tested with client versions 1.6, 1.7, 1.8 and 1.9 (with up-to-date point releases).

My consul server runs outside of k8s and is on Consul v1.6.2.

[lkysow]

@ilhaan where are you running k8s? The node IDs come from https://github.com/shirou/gopsutil/blob/master/v3/host/host_linux.go#L36 which on k8s seems to grab from:

kubectl exec consul-server-0 -- cat /proc/sys/kernel/random/boot_id

Could you check if on your cluster this file is the same on your VMs? I'd just like to make sure we know the underlying cause of this issue.

[ilhaan]

@lkysow Worker nodes in my cluster are LXD Containers launched by OpenNebula.

I have two clusters dev and prod. Workers for these clusters run on the same set of hardware.

dev with 4 consul client pods:

➜  ~ kubectl get pods --no-headers=true | awk '{print $1}' | xargs -I {} kubectl exec {} -- cat /proc/sys/kernel/random/boot_id
082fdb6c-355b-4108-8913-d1ee78f6c327
7191c7c0-db02-4d24-be9a-b4f44041d79b
60c9d2d1-c92b-4053-babc-6f6aa1dcb384
c7d04769-d61d-48e3-9272-6e248f9ecd34

prod with 5 consul client pods:

➜  ~ kgp --no-headers=true | awk '{print $1}' | xargs -I {} kubectl exec {} -- cat /proc/sys/kernel/random/boot_id
082fdb6c-355b-4108-8913-d1ee78f6c327
7191c7c0-db02-4d24-be9a-b4f44041d79b
ded8cfff-8eea-4d45-9ead-d582caacc7e5
9ec5040d-0e7e-48fe-8b96-7bce495f9764
72b2f7e7-7f6d-4f93-855e-e529ebb1f223

Seeing the same node IDs for two separate Consul client pod on two different clusters. Checked in OpenNebula and I see that these workers on the same LXD Hypervisor. These LXD containers seem to be getting /proc/sys/kernel/random/boot_id from the host they run on, hence the same node IDs.

[ndhanushkodi]

So we will want to support this for the use case: if two VMs are on the same underlying machine then they'll get clashing node ids. I've reviewed the PR, and once the changes are complete, we can merge it after the next release is cut.

[ncouse]

I was just about to raise an issue on this, as I have the same issue.

We use KinD for both automated testing and developer testing clusters. Normally these would be at least 3 node clusters to facilitate testing Consul. This setup has been working successfully for some time.

As of v0.30.0, this is broken due to the new Node ID mechanism. All our agents end up getting the same NodeID, as we see errors as reported above. I initially noticed strange behaviour as services were disappearing and reappearing, as different agents synced their catalog.

I see the problem was introduced by https://github.com/hashicorp/consul-helm/commit/694969afa26e907651b914e5ed9dc61af00fb959

I also see that on master the change above is commited, and I tried this with the client.disableHostNodeID and it solved my issue.

As of now, none of our automated tests or developer environments that are KinD based will work if we use 0.30.0 or newer.

[ncouse]

As mentioned above the NodeID is based on /proc/sys/kernel/random/boot_id.
For KinD clusters, each "node" is a container on the same host node. Therefore this ID will be the same for all.

I ran a quick check with:

kubectl -n consul get pods -l component=client -o name | xargs -I {} kubectl -n consul exec  {} -- cat /proc/sys/kernel/random/boot_id

The same ID was returned for all agent pods.

I would imagine that this will be the same issue with other k8s clusters designed to run a single actual host.

When is this fix expected to be released, and what are the side effects of using this property to get around the issue?

[ncouse]

I've created a quick repo to demonstrate installing Consul on a KinD cluster: https://github.com/ncouse/kind-consul
Just run the script, and it will demonstrate the issue.

I ran this on a Ubuntu 20.04 host, but should be the same on any Linux host. Our RHEL CI VMs demonstrated the same issue.

[ncouse]

Will there be a release soon with this change? This is blocking us upgrading past 0.29.

[ndhanushkodi]

It is in the beta release https://github.com/hashicorp/consul-helm/tree/v0.32.0-beta2 and the GA release is slated for approximately the end of June. Sorry for the inconvenience in the meantime!!

[ilhaan]

@ndhanushkodi I don't see disableHostNodeID in the beta release you linked above. I do see that this was removed in #936.

I had successfully merged #901 a few weeks ago to introduce this flag, not sure why this was removed.

[lkysow]

@ilhaan sorry for not communicating this better. We decided to always set -disable-host-node-id=true so that it always works for Kind and other VM clusters. Since the default is true ("in 0.8.5 and later defaults to true") we simply removed the flag.

So there's now no need to set disableHostNodeID: true for your use-case. The chart will just work as is.

[ndhanushkodi]

I had forgotten about that, sorry for the miscommunication @ilhaan, what @lkysow said was right.

[ncouse]

I ran my quick Kind test script (as referenced above) against the 0.32.0-beta2 and it seems to work as expected. Running consul commands on any agent do see the other agents exist also.

[ilhaan]

@lkysow & @ndhanushkodi thanks for the clarification!

[david-yu]

Hi @weitzjdevk I will go ahead and close this issue as it looks like @ncouse and @ilhaan have confirmed that the 0.32.0-beta2 release resolves this issue for them. Please stay tuned for the GA release in late June. Feel free to re-open if you still have issues with 0.32.0-beta2.