Okd: After successful deployment, etcd still trying to connect to removed bootstrap node.

[cgruver]

Release: 4.4.0-0.okd-2020-04-10-182115

I noticed the following in the logs of etcd-operator

W0411 21:17:24.716221       1 clientconn.go:1120] grpc: addrConn.createTransport failed to connect to {https://10.11.11.99:2379 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.11.11.99:2379: operation was canceled". Reconnecting...

10.11.11.99 was the bootstrap node.

[cgruver]

Ran a clean install just to verify.

I0412 11:31:35.089440       1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.11.11.60:2379 0  <nil>} {https://10.11.11.61:2379 0  <nil>} {https://10.11.11.62:2379 0  <nil>} {https://10.11.11.49:2379 0  <nil>}]
W0412 11:31:35.117948       1 clientconn.go:1120] grpc: addrConn.createTransport failed to connect to {https://10.11.11.49:2379 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.11.11.49:2379: operation was canceled". Reconnecting...

10.11.11.49 was the bootstrap node.

[cgruver]

This bootstrap node is still listed in the ENV for etcd pods.

There is a config map, etcd-pod which appears to be the source of the ENV vars that still include the bootstrap node IP address.

Additional config maps are being created periodically with have similar data. It'a a process I am unfamiliar with. It currently has etcd-pod, etcd-pod-1, etcd-pod-2, ... etcd-pod-5

[cgruver]

Also seeing the following alert:

The master nodes have 20GB RAM and 8 vCPUs. No resource alerts firing on Nodes.

[vrutkovs]

That's UPI install, right?

@hexfusion @alaypatel07 do we have a guide how to reconfigure the operator to remove bootstrap etcd instance?

[cgruver]

@vrutkovs Correct. UPI Bare Metal install.

[PiotrKlimczak]

Same problem with 4.4.0-0.okd-2020-04-16-074048-beta3 also bare metal install.

[jomeier]

Same here on beta4, vSphere UPI

[thekoma]

Also there!

[hexfusion]

/assign

cc @ironcladlou @retroflexer

[hexfusion]

@hexfusion @alaypatel07 do we have a guide how to reconfigure the operator to remove bootstrap etcd instance?

The operator has a controller to handle this. Transient failure is possible but we should eventually scale down the bootstrap etcd instance from the etcd cluster. If that never happens that would be a bug.

[thekoma]

@hexfusion @alaypatel07 do we have a guide how to reconfigure the operator to remove bootstrap etcd instance?

The operator has a controller to handle this. Transient failure is possible but we should eventually scale down the bootstrap etcd instance from the etcd cluster. If that never happens that would be a bug.

It doesn't scale down after 24hr here.

[hexfusion]

can you post you install-config so I can understand the network topology? Also, full operator logs or must-gather if apiserver is up.

[thekoma]

can you post you install-config so I can understand the network topology? Also, full operator logs or must-gather if apiserver is up.

I'm building with my https://github.com/Koma-Andrea/openshift4_vmware_infra_provisioner it's a 3 master + 3 worker.
The installation is done like baremetal passing via kernel parameters ip netmask hostname and gateway.
The DNS records are triple checked and I've a duplicate PTR for the ip of the master records and etcd records (i cannot change them RN).
I'll provide logs as soon I reinstall (it's a lab I'm working on a new feature for this Pseudo IPI)

[deanpeterson]

@hexfusion @alaypatel07 do we have a guide how to reconfigure the operator to remove bootstrap etcd instance?

The operator has a controller to handle this. Transient failure is possible but we should eventually scale down the bootstrap etcd instance from the etcd cluster. If that never happens that would be a bug.

It's not that the bootstrap etcd instance is still up, it's that the etcd-operator is still trying to connect to the bootstrap instance after it is gone. The etcd-pod.yaml file still retains references to the the bootstrap ip.

[ironcladlou]

Tracking this here: https://bugzilla.redhat.com/show_bug.cgi?id=1832923

[vrutkovs]

Lets use bugzilla for tracking