Lokomotive: Cluster creation errors out with "not all nodes became ready within the allowed time"

[iaguis]

I noticed that spinning up clusters fails sometimes with the error

Verify cluster: not all nodes became ready within the allowed time

When I encountered this, waiting a bit more allowed nodes to become ready.

This seems to not happen if I disable TLS Bootstrapping.

[knrt10]

Should we increase the retry counts only if TLS Bootstrapping is enabled.

[invidian]

It would be good to know what's the average joining time and why (and if) it takes longer with TLS bootstrapping enabled.

[knrt10]

We can add a timer and log it from time of cluster apply and cluster being created

[invidian]

#952 simplifies modifying the timeouts.

[surajssd]

@invidian does #952 take care of this? I have been bitten by this several times, consistently, now. :-\

[invidian]

Nope, but it makes it obvious what current timeout is, so it can be set to higher value :)

[surajssd]

ok I am working on fixing this.

[invidian]

ok I am working on fixing this.

I'd rather get #952 merged first, please :)

[surajssd]

@invidian I would like to but #952 does so many things. I am trying to fix specific things, so that we can move forward with the release of 0.4.1.

[surajssd]

See the following data.

One node joined within 44s of controller node.

...
2020-09-15-15-31-57
NAME                            STATUS     ROLES    AGE   VERSION
suraj-lk-cluster-controller-0   NotReady   <none>   43s   v1.18.8

2020-09-15-15-31-58
NAME                                STATUS     ROLES    AGE   VERSION
suraj-lk-cluster-controller-0       NotReady   <none>   44s   v1.18.8
suraj-lk-cluster-openebs-worker-1   NotReady   <none>   0s    v1.18.8

2020-09-15-15-32-00

Both became ready in 1.5 minutes.

...
2020-09-15-15-32-27
NAME                                STATUS     ROLES    AGE   VERSION
suraj-lk-cluster-controller-0       Ready      <none>   73s   v1.18.8
suraj-lk-cluster-openebs-worker-1   NotReady   <none>   29s   v1.18.8

2020-09-15-15-32-28
NAME                                STATUS   ROLES    AGE   VERSION
suraj-lk-cluster-controller-0       Ready    <none>   74s   v1.18.8
suraj-lk-cluster-openebs-worker-1   Ready    <none>   30s   v1.18.8

2020-09-15-15-32-30
...

But the third node took nearly 4minutes more to become ready.

...
2020-09-15-15-37-57
NAME                                STATUS     ROLES    AGE     VERSION
suraj-lk-cluster-controller-0       Ready      <none>   6m42s   v1.18.8
suraj-lk-cluster-openebs-worker-0   NotReady   <none>   4m48s   v1.18.8
suraj-lk-cluster-openebs-worker-1   Ready      <none>   5m58s   v1.18.8

2020-09-15-15-37-58
NAME                                STATUS   ROLES    AGE     VERSION
suraj-lk-cluster-controller-0       Ready    <none>   6m44s   v1.18.8
suraj-lk-cluster-openebs-worker-0   Ready    <none>   4m50s   v1.18.8
suraj-lk-cluster-openebs-worker-1   Ready    <none>   6m      v1.18.8

2020-09-15-15-37-59
...