Lokomotive: Missing dependency between DNS entries being created and running bootkube?

[rata]

On Packet (it might apply to other platforms) bootkube depends on this to start: https://github.com/kinvolk/lokomotive/blob/878b4828a20e5254cedbd6d04d6eec85be0d4901/assets/lokomotive-kubernetes/packet/flatcar-linux/kubernetes/ssh.tf#L91-L94

The DNS entries are created outside of the terraform module here: https://github.com/kinvolk/lokomotive/blob/878b4828a20e5254cedbd6d04d6eec85be0d4901/pkg/platform/packet/template.go#L190-L200

While it is likely that the DNS entries are created before bootkube starts, as there is no dependency between them, I think it _might_ happen that terraform starts bootkube before DNS entries are created. This might happen, for example, as terraform batches resource creation and DNS entries are delayed until other resources (let's say starting bootkube) is created.

I'm unsure of the effect of bootkube being started before DNS entries are created OR updated (if you are running a test cluster with the same name, DNS entries might just need to be updated). In the best case, this is not an issue (although I doubt bootkube start will finish correctly without being able to form the etcd cluster and, thus, starting the API server). But this might as well be an issue.

I'd like to check if this is an issue at all, first. But if it is, here are some ideas:

• One hacky way to solve this might be for the controller code to take one "dummy" input and uses it in the "bootkube start", just to force a dependency there. Then, this input can be the output of the dns entries created
• Another hacky way to fix this might be to run terraform apply -target for the dns entries first? This will create the controllers, but hopefully nothing else (i.e. not the "bootkube-start" null resource)
• Not sure of other options? Rethink DNS creation?

[rata]

I think option number two (terrafrom apply -target to first create dns record before running bootkube) is used here ONLY if the provider is manual. If the provider is not manual, everything is applied in one go: https://github.com/kinvolk/lokomotive/blob/878b4828a20e5254cedbd6d04d6eec85be0d4901/pkg/platform/packet/packet.go#L151-L154

Therefore, if the provider is not manual, this race can exist.

If the provider is manual, the dns resources are created first: https://github.com/kinvolk/lokomotive/blob/878b4828a20e5254cedbd6d04d6eec85be0d4901/pkg/platform/packet/packet.go#L158-L167

Then, if all went fine, applies the rest: https://github.com/kinvolk/lokomotive/blob/878b4828a20e5254cedbd6d04d6eec85be0d4901/pkg/platform/packet/packet.go#L175-L176

I'd like to confirm my understanding is correct with someone. @invidian ? @mauriciovasquezbernal ?

[rata]

Actually, no, it affects both cases as the null resource is only created first. Sorry for the confusion (it's late, so maybe this message is wrong too :-P)

[invidian]

@rata I cannot find the GitHub reference to the discussion, but we discussed that with @mauriciovasquezbernal and we came to the conclusion, that bootkube start is resilient to the DNS propagation, as it will wait until the DNS records are available. And bootkube times out after 30 minutes, so if DNS records are not created by Terraform within 30 minutes, that's means something is wrong anyway. So I would say we are aware of that, but from extensive testing, that never caused some trouble.

I think with lokomotive-kubernetes merged into lokomotive repository, we could re-think the solution and make better use of apply -target to ensure proper sequence, even if we use Route53 DNS, yes.

BTW, did you hit some issue because of this setup?

Other thing is, I'd say it's rather low priority at the moment.

[mauriciovasquezbernal]

Hi @rata, thanks for pointing this out. As confirmed by @invidian, we were aware of this at the moment of the implementation: https://github.com/kinvolk/lokomotive-kubernetes/pull/137#discussion_r368866577.

We did some tests and found that it should not be a problem for bootkube as it waits some time for the entries to be ready (I did a manual experiment delaying the creation of DNS entries for 5 minutes and it was fine). I think it could be one of the things to improve when implementing the custom DNS support for all platforms.

[rata]

@invidian

@rata I cannot find the GitHub reference to the discussion, but we discussed that with @mauriciovasquezbernal and we came to the conclusion, that bootkube start is resilient to the DNS propagation, as it will wait until the DNS records are available. And bootkube times out after 30 minutes, so if DNS records are not created by Terraform within 30 minutes, that's means something is wrong anyway. So I would say we are aware of that, but from extensive testing, that never caused some trouble.

Ohh, thanks for the context!

BTW, did you hit some issue because of this setup?

No, just need to change code that was doing the DNS manual implementation and thought: "hmm, this is broken or working by chance..." and opened the issue :-)

Other thing is, I'd say it's rather low priority at the moment.

Yeah, sure. Just want to keep track of the bugs we found. Won't solve it now unless is easier for what I need to do to solve :)

@mauriciovasquezbernal

We did some tests and found that it should not be a problem for bootkube as it waits some time for the entries to be ready (I did a manual experiment delaying the creation of DNS entries for 5 minutes and it was fine). I think it could be one of the things to improve when implementing the custom DNS support for all platforms.

Ohh, I see. Thanks a lot for the context and the links! Makes sense