Lokomotive: Add `disable_bgp` to controller nodes

[surajssd]

We have above variable only for worker nodes. We should add that to the controllers as well.

If that is already available on controllers, then please document such in https://github.com/kinvolk/lokomotive/blob/master/docs/configuration-reference/platforms/packet.md.

[johananl]

I think we may want to treat this issue in a generic way: rather than saying "enable/disable BGP" on Packet clusters only, we may want to have a platform-independent knob at the worker pool level which designates the pool as a pool of ingress nodes. We could call such a knob lb_enabled for example.
Setting this knob to true would control a platform-specific implementation. See https://github.com/kinvolk/lokomotive/issues/372 for more info.

Regardless the above, I think we should consider making this knob opt in rather than opt out with a default of false. For example, even if we leave the knob as Packet specific, we can name it enable_bgp and set the default to false. The rationale:

• Our experience working with customers shows that "ingress nodes" are typically a specific, designated worker pool within a cluster. This pool is typically small, which means most nodes aren't ingress nodes in the typical use case.
• Enabling BGP and creating cloud LBs (on AWS) by default may introduce unexpected effects for the user (e.g. security issues due to BGP being unexpectedly enabled on nodes running backing resources, unexpected charges due to LB resources being provisioned etc.). Making the knob opt-in makes the fact that we enable load balancing more visible even if we include the knob with a true value in our config samples.

[invidian]

BTW, maybe we should disable the BGP on controller nodes always? What would be the scenario to use BGP on controller nodes?

[johananl]

A scenario would be e.g. routing traffic to a pod running on a controller node via MetalLB. I'm not aware of actual use cases which need this. I think we should disable BGP on controller nodes, i.e. not even expose an option to enable it.