Lokomotive: Add Packet CCM to Packet Lokomotive clusters

[iaguis]

We should add the Packet CCM to Lokomotive clusters deployed to Packet.

The CCM handles, for example, node changes and is gonna be needed to use the peer autodiscovery feature of MetalLB once it gets merged (we're currently using a patched version of MetalLB to have the feature).

[invidian]

When implemented, it will also hit #478 :disappointed:

[iaguis]

Keep in mind this comment about things needed to bootstrap the CCM, since it'll probably apply to the Packet CCM too.

[knrt10]

Looking into this

[johananl]

https://github.com/packethost/packet-ccm#bgp

The Packet CCM enables BGP for the project and enables it on all nodes as they come up.

@knrt10 FYI ^. We should verify that users can still decide not to enable BGP for a worker pool.

[knrt10]

Current status:

• v1.1.0 for packet-ccm is buggy and it does not contain MetalLb in the release. It is expected to be fixed in a week.

I tried manually deploying ccm with current master. We had to deploy Lokomotive's MetalLb component for ccm to work. ccm watches and picks up the configMap config from metallb-system namespace. The problems that I faced are:

• Once ccm deployment is deployed on cluster, you have to apply MetalLb component. If component is re-applied, ccm does not pick configMap correctly and throws error that configMap is not correct, which should not happen. You have to delete the pod manually to get it to work again.

failed to update and sync nodes: failed to get metallb config map: could not parse config: yaml: unmarshal errors:
  line 1: field peer-autodiscovery not found in type metallb.ConfigFile

• BGP is automatically enabled for all nodes. We have to find a way for users it disable it.
• Once we deploy Lokomotive's MetalLb component, you have to manually apply Elastic IP to worker nodes on Packet console.

These things should be fixed, once packet-ccm fix in their patch release

[invidian]

@knrt10 FYI ^. We should verify that users can still decide not to enable BGP for a worker pool.

Created #1178 to avoid regressing on this.

[invidian]

Also, I think this task will be blocked by https://github.com/packethost/packet-ccm/issues/100.

Next up is a e2e test for MetalLB-compatible labels on the nodes. We should ensure that metallb.lokomotive.io/peer-address and metallb.lokomotive.io/src-address labels are properly set on the nodes with BGP enabled. The source of the label is irrelevant (whether it comes from Ignition or CCM).

[invidian]

From https://github.com/packethost/packet-ccm/issues/100#issuecomment-724439586

Why can you not do it now? The bgp speakers are enabled in the loadbalancers.yaml deployment manifest. Can you not just modify the daemonset, updating its selectors, to control where the speakers are deployed?

@johananl are there some implications of enabling BGP session on all servers and controlling which nodes handle the traffic only by node selector on MetalLB speakers?

[invidian]

@johananl are there some implications of enabling BGP session on all servers and controlling which nodes handle the traffic only by node selector on MetalLB speakers?

One thing I see is, that as a compromised node, I can start BGP client myself and then capture the traffic. But on the other hand, with compromised node, I can label the Node object myself and still capture the traffic if I want.

[invidian]

One thing I see is, that as a compromised node, I can start BGP client myself and then capture the traffic. But on the other hand, with compromised node, I can label the Node object myself and still capture the traffic if I want.

We could use restricted labels like role, to for example mark those nodes with ingress role, but we do not have the ability to do it right now.

[johananl]

@invidian what you said is correct, enabling BGP expands the attack surface of a cluster, especially since we aren't using BGP authentication (although BGP authentication is a fairly weak security mechanism and in any case you should always protect a peer with a network access list, which Equinix does AFAIK).

One thing I see is, that as a compromised node, I can start BGP client myself and then capture the traffic. But on the other hand, with compromised node, I can label the Node object myself and still capture the traffic if I want.

Not necessarily. I can see cases where I have access to a low-privileged pod which can't do damage via the k8s API but can still initiate BGP connections to the ToR switch. A possible mitigation for that is to block outbound network access on TCP 179 when a node isn't supposed to run BGP.

Another small side effect is that with BGP enabled the Equinix ToR switches continually try to establish a BGP session with the nodes, which means some infrequent network spam on TCP 179.

[invidian]

Status update about the PR #1155:

• To address only enabling BGP on nodes with disable_bgp = false, I added lokomotive.alpha.kinvolk.io/bgp-enabled label to nodes, so only nodes with lokomotive.alpha.kinvolk.io/bgp-enabled=true will have BGP enabled by CCM
• Added an e2e test which checks that nodes with lokomotive.alpha.kinvolk.io/bgp-enabled=true label has actually BGP session created on Packet API. This will ensure that we don't break this functionality when migrating from Terraform-managed BGP session to CCM management.
• I implemented https://github.com/packethost/packet-ccm/issues/100 in https://github.com/packethost/packet-ccm/issues/101, figured out how to build Docker images for arm64 and amd64 and those images are now used until BGP Node Selector feature gets merged and released.
• Configured MetalLB component to enable reading BGP peer information from annotations, which CCM will add to the nodes.

[invidian]

Next update:

• Created https://github.com/packethost/packet-ccm/pull/107 to get rid of some warning from CCM logs
• Also cleaned up the PR to make it ready for review and merging.
• Reduced RBAC grants for CCM and reported it upstream: https://github.com/packethost/packet-ccm/issues/109.