Lokomotive: Direct upgrade from 0.5.0 to latest master for MetalLB is broken

[surajssd]

Description

For a Packet cluster that is running metallb and contour, should run without downtime when updating to latest master.

Impact

If a cluster created with 0.5.0 is running metallb and contour. And if you build and update the cluster using lokoctl cluster apply, the metallb stops working.

Environment and steps to reproduce

Follow this guide to deploy metallb and contour with lokoctl 0.5.0.

Now build the latest lokoctl and run the following command to update the cluster:

lokoctl cluster apply

Expected behavior

A set of steps to upgrade cluster to latest master.

Additional Information

• Since the #1155 was merged this problem has started.
• The PR removed terraform resource packet_bgp_session, you can see following diff when doing lokoctl cluster apply:

  # module.worker-general.packet_bgp_session.bgp[0] will be destroyed                                    
  - resource "packet_bgp_session" "bgp" {                                                                
      - address_family = "ipv4" -> null             
      - default_route  = false -> null                                                                   
      - device_id      = "bc84f533-78de-41a7-b142-19cfb64942a4" -> null                                  
      - id             = "09d7054c-0f79-490e-bd7e-61f748634291" -> null                                  
      - status         = "down" -> null                                                                  
    }

Which disables BGP session on the nodes.

[invidian]

@surajssd please add more details on how MetalLB stops working.

[surajssd]

@invidian done.

[surajssd]

• Install the CCM

export PACKET_API_KEY=
export PACKET_PROJECT_ID=

echo "
apiKey: $PACKET_API_KEY
projectID: $PACKET_PROJECT_ID
" > /tmp/ccm-values.yaml

cd assets/charts/control-plane/packet-ccm/
helm install packet-ccm --namespace kube-system --values=/tmp/ccm-values.yaml . 

• Label the nodes

kubectl label node <nodename> lokomotive.alpha.kinvolk.io/bgp-enabled=true

• Add the flag to kubelet

          --cloud-provider=external \

• Remove terraform object

cd $ASSETS_DIR/terraform
terraform state rm 'module.worker-general.packet_bgp_session.bgp[0]'

• Now if you want to add new nodes, first update the metallb component.

[invidian]

kubectl label node lokomotive.alpha.kinvolk.io/bgp-enabled=true

This will break when node reboots. One must modify the file on each host to change that.

[surajssd]

I am using a script for this now, as mentioned in the release notes: https://github.com/kinvolk/lokomotive/pull/1292/files#diff-f88970a1b4f490aaf3a1c85f075da9f957fc47d421cbca843c5e329bcd042e3cR1