Crossplane: Controller manager should use leader election

[jbw976]

The controller manager that hosts all the controllers should be managed by a Deployment for resiliency and life-cycle management. For high availability, at least 2 replicas should be running and they should use leader election to determine which is active/authoritative.

The etcd-operator has an example of leader election: https://github.com/coreos/etcd-operator/blob/master/cmd/operator/main.go#L121

[bassam]

look like the control-runtime has built in support for this.

[negz]

We're going to need this for Crossplane itself and each addon/provider/etc controller.

[negz]

Updated the title of this issue; the controller managers we run all use single pod deployments now. We'd be in a bad place if two replicas were active at once though, and thus still would like leader election.

[muvaf]

I remember controller-runtime had this problem of two replicas reconciling the same resource during updates, i.e. when you update the image in Deployment there is no guarantee that different versions will not be up at the same time, which is much worse than having two pods with same image reconciling the same resource.

Then they started to use StatefulSet, which does have such a guarantee, for controllers but it was really awkward to use StatefulSet when the thing is not stateful at all; CRs are the state. (I remember criticizing a team at my old company who deployed the controllers as StatefulSet and was surprised that it was controller-runtime suggestion at the time).

Then they introduced the leader election and switched back to Deployment, now it basically operates on a ConfigMap to reach the consensus about which pod should reconcile. It increases the boot time for the controller but does guarantee that only one pod will reconcile a resource. It's fairly easy to enable but require RBAC expansion to operate on ConfigMap in the namespace.

[khos2ow]

As of now they are actually in the middle of moving away from ConfigMap (this commit) which reads:

This commit changes the leaderlock to ConfigMapsLeasesResourceLock,
which is intended to transition ppl from the ConfigMap to the more
modern Leases. It is breaking because additional RBAC is required.

This is currently available in controller-runtime in v0.7.0-alpha.3. I haven't checked for the release cycle of controller-runtime or crossplane but I think it's best to coincide them together that v0.14.0 (which would be the first version of crossplane that introduces leader election) will have the v0.7.0 of controller-runtime to prevent any extra migration needed from the end-user to migrate out of ConfigMap.

[negz]

Thanks for looking into this @khos2ow! I agree that it would be best to go directly to Leases, rather than having to migrate from ConfigMaps to Leases.

[negz]

I've now merged all the PRs pertaining to the providers the core Crossplane team maintains, so I'm going to mark this done. I'll leave the other PRs for the maintainers of those repos to merge. Thanks so much for finally getting this done @khos2ow!