Cloud-on-k8s: Support multi-namespace watches

[sebgl]

Configurable operator meta issue and design proposal.

In order to restrict RBAC permissions required by the operator watching resources in multiple namespaces, we need to support multi-namespaces watch. So far, the controller-runtime only supports watching resources in one namespace, or all of them.

There is already an issue open for it, as a follow-up for the one-namespace restriction: kubernetes-sigs/controller-runtime#218
Looks like it's long-termed planned 👍

operator-sdk folks seem to want that feature as well, and might contribute to the controller-runtime: operator-framework/operator-sdk#767

Meanwhile, the issue above suggests an interesting workaround: implement our own Manager that embeds the controller-runtime Manager, but override the cache to support something like prometheus-operator MultiListWatcher.

My take on it would be to:

1. Try implementing the multi-namespaces watches in the controller-runtime itself and create a PR upstream.
2. If 1. turns out not to work that well, use our own cache implementation (the workaround described above).

[sebgl]

Just noticed there is an in-flight PR for this in the controller-runtime: https://github.com/kubernetes-sigs/controller-runtime/pull/267.
Implementation seems completely ok for us to use 👍
Let's hope it gets merged soon.

[sebgl]

https://github.com/kubernetes-sigs/controller-runtime/pull/267 merged 🎉.
We should be able to test it if we use the controller-runtime latest master (or wait for the next release).

[pebrc]

With the upgrade to kubebuilder v2/controller-runtime 0.2 in #1723 completed we still need to enable the multi-namespace cache if a user chooses to restrict ECK to more than one but not all namespaces.

https://github.com/elastic/cloud-on-k8s/blob/3ea08d293a75257cfdeb5993c8f1d3f385476cbf/cmd/manager/main.go#L214-L218

needs to get a custom CacheBuilder https://github.com/kubernetes-sigs/controller-runtime/blob/59b131b7cd54d56ec74a66b084a53dd1c3e4843f/pkg/cache/multi_namespace_cache.go#L40

something like

manager.Options{
        NewCache: cache.MultiNamespacedCacheBuilder([]string{"namespace1", "namespace2"}),
    }

Unfortunately we chose to make the the namespace parameter singular, so either we introduce a breaking change between ECK versions and pluralize the parameter or add a new parameter.

https://github.com/elastic/cloud-on-k8s/blob/3ea08d293a75257cfdeb5993c8f1d3f385476cbf/cmd/manager/main.go#L84-L87