Spark-on-k8s-operator: Support for multiple namespaces

[skonto]

There is an implementation of a multi-watcher mentioned here: https://github.com/kubernetes/kubernetes/issues/74415#issuecomment-497570002
We could utilize that. Thoughts?

[skonto]

@liyinan926 what do you think moving to controller-runtime (kubebuilder), it is more uptodate https://github.com/kubernetes-sigs/controller-runtime/blob/master/pkg/cache/multi_namespace_cache.go
It would be a big change though.

[liyinan926]

It would be great to move to using controller-runtime. But it looks like possible to just use that particular package without doing a huge round of refactoring.

[skonto]

@liyinan926 I understands do you have any insights as what the future might be controller-runtime vs the client lib thing. Long term what should we support?

[liyinan926]

Long term controller-runtime is the way to go I think.

[doctapp]

Any way of deploying multiple spark operators on k8s, like one per namespace?

[zzvara]

I deploy multiple Spark Operators using:

apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: spark-operator
  namespace: production
spec:
  chart:
    repository: https://googlecloudplatform.github.io/spark-on-k8s-operator
    name: sparkoperator
    version: 1.0.5
  releaseName: spark-operator
  values:
    webhook:
      enabled: true
    resyncInterval: 5
    metrics:
      enable: false
    controllerThreads: 2
    batchScheduler:
      enable: true

and then I have one HelmRelease for each namespace.

[doctapp]

Try removing any one of them and it will corrupt all installations

[zzvara]

Try removing any one of them and it will corrupt all installations

What do you mean by remove and what do you mean by corrupt installation?

[doctapp]

We constantly bring up k8s namespaces and then tear them down, therefore, we install/uninstall many Spark Operators. If you only do the above, then all Spark Operators will become corrupted and no longer work as expected except the most recent version. You need many hacks to make it work correctly.

[zzvara]

@doctapp when you tear down Spark Operator before deleting the namespace and you install it after the namespace is created, what can go wrong?

[doctapp]

CRDs are defined at cluster level along with the web hook, i.e., they are
not per-namespace...

On Tue, Jan 12, 2021 at 12:46 PM Zoltán Zvara [e-mail]
wrote:

@doctapp https://github.com/doctapp when you tear down Spark Operator
before deleting the namespace and you install it after the namespace is
created, what can go wrong?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/507#issuecomment-758826785,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA3OUNBD6GBI4RSNJ4OSNP3SZSDFXANCNFSM4HSGESOA
.

[zzvara]

Is there any way to overwrite CRD's in Helm so that they are not deleted when the chart is removed?

[doctapp]

No standard way other than hacking

On Wed, Jan 13, 2021 at 11:17 AM Zoltán Zvara [e-mail]
wrote:

Is there any way to overwrite CRD's in Helm so that they are not deleted
when the chart is removed?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/507#issuecomment-759557280,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA3OUNBGKU4TJYUL3FFGVWDSZXBQHANCNFSM4HSGESOA
.