Jaeger-operator: Settings `WATCH_NAMESPACE` to multiple namespaces does not work

[davelosert]

When setting the WATCH_NAMESPACE flag to a comma-separated list as described in the docs, the list actually gets interpreted as one single namespace.

For example setting this:

WATCH_NAMESPACE=namespace1,namespace2

will result in this error:

E0525 08:26:15.626046       1 reflector.go:123] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:96: Failed to list *v1.Deployment: deployments.apps is forbidden: User "system:serviceaccount:jaeger-operator:jaeger-operator" cannot list resource "deployments" in API group "apps" in the namespace "namespace1,namespace2"

This is probably due to a bug in the OperatorSDK (https://github.com/operator-framework/operator-sdk/pull/2522) that got fixed with Version 0.16.0

However, if I interpret it correctly (I am not a Go-Pro, pun intended), this repository still uses version 0.15.1.

Would be great if either the version could be increased or the docs could be updated.
I can't do the former (again: not a go-pro 😉 ) but if you say an upgrade of the operator will not be doable, I'll happily create a PR in the documentation-repository to fix the latter.

[pavolloffay]

@davelosert thanks for reporting.

regarding the docs - is there any workaround or do you want to remove the section that describes how to configure list of namespaces?

In the long run we definitely want to upgrade the operator SDK.

[davelosert]

@pavolloffay : Thanks for the quick answer!

Unfortunately, I do not have a workaround.

For the docs: I'd suggest to change

set the WATCH_NAMESPACE to the comma-separated list of namespaces that the Jaeger Operator should watch

to the singular version:

set the WATCH_NAMESPACE to the namespace that the Jaeger Operator should watch

And maybe also add a warning box below which explains that setting multiple namespaces is currently not possible with a link to this issue.

[jpkrohling]

One of my next tasks is to upgrade the SDK, so, hopefully this will get fixed soon.

[syck40]

Hi, I just hit this issue as well. Is it correct that currently there is no way to do the following?

Without clusterrole binding, I am unable to have 1 clusterwide Operator that will create Jaeger backend in foo namespace and bar namespace as well as inject Jaeger sidecars in those namespaces due to the inability to set WATCH_NAMESPACE = "foo, bar"?

Thanks.

[jpkrohling]

You can leave the WATCH_NAMESPACE env var empty, and it will act as a cluster-wide operator, requiring the cluster role bindings. Listing the namespaces in WATCH_NAMESPACE isn't a widely used pattern, but it should be fixed in the next version of the operator.

[syck40]

Ya, I am trying to avoid cluster role, thank you guys.

[davelosert]

@jpkrohling : If I see it correctly, the SDK was updated recently - so this should be working now, right? (currently can't try it myself)

[jpkrohling]

It should indeed work with master at least. Not sure we released a version with this fix already though.

[lambdanis]

Can confirm that with the latest release (1.19.0) watching multiple namespaces works as expected for me.

[jpkrohling]

Thanks for the confirmation!

[dangtrinhnt]

I'm on v1.20.0 and still get this issue.

[jpkrohling]

@dangtrinhnt could you please show how you are setting it? Perhaps by sharing your CR.

[dangtrinhnt]

I used this operator yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger-operator
  namespace: jaeger
spec:
  replicas: 1
  selector:
    matchLabels:
      name: jaeger-operator
  template:
    metadata:
      labels:
        name: jaeger-operator
    spec:
      serviceAccountName: jaeger-operator
      containers:
      - name: jaeger-operator
        image: jaegertracing/jaeger-operator:1.20.0
        ports:
        - containerPort: 8383
          name: http-metrics
        - containerPort: 8686
          name: cr-metrics
        args: ["start"]
        imagePullPolicy: Always
        env:
        - name: WATCH_NAMESPACE
          value: md-dev,wpo2-dev,col2-dev
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: OPERATOR_NAME
          value: "jaeger-operator"

[jpkrohling]

@dangtrinhnt are you sure it's the same issue? Are you getting the same error message as this issue's description? I just tried it here, and I was able to get it working with your CR. The only thing to notice is that you'll need to use the cluster_role.yaml and cluster_role_binding.yaml instead of role.yaml and role_binding.yaml .

[dangtrinhnt]

Running it with cluster role will work of-course. I want to limit its permission to specific namespaces only to reduce the risk. I thought using cluster role is just a workaround?

[jpkrohling]

Running it with cluster role will work of-course. I want to limit its permission to specific namespaces only to reduce the risk. I thought using cluster role is just a workaround?

Let me rephrase the question then: have you created a role and role binding for the jaeger-operator service account in the namespaces it needs to act on? For instance, change the role_binding.yaml to specify the jaeger-operator service account's namespace, like this:

subjects:
- kind: ServiceAccount
  name: jaeger-operator
  namespace: observability

Then create a role/role binding on each namespace you want to watch:

kubectl create namespace wpo2-dev
kubectl apply -f deploy/role.yaml -n wpo2-dev
kubectl apply -f deploy/role_binding.yaml -n wpo2-dev