Jaeger-operator: Injected sidecar references wrong jaeger instance

[alexrashed]

Hi there.

I'm currently stumbling across a strange issue with the Jaeger operator in our cluster:

• Cluster: OKD 4.7
• Operator: Community Jaeger Operator 1.21.3

I do have multiple instances of Jaeger in different namespaces (about 3 or four, all managed by the operator), but only a single instance in my namespace:

$ kubectl get jaeger
NAME     STATUS    VERSION   STRATEGY   STORAGE   AGE
jaeger   Running   1.21.0    allinone   memory    82s

My deployment has the annotation sidecar.jaegertracing.io/inject=true.
Since there is only one instance of Jaeger managed by the operator (which was available prior to the deployment), it should automatically select the jaeger instance called jaeger from the namespace.
However, the operator selects an instance from a different namespace:

$ kubectl describe deployment test-deployment
Name:                   test-deployment
Namespace:              jaeger-sidecar-test
CreationTimestamp:      Thu, 18 Mar 2021 15:27:07 +0100
Labels:                 app.kubernetes.io/instance=test-deployment
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=test-deployment
                        app.kubernetes.io/version=1.0.0
                        helm.sh/chart=test-deployment-1.0.0
                        sidecar.jaegertracing.io/injected=other-jaeger
Annotations:            deployment.kubernetes.io/revision: 2
                        meta.helm.sh/release-name: test-deployment
                        meta.helm.sh/release-namespace: jaeger-sidecar-test
                        prometheus.io/port: 14271
                        prometheus.io/scrape: true
                        sidecar.jaegertracing.io/inject: true
Selector:               app.kubernetes.io/instance=test-deployment,app.kubernetes.io/name=test-deployment
Replicas:               1 desired | 1 updated | 2 total | 0 available | 2 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app.kubernetes.io/instance=test-deployment
                    app.kubernetes.io/name=test-deployment
  ...
  Containers:
   test-deployment:
    ...
   jaeger-agent:
    Image:       jaegertracing/jaeger-agent:1.21.0
    Ports:       5775/UDP, 5778/TCP, 6831/UDP, 6832/UDP, 14271/TCP
    Host Ports:  0/UDP, 0/TCP, 0/UDP, 0/UDP, 0/TCP
    Args:
      --agent.tags=cluster=undefined,container.name=test-deployment,deployment.name=test-deployment-test-deployment,host.ip=${HOST_IP:},pod.name=${POD_NAME:},pod.namespace=jaeger-sidecar-test
      --reporter.grpc.host-port=dns:///other-jaeger-collector-headless.other-deployment.svc:14250
      --reporter.grpc.tls.ca=/etc/pki/ca-trust/source/service-ca/service-ca.crt
      --reporter.grpc.tls.enabled=true
    Environment:
      POD_NAME:   (v1:metadata.name)
      HOST_IP:    (v1:status.hostIP)
    Mounts:
      /etc/pki/ca-trust/extracted/pem from other-jaeger-trusted-ca (ro)
      /etc/pki/ca-trust/source/service-ca from other-jaeger-service-ca (ro)
  Volumes:
   other-jaeger-trusted-ca:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      other-jaeger-trusted-ca
    Optional:  false
   other-jaeger-service-ca:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      other-jaeger-service-ca
    Optional:  false
...

As you can see the jaeger agent sidecar's host-port points to the "other-jaeger" and the volumes reference the ConfigMaps of the other jaeger as well. Therefore the deployment also cannot start, since the volumes cannot be mounted from the other namespaces:

Warning  FailedMount  10m                   kubelet            Unable to attach or mount volumes: unmounted volumes=[other-jaeger-trusted-ca other-jaeger-service-ca], unattached volumes=[test-deployment-test-deployment-token-d7wq7 other-jaeger-trusted-ca other-jaeger-service-ca]: timed out waiting for the condition

If the Jaeger instance which should be used is pinned (instead of using the annotation value true), it works just as expected.

Since this is a quite trivial scenario (only a few instances, using the default injection), I currently doubt that this could be an issue with the operator itself. However, I don't really know what else it could be...

[jpkrohling]

Just to confirm, you are indeed using a different name for each Jaeger instance, right? We currently have a bug/limitation in that Jaeger instances with the same name, but from different namespaces, might get mixed up.

cc @rubenvp8510

[alexrashed]

Yes, all instances in the cluster are named differently and each is in a separate namespace.

[jpkrohling]

Would you be able to produce the debug logs from the operator?

[HaehnleinMar]

Here are the debug logs:

time="2021-03-18T15:12:07Z" level=debug msg="CA: Creating the trustedCABundle configmap" instance=other-jaeger namespace=other-deployment
time="2021-03-18T15:12:07Z" level=debug msg="CA: Creating the service CA configmap" instance=other-jaeger namespace=other-deployment
time="2021-03-18T15:12:07Z" level=debug msg="annotation present on deployment" deployment=jaeger-sidecar-test-application namespace=jaeger-sidecar-test
time="2021-03-18T15:12:07Z" level=info msg="injecting Jaeger Agent sidecar" jaeger=other-jaeger jaeger-namespace=other-deployment name=jaeger-sidecar-test-application namespace=jaeger-sidecar-test
time="2021-03-18T15:12:07Z" level=debug msg="injecting sidecar" deployment=jaeger-sidecar-test-application instance=other-jaeger namespace=other-deployment
time="2021-03-18T15:12:07Z" level=error msg="failed to update deployment with sidecar" error="Operation cannot be fulfilled on deployments.apps \"jaeger-sidecar-test-application\": the object has been modified; please apply your changes to the latest version and try again" jaeger=other-jaeger jaeger-namespace=other-deployment name=jaeger-sidecar-test-application namespace=jaeger-sidecar-test
time="2021-03-18T15:12:08Z" level=debug msg="Reconciling Deployment" name=jaeger-sidecar-test-application namespace=jaeger-sidecar-test
time="2021-03-18T15:12:08Z" level=debug msg="annotation present on deployment" deployment=jaeger-sidecar-test-application namespace=jaeger-sidecar-test
time="2021-03-18T15:12:09Z" level=debug msg="CA: Creating the trustedCABundle configmap" instance=other-jaeger namespace=other-deployment
time="2021-03-18T15:12:09Z" level=debug msg="CA: Creating the service CA configmap" instance=other-jaeger namespace=other-deployment
time="2021-03-18T15:12:09Z" level=debug msg="annotation present on deployment" deployment=jaeger-sidecar-test-application namespace=jaeger-sidecar-test
time="2021-03-18T15:12:09Z" level=info msg="injecting Jaeger Agent sidecar" jaeger=other-jaeger jaeger-namespace=other-deployment name=jaeger-sidecar-test-application namespace=jaeger-sidecar-test
time="2021-03-18T15:12:09Z" level=debug msg="injecting sidecar" deployment=jaeger-sidecar-test-application instance=other-jaeger namespace=other-deployment

[rubenvp8510]

Question: Do you have an annotation in your namespace?
Something like this:

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    sidecar.jaegertracing.io/inject: other-jaeger

In that case, the namespace annotation should have precedence, Is the only way I found to reproduce this behaviour so far.

[alexrashed]

No, the only annotations on the namespace are the annotations OKD creates when creating a namespace for a new project (openshift.io/*):

$ kubectl describe namespace jaeger-sidecar-test
Name:         jaeger-sidecar-test
Labels:       <none>
Annotations:  openshift.io/description: This environment is used for testing Jaeger sidecar injections
              openshift.io/display-name: Jaeger Sidecar Test
              openshift.io/requester: alexrashed
              openshift.io/sa.scc.mcs: ...
              openshift.io/sa.scc.supplemental-groups: ...
              openshift.io/sa.scc.uid-range: ...
Status:       Active

No resource quota.

No LimitRange resource.