Jaeger-operator: Inject sidecars from different namespaces

[jpkrohling]

It's quite possible that Jaeger instances will live in one namespace and applications in another. We need a way then to inject sidecars into deployments from one namespace pointing to Jaeger instances in another namespace.

Tasks:

• [x] Figure out whether the JaegerSpec should have a list of namespaces the operator should watch
• [x] The RBAC implications: what are the failure scenarios and how to recover from that
• [ ] On OpenShift, one namespace cannot see things from another namespace by default. What should we do here?

[annanay25]

Duplicate of #24.

[gregoryfranklin]

Duplicate of #24.

This is not a duplicate.

24 is about the jaeger being in a different namespace to the operator.

60 is about the agent sidecar being in a different namespace to the jaeger.

[jpkrohling]

@gregoryfranklin and all who expressed interest in this issue: if you have a concrete use case in mind, we'd really benefit from hearing about it.

[gregoryfranklin]

We have Jaeger installed in one namespace and applications using that Jaeger instance deployed in many other namespaces.
We have separate namespaces for different teams and products. The Jaeger setup is maintained by an infrastructure team.
In a service-oriented-architecture setup traces can be distributed over multiple applications (in different namespaces) so they use a common Jaeger. Also, with lots of microservices, each maintained by a different team, having a separate Jaeger for each namespace is a lot of overhead.
We therefore have a single Jaeger instance for each environment used by all the applications / namespaces in that environment.
We have a separate Kubernetes cluster for each environment so each cluster usually has one jaeger-operator and one Jaeger.
We may occasionally have a second Jaeger for testing / migrations / some sort of specific purpose.

At the moment, we inject the Jaeger agent by providing a common helm chart that includes templates for sidecars that our developers can include in their own helm chart. We also provide to our developers, a standard helm charts from common workloads (eg a Java webapp) that include the Jaeger agent.

In other cases I've seen, this has been implemented by a Mutating Webhook Admission Controller acting on Pods.

[jpkrohling]

This is great feedback, @gregoryfranklin. Thanks! Have you considered adding yourselves to https://github.com/jaegertracing/jaeger/blob/master/ADOPTERS.md ?

[tillig]

Sort of similar use case - If you use Istio it wants to deploy its own Jaeger instance that uses a deployment/service for the agent. If you want to use the Istio Jaeger instance but would rather use the sidecar for the agent instead of a central agent service, you'd need to be able to point the sidecar at the Jaeger instance over there.

We took it a step further and deployed our own Jaeger in a central monitoring namespace via Jaeger Operator and pointed Istio at that... however we're now finding that we can't use the sidecar injection for the same reason - unable to point at the central instance.

[jpkrohling]

I'll add this feature to my queue, and I might implement it in the next couple of weeks or next month. Of course, this can move faster if someone is willing to contribute the code.

I might be missing something obvious, but it shouldn't be that hard: it would require a new flag so that the owner of the operator can specify which namespaces to watch deployments for. Note that we currently use the same namespace when watching for new jaeger instances, so, it would probably require a split there, to use different watchers for different object types.

[marcindulak]

Is this auto-injection feature of sidercars into pods from a different namespace than the jaeger-operator already implemented?

I have jaeger-operator 2.6.0 installed with helm https://github.com/helm/charts/commit/0007ebb1587b3858c0bfb25a2210824a86434d3f
that provides jaegertracing/jaeger-operator:1.12.0.

Elasticsearch is a single container

kubectl create namespace observability
kubectl -n observability run elasticsearch --image=docker.elastic.co/elasticsearch/elasticsearch:6.7.2 --env="discovery.type=single-node" --port=9200
kubectl -n observability expose deployment elasticsearch --type=ClusterIP --name=elasticsearch

The jaeger 1.12 components are installed in the observability namespace:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger-elasticsearch
  namespace: observability
spec:
  agent:
    #strategy: DaemonSet  # inject jaeger-agent as a sidecar
    options:
      log-level: debug
  ingress:
    enabled: false
  resources:
    limits:
      memory: 192Mi
    requests:
      cpu: 100m
      memory: 128Mi
  storage:
    type: elasticsearch
    options:
      es:
        server-urls: http://elasticsearch:9200
  strategy: production

kubectl -n observability get pod -l app=jaeger -oyaml | grep "image:" | sort | uniq
      image: jaegertracing/jaeger-agent:1.12
      image: jaegertracing/jaeger-collector:1.12
      image: jaegertracing/jaeger-query:1.12

However, the jaeger-agent sidecar is only injected in the deployments in the observability namespace.

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    sidecar.jaegertracing.io/inject: jaeger-elasticsearch
  name: nginx-deployment
  namespace: production  # sidecar injected in observability namespace
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

kubectl -n production get pod
NAME                                READY     STATUS    RESTARTS   AGE
nginx-deployment-75675f5897-cj8pm   1/1       Running   0          13s

If I deploy nginx into the observability namespace then the jaeger-agent sidecar is injected

kubectl -n observability get pod
NAME                                              READY     STATUS    RESTARTS   AGE
elasticsearch-c46d956fb-rgdtj                     1/1       Running   1          3h
jaeger-elasticsearch-collector-76f485d9d8-w66r4   1/1       Running   4          1h
jaeger-elasticsearch-query-d9fdbf68f-g9np5        2/2       Running   4          1h
jaeger-operator-7bb8f65994-6zj22                  1/1       Running   3          3h
nginx-deployment-669bbd857-54sw5                  2/2       Running   0          17s

[jpkrohling]

What's the value of your WATCH_NAMESPACES env var, set in the operator.yaml? If it's empty, I'd expect the injection to also work cross-namespaces, but if it's not working, I'd have to check the code.

[marcindulak]

kubectl -n observability get pod jaeger-operator-7bb8f65994-6zj22 -oyaml | grep -A 4 WATCH_NAMESPACE
    - name: WATCH_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace

So it could be a problem in the helm chart https://github.com/helm/charts/commit/0007ebb1587b3858c0bfb25a2210824a86434d3f
I don't see any option to control this value in the chart.

They are hard-coded https://github.com/helm/charts/blob/0007ebb1587b3858c0bfb25a2210824a86434d3f/stable/jaeger-operator/templates/deployment.yaml#L41-L44

[christianhuening]

i've set the WATCH_NAMESPACE env var to "" :

 kubectl -n jaeger get pod jo-jaeger-operator-796d68cdb4-hhgl4 -oyaml |grep -A 4 WATCH_NAMESPACE 
    - name: WATCH_NAMESPACE
    - name: POD_NAME
      valueFrom:    
        fieldRef:   
          apiVersion: v1

However it still is not injecting the sidecar in another namespace.