Jaeger-operator: ServiceAccount needs to be created before ES rollover job

[bastianeicher]

I deployed Jaeger Operator v1.19.0 and an Elasticsearch instance in the same Kubernetes namespace. I then created the following resource:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger
spec:
  strategy: production
  collector:
    maxReplicas: 20
    autoScale: true
  storage:
    type: elasticsearch
    options:
      es:
        server-urls: http://jaeger-elasticsearch-master:9200
        use-aliases: true
    esRollover:
      enabled: true
      conditions: "{\"max_age\": \"2d\"}"
      readTTL: 168h
      schedule: "55 23 * * *"

Jaeger Operator then creates a Job named jaeger-es-rollover-create-mapping. However this Job fails to start because it references a not-yet-existing ServiceAccount named jaeger. If I do nothing the Operator keeps waiting for this Job and eventually marks the Jaeger instance as "Failed".

level=debug msg="Waiting for dependency to complete" name=jaeger-es-rollover-create-mapping
level=error msg="failed to apply the changes" error="timed out waiting for the condition"

If I run kubectl create sa jaeger the Job succeeds, allowing Jaeger Operator to progress.
Then the Operator tries to create the ServiceAccount itself but fails because it already exists.

level=error msg="failed to apply the changes" error="serviceaccounts \"jaeger\" already exists"

TL;DR: Jaeger Operator seems to be creating Jobs before ServiceAccounts but this should probably be the other way round.

[jpkrohling]

I think @pavolloffay fixed this (or a similar) issue a few days ago: https://github.com/jaegertracing/jaeger-operator/pull/1196

Are you able to use the image tagged with master for the operator, and see if this problem is fixed?

[bastianeicher]

Just tried the image jaegertracing/jaeger-operator:master and can confirm the problem is fixed there. Thanks! :)

Will there be a 1.19.1 release containing this fix?

[jpkrohling]

We don't have a plan for it yet, but I guess we could release one soon.

@rubenvp8510, @objectiser, what do you think?

[Baykonur]

I am also having the same issue which did not happen at first (2 weeks ago) but I guess the reason is the following:
I deployed a simple Jaeger CR all-in one with in-memory storage and then started using ES as backend for the existing deployment, when ES was introduced the sa was already present so all worked out.
Once I recreate everything from scratch (recreate the namespace, deploy Jaeger and ES CRs..), I have the exact same problem.
A new release will be much appreciated, till then, I do the following work-around:

I create a simple Jaeger CR as below:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: tracing

And let the Jaeger Operator do the magic (basically create the sa and the deployment) and then I push the real Jaeger CR which has ES as backend, make sure the metadata.name is the same.

[jpkrohling]

The release of Jaeger 1.20.0 is being planned at the moment and might happen very soon (this week?). A release of the operator will happen soon after, typically on the same day or the next day.

[jpkrohling]

The release is now out! I'm closing this, but feel free to reopen if the original problem is still happening.