Jaeger-operator: Migration to 1.22 missed Zipkin port change

Created on 26 Apr 2021  路  10Comments  路  Source: jaegertracing/jaeger-operator

Describe the bug
Starting on jaegertracing/jaeger-operator:1.22.1, jaeger-collector does not start Zipkin server automatically.

{"level":"info","ts":1619421380.7131286,"caller":"server/zipkin.go:48","msg":"Not listening for Zipkin HTTP traffic, port not configured"}

CR

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: mon-jaeger
  namespace: observability
spec:
  log-level: debug
  strategy: streaming
  sampling:
    options:
      default_strategy:
        type: const
        param: 100 
  collector:
    options:
      kafka: 
        producer:
          topic: jaeger-spans
          brokers: knative-kafka-cluster.knative-kafka:9092
  ingester:
    options:
      kafka:
        consumer:
          topic: jaeger-spans
          brokers: knative-kafka-cluster.knative-kafka:9092
      ingester:
        deadlockInterval: 0s
  storage:
    type: elasticsearch
    options:
      es:
        server-urls: https://elastic-es-http:9200
        index-prefix: jaeger-tracing
        tls:
          ca: /es/certificates/ca.crt
    secretName: jaeger-secret
  volumeMounts:
    - name: certificates
      mountPath: /es/certificates/
      readOnly: true
  volumes:
    - name: certificates
      secret:
        secretName: elastic-es-http-certs-public

Discussion
https://github.com/jaegertracing/jaeger/discussions/2959

To Reproduce
Steps to reproduce the behavior:

  1. Install operator jaegertracing/jaeger-operator:1.22.1 with log-level: debug
  2. Fire a valid CR to deploy Jaeger.
  3. Grab the collector's logs

You should see the message "Not listening for Zipkin HTTP traffic, port not configured" and Jaeger does not take any Zipking trace.

Expected behavior
Collector should start Zipkin server automatically.

Version (please complete the following information):

  • OS: Ubuntu 20.04
  • Jaeger version: 1.22.1
  • Deployment: Kubernetes - 1.18.16-gke.502

What troubleshooting steps did you try?
The same CR works with the previous version jaegertracing/jaeger-operator:1.21.3.

{"level":"info","ts":1619449101.7558992,"caller":"server/zipkin.go:49","msg":"Listening for Zipkin HTTP traffic","zipkin host-port":":9411"} 

cc @jpkrohling

bug good first issue help wanted

Most helpful comment

@gsagula You can fix it (for now) by using the following spec:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: tracing
spec:
  ...
  collector:
    options:
      collector.zipkin.host-port: ':9411'
     ...

(I've only verified this using the allInOne strategy.)

All 10 comments

@rubenvp8510, could you please take a look at this one?

@jpkrohling @rubenvp8510 Debugged this quickly because i ran into the same problem.
Seems like the Operator still reconciles the deployments with the env variable COLLECTOR_ZIPKIN_HTTP_PORT.
Since this (or better, the underlying arg --collector.zipkin.http-port) has been changed (in https://github.com/jaegertracing/jaeger/releases/tag/v1.22.0) to ...host-port instead, this is probably what's causing the issue.

@gsagula You can fix it (for now) by using the following spec:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: tracing
spec:
  ...
  collector:
    options:
      collector.zipkin.host-port: ':9411'
     ...

(I've only verified this using the allInOne strategy.)

Thanks @jeremyzahner

I did that but with "9411" instead of ":9411". It's working now!

Would one of you be interested in contributing a fix? Looks like we should have migrated the Zipkin parameter and value. This is where the code needs to be added:

https://github.com/jaegertracing/jaeger-operator/blob/5e9b39e9e091bb171b27edd61d02c7273cf6f55f/pkg/upgrade/v1_22_0.go#L11-L40

@jpkrohling You can assign this issue to me if nobody else is taking it.

Thanks for volunteering, @gsagula!

Sorry guys, I just started looking into this one today but I see that @rubenvp8510 already did it.

That's ok, we have plenty of other issues if you are interested :-)

Awesome, I can give this one a shot if nobody else is taking it. https://github.com/jaegertracing/jaeger-operator/issues/1422

Was this page helpful?
0 / 5 - 0 ratings

Related issues

wangycc picture wangycc  路  5Comments

qbast picture qbast  路  8Comments

daroga0002 picture daroga0002  路  3Comments

jkandasa picture jkandasa  路  3Comments

jpkrohling picture jpkrohling  路  7Comments