Jaeger-operator: Jaeger Operator not create query & collector pods

Created on 10 May 2020  路  23Comments  路  Source: jaegertracing/jaeger-operator

After I deployed the Jaeger operator on my cluster with the following configuration :
on kubernetes version 1.12.8
works with Elasticsearch operator as the DB source 

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger-operator
spec:
  strategy: production
  ingress:
    hosts:
      - jaeger-stg.nextinsurance.io
  storage:
    type: elasticsearch
    options:
      es:
        server-urls: https://jaeger-storage-es-http:9200
        tls:
          ca: /es/certificates/ca.crt
        num-shards: 1
        num-replicas: 0
    secretName: jaeger-secret
    esIndexCleaner:
      enabled: true
      image: jaegertracing/jaeger-es-index-cleaner:1.16.0
      numberOfDays: 7
      resources: {}
      schedule: 55 23 * * *
  volumeMounts:
    - name: certificates
      mountPath: /es/certificates/
      readOnly: true
  volumes:
    - name: certificates
      secret:
        secretName: jaeger-storage-es-http-certs-public

I try to deploy it again on a new cluster with version 1.16.9
And it didn't create the jaeger collector and the query pods.
What is missing?
Is there any way to find logs the show Jaeger errors?

bug needs-info question
Source
picture YonaLugassi

Most helpful comment

I don't even see the deployment. 

$ kubectl get jaegers
NAME       STATUS   VERSION   STRATEGY   STORAGE   AGE
simplest                                           46h
$ kubectl get deploy -A | grep simplest

(Also the cluster's not out of resources, I can spin up and down other pods just fine)

picture surlyengineer on 13 Aug 2020
👍3

All 23 comments

Can you share your jaeger-operator deployment specification e.g. k get deploy jaeger-operator?

It might be that the operator is not watching the namespace where the CR is created.

pavolloffay picture pavolloffay on 11 May 2020

Here is the jaeger deployment on the new k8s cluster:

ku -n observability get deployment jaeger-operator -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger-operator
  namespace: observability
  resourceVersion: "1184256"
  selfLink: /apis/apps/v1/namespaces/observability/deployments/jaeger-operator
  uid: 3ddfdaae-55cc-47e9-983a-59aec1658e49
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      name: jaeger-operator
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        name: jaeger-operator
    spec:
      containers:
      - args:
        - start
        env:
        - name: WATCH_NAMESPACE
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: OPERATOR_NAME
          value: jaeger-operator
        image: jaegertracing/jaeger-operator:1.16.0
        imagePullPolicy: Always
        name: jaeger-operator
        ports:
        - containerPort: 8383
          name: metrics
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: jaeger-operator
      serviceAccountName: jaeger-operator
      terminationGracePeriodSeconds: 30

I saw that there were differences in ServiceAccount on the new k8s cluster that the Jaeger Operator would move to

Here is the old version 

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},"name":"jaeger-operator","namespace":"monitoring-stg"}}
  creationTimestamp: "2020-05-11T11:30:59Z"
  labels:
    app: jaeger
    app.kubernetes.io/component: service-account
    app.kubernetes.io/instance: jaeger-operator
    app.kubernetes.io/managed-by: jaeger-operator
    app.kubernetes.io/name: jaeger-operator
    app.kubernetes.io/part-of: jaeger
  name: jaeger-operator
  namespace: monitoring-stg
  ownerReferences:
  - apiVersion: jaegertracing.io/v1
    controller: true
    kind: Jaeger
    name: jaeger-operator
    uid: 85d742d2-76f3-4ba2-9808-555d7725006e
  resourceVersion: "1189791"
  selfLink: /api/v1/namespaces/monitoring-stg/serviceaccounts/jaeger-operator
  uid: 02f986d8-1456-4672-bbf4-0c1765d458ea
secrets:
- name: jaeger-operator-token-vwwql

and the new version looks like this

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},"name":"jaeger-operator-new","namespace":"monitoring-stg"}}
  creationTimestamp: "2020-05-13T14:18:51Z"
  name: jaeger-operator
  namespace: monitoring-stg
  resourceVersion: "1733099"
  selfLink: /api/v1/namespaces/monitoring-stg/serviceaccounts/jaeger-operator-new
  uid: 791497f6-7ecb-4824-883a-27ef22ff3950
secrets:
- name: jaeger-operator-token-k5tlv

after I comper between them and delete the Jaeger operator and apply it again the resources were created on the new cluster.
from some reason, the ownerReferences and the labels were missing on the new k8s cluster

YonaLugassi picture YonaLugassi on 13 May 2020

+1

tanjunchen picture tanjunchen on 1 Jun 2020

here is the jaeger-operator.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"name":"jaeger-operator","namespace":"observability"},"spec":{"replicas":1,"selector":{"matchLabels":{"name":"jaeger-operator"}},"template":{"metadata":{"labels":{"name":"jaeger-operator"}},"spec":{"containers":[{"args":["start"],"env":[{"name":"WATCH_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"OPERATOR_NAME","value":"jaeger-operator"}],"image":"jaegertracing/jaeger-operator:1.18.0","imagePullPolicy":"Always","name":"jaeger-operator","ports":[{"containerPort":8383,"name":"metrics"}]}],"serviceAccountName":"jaeger-operator"}}}}
  creationTimestamp: "2020-05-29T07:57:49Z"
  generation: 2
  name: jaeger-operator
  namespace: observability
  resourceVersion: "493461"
  selfLink: /apis/apps/v1/namespaces/observability/deployments/jaeger-operator
  uid: 60487a2b-1ad9-403e-902f-c842b33ad09b
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      name: jaeger-operator
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        name: jaeger-operator
    spec:
      containers:
      - args:
        - start
        env:
        - name: WATCH_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: OPERATOR_NAME
          value: jaeger-operator
        image: jaegertracing/jaeger-operator:1.18.0
        imagePullPolicy: Always
        name: jaeger-operator
        ports:
        - containerPort: 8383
          name: metrics
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: jaeger-operator
      serviceAccountName: jaeger-operator
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2020-05-29T08:10:49Z"
    lastUpdateTime: "2020-05-29T08:10:49Z"
    message: ReplicaSet "jaeger-operator-547567dddb" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2020-06-01T01:16:19Z"
    lastUpdateTime: "2020-06-01T01:16:19Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 2
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1
tanjunchen picture tanjunchen on 1 Jun 2020

@tanjunchen , could you please share the jaeger-operator logs?

kubectl logs -n observability deployment/jaeger-operator
jpkrohling picture jpkrohling on 10 Jun 2020

I'm having the same problem with a docker-for-desktop getting started with the all-in-one deployment.

surlyengineer picture surlyengineer on 11 Aug 2020

@surlyengineer, could you please share the jaeger-operator logs? 

$ kubectl logs -n observability deployment/jaeger-operator
jpkrohling picture jpkrohling on 12 Aug 2020

Sure - coming back to this this morning has been weird because we have: 

kubectl logs -n observability deployment/jaeger-operator
time="2020-08-11T18:07:14Z" level=info msg=Versions arch=amd64 identity=observability.jaeger-operator jaeger=1.17.1 jaeger-operator=v1.17.1 operator-sdk=v0.15.1 os=linux version=go1.13.3
time="2020-08-11T18:07:15Z" level=info msg="Consider running the operator in a cluster-wide scope for extra features"
time="2020-08-11T18:07:15Z" level=info msg="Auto-detected the platform" platform=kubernetes
time="2020-08-11T18:07:15Z" level=info msg="Automatically adjusted the 'es-provision' flag" es-provision=no
time="2020-08-11T18:07:15Z" level=info msg="Automatically adjusted the 'kafka-provision' flag" kafka-provision=no
time="2020-08-11T18:07:15Z" level=info msg="The service account running this operator has the role 'system:auth-delegator', enabling OAuth Proxy's 'delegate-urls' option"
time="2020-08-11T18:07:17Z" level=info msg="Install prometheus-operator in your cluster to create ServiceMonitor objects" error="no ServiceMonitor registered with the API"

After I first noticed the issue (note the timestamp from two days ago). The kubectl get jaegers command works just fine but no pods are created in response. The version being a few down was me intentionally downgrading it to see if it was an issue only in the latest release. The logs are silent until the next day I start seeing:

time="2020-08-12T11:22:39Z" level=error msg="error getting a list of deployments to analyze in namespace" error="Get https://10.96.0.1:443/apis/apps/v1/namespaces/observability/deployments: http2: server sent GOAWAY and closed the connection; LastStreamID=78745, ErrCode=NO_ERROR, debug=\"\"" namespace=observability
E0812 11:23:17.841840       1 reflector.go:280] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:96: Failed to watch *v1.Jaeger: Get https://10.96.0.1:443/apis/jaegertracing.io/v1/namespaces/observability/jaegers?allowWatchBookmarks=true&resourceVersion=9452135&timeoutSeconds=422&watch=true: dial tcp 10.96.0.1:443: i/o timeout
E0812 11:23:18.316640       1 reflector.go:280] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:96: Failed to watch *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?allowWatchBookmarks=true&resourceVersion=9452033&timeoutSeconds=329&watch=true: dial tcp 10.96.0.1:443: connect: connection refused
E0812 11:23:18.320099       1 reflector.go:280] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:96: Failed to watch *v1.Deployment: Get https://10.96.0.1:443/apis/apps/v1/namespaces/observability/deployments?allowWatchBookmarks=true&resourceVersion=9452090&timeoutSeconds=411&watch=true: dial tcp 10.96.0.1:443: connect: connection refused
time="2020-08-12T11:23:18Z" level=error msg="error getting a list of existing jaeger instances in namespace" error="Get https://10.96.0.1:443/apis/jaegertracing.io/v1/namespaces/observability/jaegers: dial tcp 10.96.0.1:443: connect: connection refused" namespace=observability
time="2020-08-12T11:23:18Z" level=info msg="failed to determine the platform capabilities, auto-detected properties will fallback to their default values." error="Get https://10.96.0.1:443/api?timeout=32s: dial tcp 10.96.0.1:443: connect: connection refused"

This repeats for pretty much every resource for a few hours. I'm going to tear down the whole deployment and start over just to make sure that gap in timing wasn't a coincidence.

surlyengineer picture surlyengineer on 13 Aug 2020

Looks like your cluster is out of resources? The operator creates a deployment only, which then causes a replica set to be created, which in turn creates the actual pod.

Could you please check whether you have deployments and replica sets for the expected pods? Like:

$ kubectl get deployments
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
simplest   1/1     1            1           4m

$ kubectl get replicasets
NAME                  DESIRED   CURRENT   READY   AGE
simplest-77b8c6fc95   1         1         1       4m8s

$ kubectl get pods
NAME                        READY   STATUS    RESTARTS   AGE
simplest-77b8c6fc95-22pr6   1/1     Running   0          4m15s

If you see a replica set but not a pod, check the replica's status object for the causes. If you don't even see a deployment, then it's certainly something wrong with the operator.

jpkrohling picture jpkrohling on 13 Aug 2020

I don't even see the deployment. 

$ kubectl get jaegers
NAME       STATUS   VERSION   STRATEGY   STORAGE   AGE
simplest                                           46h
$ kubectl get deploy -A | grep simplest

(Also the cluster's not out of resources, I can spin up and down other pods just fine)

surlyengineer picture surlyengineer on 13 Aug 2020
👍3

Thanks! That does sound odd. Would you please provide the debug logs? In your operator.yaml, add the following flag to the args:

        args: ["start", "--log-level=debug"]
jpkrohling picture jpkrohling on 17 Aug 2020

That didn't seem to change the output:

$ kubectl logs -n observability deployment/jaeger-operator -f
time="2020-08-17T13:17:31Z" level=info msg=Versions arch=amd64 identity=observability.jaeger-operator jaeger=1.17.1 jaeger-operator=v1.17.1 operator-sdk=v0.15.1 os=linux version=go1.13.3
time="2020-08-17T13:17:31Z" level=info msg="Consider running the operator in a cluster-wide scope for extra features"
time="2020-08-17T13:17:32Z" level=info msg="Auto-detected the platform" platform=kubernetes
time="2020-08-17T13:17:32Z" level=info msg="Automatically adjusted the 'es-provision' flag" es-provision=no
time="2020-08-17T13:17:32Z" level=info msg="Automatically adjusted the 'kafka-provision' flag" kafka-provision=no
time="2020-08-17T13:17:32Z" level=info msg="The service account running this operator has the role 'system:auth-delegator', enabling OAuth Proxy's 'delegate-urls' option"
time="2020-08-17T13:17:33Z" level=info msg="Install prometheus-operator in your cluster to create ServiceMonitor objects" error="no ServiceMonitor registered with the API"
surlyengineer picture surlyengineer on 17 Aug 2020

One point of curiosity - in the logs there we have Consider running the operator in a cluster-wide scope for extra features but I have the cluster roles enabled. That's a little odd, no?

surlyengineer picture surlyengineer on 17 Aug 2020

Huh - Okay. Made some progress. The key seemed to be this line in the README:

You probably also want to download and customize the operator.yaml, setting the env var WATCH_NAMESPACE to have an empty value, so that it can watch for instances across all namespaces.

Once I did that, it was able to create the query pods and whatnot, but it's creating them in the default namespace which seems off.

surlyengineer picture surlyengineer on 17 Aug 2020

@surlyengineer about not changing the output: you might be affected by #1129 , could you try the master image? Here's what to expect when you run the Jaeger Operator in the observability namespace and created an instance named simplest in the same observability namespace:

$ kubectl logs jaeger-operator-fcb558d65-hshmp
time="2020-08-17T13:31:20Z" level=info msg=Versions arch=amd64 identity=default.jaeger-operator jaeger=1.18.1 jaeger-operator=master operator-sdk=v0.18.2 os=linux version=go1.14.4
I0817 13:31:21.997539       1 request.go:621] Throttling request took 1.039002289s, request: GET:https://172.25.0.1:443/apis/storage.k8s.io/v1beta1?timeout=32s
time="2020-08-17T13:31:39Z" level=info msg="Consider running the operator in a cluster-wide scope for extra features"
I0817 13:31:40.972175       1 request.go:621] Throttling request took 1.047980702s, request: GET:https://172.25.0.1:443/apis/storage.k8s.io/v1?timeout=32s
time="2020-08-17T13:31:42Z" level=info msg="Auto-detected the platform" platform=openshift
time="2020-08-17T13:31:42Z" level=info msg="Auto-detected ingress api" ingress-api=networking
time="2020-08-17T13:31:42Z" level=info msg="Automatically adjusted the 'es-provision' flag" es-provision=no
time="2020-08-17T13:31:42Z" level=info msg="Automatically adjusted the 'kafka-provision' flag" kafka-provision=no
time="2020-08-17T13:31:42Z" level=info msg="The service account running this operator does not have the role 'system:auth-delegator', consider granting it for additional capabilities"
time="2020-08-17T13:31:48Z" level=warning msg="could not create ServiceMonitor object" error="servicemonitors.monitoring.coreos.com \"jaeger-operator-metrics\" already exists"
time="2020-08-17T13:31:48Z" level=info msg="OAuthProxy ImageStream namespace and/or name not defined" name= namespace=
time="2020-08-17T13:31:48Z" level=debug msg="Reconciling Jaeger" execution="2020-08-17 13:31:48.723313894 +0000 UTC" instance=simplest namespace=default
time="2020-08-17T13:31:48Z" level=debug msg="Strategy chosen" instance=simplest namespace=default strategy=allinone
time="2020-08-17T13:31:48Z" level=debug msg="Creating all-in-one deployment" instance=simplest namespace=default
time="2020-08-17T13:31:48Z" level=debug msg="Assembling the UI configmap" instance=simplest namespace=default
time="2020-08-17T13:31:48Z" level=debug msg="Assembling the Sampling configmap" instance=simplest namespace=default
time="2020-08-17T13:31:48Z" level=debug msg="CA: Creating the trustedCABundle configmap" instance=simplest namespace=default
time="2020-08-17T13:31:48Z" level=debug msg="CA: Creating the service CA configmap" instance=simplest namespace=default
time="2020-08-17T13:31:48Z" level=debug msg="Assembling an all-in-one deployment" instance=simplest namespace=default
time="2020-08-17T13:31:48Z" level=debug msg="found a volume mount with the trusted-ca" instance=simplest namespace=default
time="2020-08-17T13:31:48Z" level=debug msg="skipping agent daemonset" instance=simplest namespace=default strategy=
time="2020-08-17T13:31:48Z" level=debug msg="updating service account" account=simplest-ui-proxy instance=simplest namespace=default
time="2020-08-17T13:31:48Z" level=debug msg="updating service account" account=simplest instance=simplest namespace=default
time="2020-08-17T13:31:48Z" level=debug msg="creating config maps" configMap=simplest-service-ca instance=simplest namespace=default
time="2020-08-17T13:31:48Z" level=debug msg="updating config maps" configMap=simplest-ui-configuration instance=simplest namespace=default
time="2020-08-17T13:31:48Z" level=debug msg="updating config maps" configMap=simplest-sampling-configuration instance=simplest namespace=default
time="2020-08-17T13:31:48Z" level=debug msg="updating config maps" configMap=simplest-trusted-ca instance=simplest namespace=default
time="2020-08-17T13:31:48Z" level=debug msg="updating service" instance=simplest namespace=default service=simplest-collector-headless
time="2020-08-17T13:31:48Z" level=debug msg="updating service" instance=simplest namespace=default service=simplest-collector
time="2020-08-17T13:31:48Z" level=debug msg="updating service" instance=simplest namespace=default service=simplest-query
time="2020-08-17T13:31:48Z" level=debug msg="updating service" instance=simplest namespace=default service=simplest-agent
time="2020-08-17T13:31:48Z" level=debug msg="updating deployment" deployment=simplest instance=simplest namespace=default
time="2020-08-17T13:31:48Z" level=debug msg="Deployment has stabilized" desired=1 name=simplest namespace=default ready=1
time="2020-08-17T13:31:48Z" level=debug msg="updating route" instance=simplest namespace=default route=simplest
time="2020-08-17T13:31:48Z" level=debug msg="Reconciling Jaeger completed" execution="2020-08-17 13:31:48.723313894 +0000 UTC" instance=simplest namespace=default
jpkrohling picture jpkrohling on 17 Aug 2020

One point of curiosity - in the logs there we have Consider running the operator in a cluster-wide scope for extra features but I have the cluster roles enabled. That's a little odd, no?

Cluster-wide means setting WATCH_NAMESPACE to empty, so that it watches all namespaces in the cluster.

it was able to create the query pods and whatnot, but it's creating them in the default namespace which seems off.

If you are creating the Jaeger CR in the default namespace, then it will create all the pods/deployments in the default namespace. This might explain why you are not seeing any pods: you are creating the CR in a namespace that the operator isn't watching.

jpkrohling picture jpkrohling on 17 Aug 2020

That seems off, though, since CR's aren't namespaced (https://kubernetes.io/docs/reference/access-authn-authz/rbac/#clusterrole-example) right? Possible I'm missing something obvious though.

surlyengineer picture surlyengineer on 17 Aug 2020

@surlyengineer about not changing the output: you might be affected by #1129 , could you try the master image? Here's what to expect when you run the Jaeger Operator in the observability namespace and created an instance named simplest in the same observability namespace:

I'll give this a shot this afternoon (US East)

surlyengineer picture surlyengineer on 17 Aug 2020

That seems off, though, since CR's aren't namespaced

Not all of them. The one you linked (ClusterRole) certainly is, as it's about creating a role that is valid for the whole cluster. Jaeger instances are indeed namespaced, like Deployment, Service, Pod, and other resources, making it easier to protect and isolate them.

jpkrohling picture jpkrohling on 18 Aug 2020

For reference, you may check which custom resources are namespaced and which aren't by looking into the .Spec.Scope field. Here's Jaeger's: https://github.com/jaegertracing/jaeger-operator/blob/f443e78e18d14e8e18d9ce2ed162346cf90583ad/deploy/crds/jaegertracing.io_jaegers_crd.yaml#L35

jpkrohling picture jpkrohling on 18 Aug 2020

@jpkrohling I also met a similar issue.. my question is that even I have set spec.strategy = production as below.

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger-production
  namespace: istio-system
spec:
  strategy: production
  collector:
    options:
      log-level: debug
    replicas: 3
    resources:
      limits:
        cpu: 100m
        memory: 128Mi
    storage:
      type: elasticsearch
      options:
        es:
          server-urls: http://x.x.x.x:9200
          username: xxx
          password: xxx
          index-prefix: jaeger-
  query:
    options:
      log-level: debug
  ingress:
    enabled: false

The operator log is still like strategy=allinone as follows...

time="2020-09-15T10:17:47Z" level=debug msg="Strategy chosen" instance=jaeger-production namespace=istio-system strategy=allinone
time="2020-09-15T10:17:47Z" level=debug msg="Creating all-in-one deployment" instance=jaeger-production namespace=istio-system
time="2020-09-15T10:17:47Z" level=debug msg="Assembling the UI configmap" instance=jaeger-production namespace=istio-system
time="2020-09-15T10:17:47Z" level=debug msg="Assembling the Sampling configmap" instance=jaeger-production namespace=istio-system
time="2020-09-15T10:17:47Z" level=debug msg="Assembling an all-in-one deployment" instance=jaeger-production namespace=istio-system
time="2020-09-15T10:17:47Z" level=debug msg="skipping agent daemonset" instance=jaeger-production namespace=istio-system strategy=
time="2020-09-15T10:17:47Z" level=debug msg="updating service account" account=jaeger-production instance=jaeger-production namespace=istio-system
time="2020-09-15T10:17:47Z" level=debug msg="updating config maps" configMap=jaeger-production-ui-configuration instance=jaeger-production namespace=istio-system
time="2020-09-15T10:17:47Z" level=debug msg="updating config maps" configMap=jaeger-production-sampling-configuration instance=jaeger-production namespace=istio-system
time="2020-09-15T10:17:47Z" level=debug msg="updating service" instance=jaeger-production namespace=istio-system service=jaeger-production-collector
time="2020-09-15T10:17:47Z" level=debug msg="updating service" instance=jaeger-production namespace=istio-system service=jaeger-production-query
time="2020-09-15T10:17:47Z" level=debug msg="updating service" instance=jaeger-production namespace=istio-system service=jaeger-production-agent
time="2020-09-15T10:17:47Z" level=debug msg="updating service" instance=jaeger-production namespace=istio-system service=jaeger-production-collector-headless
time="2020-09-15T10:17:47Z" level=debug msg="updating deployment" deployment=jaeger-production instance=jaeger-production namespace=istio-system
time="2020-09-15T10:17:47Z" level=debug msg="Deployment has stabilized" desired=1 name=jaeger-production namespace=istio-system ready=1
time="2020-09-15T10:17:47Z" level=debug msg="Reconciling Jaeger completed" execution="2020-09-15 10:17:47.391648149 +0000 UTC" instance=jaeger-production namespace=istio-system

And when I get the YAML of Jaeger, Got the same result...

API Version:  jaegertracing.io/v1
Kind:         Jaeger
Metadata:
  Creation Timestamp:  2020-09-15T10:17:43Z
  Generation:          3
  Resource Version:    130744391
  Self Link:           /apis/jaegertracing.io/v1/namespaces/istio-system/jaegers/jaeger-production
  UID:                 ab4e0750-da72-4846-b4ca-7637a20b2b4e
Spec:
  Agent:
    Config:
    Options:
    Resources:
  All In One:
    Config:
    Options:
    Resources:
  Collector:
    Config:
    Options:
      Log - Level:  debug
    Replicas:       3
    Resources:
      Limits:
        Cpu:     100m
        Memory:  128Mi
  Ingester:
    Config:
    Options:
    Resources:
  Ingress:
    Enabled:  false
    Openshift:
    Options:
    Resources:
    Security:  none
  Query:
    Options:
      Log - Level:  debug
    Resources:
  Resources:
  Sampling:
    Options:
  Storage:
    Cassandra Create Schema:
    Dependencies:
      Resources:
      Schedule:  55 23 * * *
    Elasticsearch:
      Node Count:         3
      Redundancy Policy:  SingleRedundancy
      Resources:
        Limits:
          Memory:  16Gi
        Requests:
          Cpu:     1
          Memory:  16Gi
      Storage:
    Es Index Cleaner:
      Number Of Days:  7
      Resources:
      Schedule:  55 23 * * *
    Es Rollover:
      Resources:
      Schedule:  0 0 * * *
    Options:
    Type:    memory
  Strategy:  allinone
  Ui:
    Options:
      Menu:
        Items:
          Label:  Documentation
          URL:    https://www.jaegertracing.io/docs/1.19
        Label:    About
Status:
  Phase:    Running
  Version:  1.19.2
Events:     <none>

My Operator version is 1.19.0, and the Kubernetes version is 1.16.1. I am wondering is there any other configuration that affects the strategy chosen of Jaeger Operator? Any advice would be very helpful. thank you

kidrocknroll picture kidrocknroll on 15 Sep 2020

It looks like your storage node is misaligned. Yours is under .spec.collector.storage, and should be directly under .spec. As the operator can't find the storage node, it assumes you want a default one (all in one).

In any case, the operator seems to have created a Jaeger deployment:

time="2020-09-15T10:17:47Z" level=debug msg="updating deployment" deployment=jaeger-production instance=jaeger-production namespace=istio-system

Do you see pods for this deployment?

jpkrohling picture jpkrohling on 15 Sep 2020

It looks like your storage node is misaligned. Yours is under .spec.collector.storage, and should be directly under .spec. As the operator can't find the storage node, it assumes you want a default one (all in one).

In any case, the operator seems to have created a Jaeger deployment:

time="2020-09-15T10:17:47Z" level=debug msg="updating deployment" deployment=jaeger-production instance=jaeger-production namespace=istio-system

Do you see pods for this deployment?

Thanks a lot. It indeed solves the problem, Now i got the production strategy. Yes, I deed see these logs after every time of change the Jaeger YAML, So I thought the configuration is OK. Sorry, my Bad, I should double-check... But is really trick that other field would affect the way how Jaeger is deployed, and the log does not show any error msg.

kidrocknroll picture kidrocknroll on 15 Sep 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

daroga0002 picture daroga0002  路  3Comments
figaw picture figaw  路  4Comments
majidazimi picture majidazimi  路  5Comments
ewohltman picture ewohltman  路  7Comments
qbast picture qbast  路  8Comments