Jaeger-operator: jaeger-query route not created in OpenShift

[jkandasa]

Jaeger Operator: jaegertracing/jaeger-operator:1.9.2

OpenShift Master: v3.10.14
Kubernetes Master: v1.10.0+b81c8f8
OpenShift Web Console: v3.10.14

Steps followed,

export OS_NAMESPACE=observability

# download yaml files
curl https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/operator.yaml --output operator.yaml
curl https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role_binding.yaml --output role_binding.yaml
curl https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role.yaml --output role.yaml
curl https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/service_account.yaml --output service_account.yaml
curl https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/crds/io_v1alpha1_jaeger_crd.yaml --output io_v1alpha1_jaeger_crd.yaml

# create new project
oc new-project ${OS_NAMESPACE}

# deploy jaeger operator
oc create -f io_v1alpha1_jaeger_crd.yaml -n ${OS_NAMESPACE}
oc create -f service_account.yaml -n ${OS_NAMESPACE}
oc create -f role.yaml -n ${OS_NAMESPACE}
oc create -f role_binding.yaml -n ${OS_NAMESPACE}
oc create -f operator.yaml -n ${OS_NAMESPACE}

Steps followed to create jaeger services,

export OS_NAMESPACE=jkandasa
# deploy Elasticsearch
curl https://raw.githubusercontent.com/RHsyseng/docker-rhel-elasticsearch/5.x/es-cluster-deployment.yml --output es-cluster-deployment.yml
oc create -f es-cluster-deployment.yml -n ${OS_NAMESPACE}

oc apply -f jaeger-automation.yaml -n ${OS_NAMESPACE}

jaeger-automation.yaml:

apiVersion: io.jaegertracing/v1alpha1
kind: Jaeger
metadata:
  name: jaegerqe-automation
spec:
  strategy: production
  ingress:
    security: none
  collector:
    image: jaegertracing/jaeger-collector:1.10
    size: 2
    options:
      metrics-backend: expvar
  query:
    image: jaegertracing/jaeger-query:1.10
    size: 1
    options:
      metrics-backend: expvar
  storage:
    type: elasticsearch
    options:
      es:
        server-urls: http://elasticsearch:9200

logs from jaeger-operator:

time="2019-02-26T12:39:51Z" level=info msg=Go arch=amd64 os=linux version=go1.11.1
time="2019-02-26T12:39:51Z" level=info msg=operator-sdk version=v0.4.1
time="2019-02-26T12:39:51Z" level=info msg="Registering Components."
time="2019-02-26T12:39:51Z" level=info msg="Starting the Cmd."
time="2019-02-26T12:39:52Z" level=info msg="Reconciling Deployment" name=jaeger-collector namespace=jaeger-pipeline
time="2019-02-26T12:39:52Z" level=info msg="Reconciling Deployment" name=jaeger-query namespace=jaeger-pipeline
time="2019-02-26T12:39:52Z" level=info msg="Reconciling Deployment" name=jaeger-spans-reporter-tool namespace=jaeger-pipeline
time="2019-02-26T12:39:52Z" level=info msg="Reconciling Deployment" name=jaeger-operator namespace=observability
time="2019-02-26T12:39:52Z" level=info msg="Reconciling Deployment" name=webconsole namespace=openshift-web-console
time="2019-02-26T12:39:52Z" level=info msg="Reconciling Deployment" name=jaeger-operator namespace=observability
ERROR: logging before flag.Parse: W0226 12:47:52.932239       1 reflector.go:270] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: watch of *v1.Deployment ended with: The resourceVersion for the provided watch is too old.
time="2019-02-26T12:47:53Z" level=info msg="Reconciling Deployment" name=jaeger-collector namespace=jaeger-pipeline
time="2019-02-26T12:47:53Z" level=info msg="Reconciling Deployment" name=jaeger-query namespace=jaeger-pipeline
time="2019-02-26T12:47:53Z" level=info msg="Reconciling Deployment" name=jaeger-spans-reporter-tool namespace=jaeger-pipeline
time="2019-02-26T12:47:53Z" level=info msg="Reconciling Deployment" name=jaeger-operator namespace=observability
time="2019-02-26T12:47:53Z" level=info msg="Reconciling Deployment" name=webconsole namespace=openshift-web-console
ERROR: logging before flag.Parse: W0226 12:49:33.926580       1 reflector.go:270] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: watch of *v1alpha1.Jaeger ended with: The resourceVersion for the provided watch is too old.
time="2019-02-26T12:49:34Z" level=info msg="Reconciling Jaeger" name=jaegerqe-automation namespace=jkandasa
time="2019-02-26T12:49:34Z" level=info msg="Reconciling Deployment" name=jaegerqe-automation-collector namespace=jkandasa
time="2019-02-26T12:49:34Z" level=info msg="Reconciling Deployment" name=jaegerqe-automation-query namespace=jkandasa
time="2019-02-26T12:49:34Z" level=info msg="Reconciling Deployment" name=jaegerqe-automation-collector namespace=jkandasa
time="2019-02-26T12:49:34Z" level=info msg="Reconciling Deployment" name=jaegerqe-automation-query namespace=jkandasa
time="2019-02-26T12:49:34Z" level=info msg="Reconciling Deployment" name=jaegerqe-automation-collector namespace=jkandasa
time="2019-02-26T12:49:34Z" level=info msg="Reconciling Deployment" name=jaegerqe-automation-query namespace=jkandasa
time="2019-02-26T12:49:34Z" level=info msg="Configured Jaeger instance" name=jaegerqe-automation
time="2019-02-26T12:49:35Z" level=info msg="Reconciling Jaeger" name=jaegerqe-automation namespace=jkandasa
time="2019-02-26T12:49:35Z" level=info msg="Reconciling Deployment" name=jaegerqe-automation-collector namespace=jkandasa
time="2019-02-26T12:49:35Z" level=info msg="Reconciling Deployment" name=jaegerqe-automation-query namespace=jkandasa
time="2019-02-26T12:49:35Z" level=info msg="Reconciling Jaeger" name=jaegerqe-automation namespace=jkandasa
time="2019-02-26T12:49:46Z" level=info msg="Reconciling Deployment" name=jaegerqe-automation-collector namespace=jkandasa
time="2019-02-26T12:49:50Z" level=info msg="Reconciling Deployment" name=jaegerqe-automation-collector namespace=jkandasa
time="2019-02-26T12:49:50Z" level=info msg="Reconciling Deployment" name=jaegerqe-automation-query namespace=jkandasa
ERROR: logging before flag.Parse: W0226 12:58:00.951189       1 reflector.go:270] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: watch of *v1alpha1.Jaeger ended with: The resourceVersion for the provided watch is too old.
time="2019-02-26T12:58:01Z" level=info msg="Reconciling Jaeger" name=jaegerqe-automation namespace=jkandasa
ERROR: logging before flag.Parse: W0226 13:00:23.969054       1 reflector.go:270] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: watch of *v1.Deployment ended with: The resourceVersion for the provided watch is too old.
time="2019-02-26T13:00:24Z" level=info msg="Reconciling Deployment" name=jaeger-collector namespace=jaeger-pipeline
time="2019-02-26T13:00:24Z" level=info msg="Reconciling Deployment" name=jaeger-query namespace=jaeger-pipeline
time="2019-02-26T13:00:24Z" level=info msg="Reconciling Deployment" name=jaeger-spans-reporter-tool namespace=jaeger-pipeline
time="2019-02-26T13:00:24Z" level=info msg="Reconciling Deployment" name=jaegerqe-automation-collector namespace=jkandasa
time="2019-02-26T13:00:24Z" level=info msg="Reconciling Deployment" name=jaegerqe-automation-query namespace=jkandasa
time="2019-02-26T13:00:24Z" level=info msg="Reconciling Deployment" name=jaeger-operator namespace=observability
time="2019-02-26T13:00:24Z" level=info msg="Reconciling Deployment" name=webconsole namespace=openshift-web-console
ERROR: logging before flag.Parse: W0226 13:12:35.046998       1 reflector.go:270] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: watch of *v1alpha1.Jaeger ended with: The resourceVersion for the provided watch is too old.
time="2019-02-26T13:12:36Z" level=info msg="Reconciling Jaeger" name=jaegerqe-automation namespace=jkandasa
ERROR: logging before flag.Parse: W0226 13:14:03.025645       1 reflector.go:270] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: watch of *v1.Deployment ended with: The resourceVersion for the provided watch is too old.
time="2019-02-26T13:14:04Z" level=info msg="Reconciling Deployment" name=jaeger-collector namespace=jaeger-pipeline
time="2019-02-26T13:14:04Z" level=info msg="Reconciling Deployment" name=jaeger-query namespace=jaeger-pipeline
time="2019-02-26T13:14:04Z" level=info msg="Reconciling Deployment" name=jaeger-spans-reporter-tool namespace=jaeger-pipeline
time="2019-02-26T13:14:04Z" level=info msg="Reconciling Deployment" name=jaegerqe-automation-collector namespace=jkandasa
time="2019-02-26T13:14:04Z" level=info msg="Reconciling Deployment" name=jaegerqe-automation-query namespace=jkandasa
time="2019-02-26T13:14:04Z" level=info msg="Reconciling Deployment" name=jaeger-operator namespace=observability
time="2019-02-26T13:14:04Z" level=info msg="Reconciling Deployment" name=webconsole namespace=openshift-web-console

[jpkrohling]

~It won't create if the ingress security is set to None.~

apiVersion: io.jaegertracing/v1alpha1
kind: Jaeger
metadata:
  name: jaegerqe-automation
spec:
  strategy: production
  ingress:
    security: none # <<<<<<<<<<

~Could you try entirely removing the "ingress" part?~

Sorry, that's the OAuth Proxy that I meant. It should still create the route.

[jpkrohling]

Looks like we are hitting some naming limits here: renaming your instance to jaegerqe made it work for me.

[jkandasa]

@jpkrohling I tried with name: jaegerqe. but no success.

[jpkrohling]

I'm not sure what's going on, I can't reproduce this. Would you please follow the same steps I did below, sending me what's your output? I would also be interesting to run the operator with --log-level=debug.

$ kubectl get pods
NAME                     READY     STATUS    RESTARTS   AGE
elasticsearch-0          1/1       Running   0          4m
myapp-5c5f65d45b-88rv2   2/2       Running   0          35m


$ make run WATCH_NAMESPACE="" 
INFO[0000] Versions                                      arch=amd64 operator-sdk=v0.4.1 os=linux version=go1.11.1
INFO[0000] Auto-detected the platform                    platform=openshift
INFO[0000] Starting the Cmd.                            


$ cat /tmp/jaegerqe.yaml
apiVersion: io.jaegertracing/v1alpha1
kind: Jaeger
metadata:
  name: jaegerqe
spec:
  strategy: production
  ingress:
    security: none
  collector:
    image: jaegertracing/jaeger-collector:1.10
    size: 2
    options:
      metrics-backend: expvar
  query:
    image: jaegertracing/jaeger-query:1.10
    size: 1
    options:
      metrics-backend: expvar
  storage:
    type: elasticsearch
    options:
      es:
        server-urls: http://elasticsearch:9200


$ kubectl apply -f /tmp/jaegerqe.yaml 
jaeger.io.jaegertracing/jaegerqe created


$ kubectl get pods
NAME                                READY     STATUS    RESTARTS   AGE
elasticsearch-0                     1/1       Running   0          5m
jaegerqe-collector-ff4c98b6-5tv2r   1/1       Running   0          14s
jaegerqe-collector-ff4c98b6-q8kxg   1/1       Running   0          14s
jaegerqe-query-66845f456f-h5n66     1/1       Running   0          14s
myapp-5c5f65d45b-88rv2              2/2       Running   0          35m


$ kubectl get routes
NAME       HOST/PORT                                  PATH      SERVICES         PORT      TERMINATION   WILDCARD
jaegerqe   jaegerqe-myproject.192.168.42.198.nip.io             jaegerqe-query   <all>     edge          None
myapp      myapp-myproject.192.168.42.198.nip.io                myapp            http                    None

[jkandasa]

@jkandasa Looks like you are running on kubernetes environment?
I executed the steps mentioned on OpenShift environment,

OpenShift Master: v3.10.14
Kubernetes Master: v1.10.0+b81c8f8
OpenShift Web Console: v3.10.14

$ oc new-project observability
Now using project "observability" on server "https://internal.redhat.com:8443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-ex.git

to build a new example application in Ruby.
$ oc create -f io_v1alpha1_jaeger_crd.yaml -n observability
customresourcedefinition "jaegers.io.jaegertracing" created
$ oc create -f service_account.yaml -n observability
serviceaccount "jaeger-operator" created
$ oc create -f role.yaml -n observability
clusterrole "jaeger-operator" created
$ oc create -f role_binding.yaml -n observability
clusterrolebinding "jaeger-operator" created
$ oc create -f operator.yaml -n observability
deployment "jaeger-operator" created
$ oc get pods -n observability
NAME                               READY     STATUS    RESTARTS   AGE
jaeger-operator-68bf988d75-7r2ff   1/1       Running   0          25s
$ oc new-project jkandasa
Now using project "jkandasa" on server "https://internal.redhat.com:8443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-ex.git

to build a new example application in Ruby.
$ oc get pods -n jkandasa
No resources found.
$ oc create -f es-cluster-deployment.yml
configmap "elasticsearch-conf" created
statefulset "elasticsearch" created
service "elasticsearch" created
service "elasticsearch-cluster" created
imagestream "elasticsearch" created
$ oc get pods -n jkandasa
NAME              READY     STATUS    RESTARTS   AGE
elasticsearch-0   1/1       Running   0          18s
elasticsearch-1   1/1       Running   0          15s
elasticsearch-2   1/1       Running   0          11s
$ cat jaeger-automation.yaml 
apiVersion: io.jaegertracing/v1alpha1
kind: Jaeger
metadata:
  name: jaegerqe
spec:
  strategy: production
  ingress:
    security: none
  collector:
    image: jaegertracing/jaeger-collector:1.10
    size: 1
    options:
      metrics-backend: expvar
  query:
    image: jaegertracing/jaeger-query:1.10
    size: 1
    options:
      metrics-backend: expvar
  storage:
    type: elasticsearch
    options:
      es:
        server-urls: http://elasticsearch:9200
$ oc apply -f jaeger-automation.yaml 
jaeger "jaegerqe" created
$ oc get pods -n jkandasa
NAME                               READY     STATUS    RESTARTS   AGE
elasticsearch-0                    1/1       Running   0          1m
elasticsearch-1                    1/1       Running   0          1m
elasticsearch-2                    1/1       Running   0          1m
jaegerqe-collector-fb8688c-w2vhw   1/1       Running   0          17s
jaegerqe-query-6fbc6c9f5d-c7p8j    1/1       Running   0          17s
$ oc get routes
No resources found.

Jaeger Operator debug log: jaeger-operator-68bf988d75-7r2ff.log

[jpkrohling]

Looks like you are running on kubernetes environment?

No, routes are exclusive to OpenShift. Also, from the logs I sent earlier:

INFO[0000] Auto-detected the platform                    platform=openshift

It is possible that you are hitting a bug that was fixed already. I'll release a new version of the Operator shortly and would appreciate if you could re-test it.

[jpkrohling]

I'm closing this, as your screenshot in #243 imply that you have a route now.

[jkandasa]

@jpkrohling Tested jaeger-operator v1.10.0, route available for jaeger-query service