Jaeger-operator: Getting Started with Jaeger Operator and Elasticsearch

[jungopro]

Hello

I'm new to the whole jaeger and elasticsearch world and I'm trying to verify that my Jaeger is properly configured and connected to elasticsearch

• I deployed the jaeger-operator helm chart and the elastic-stack helm chart from the helm/stable repo
• I verified that elasticsearch is healthy:

[root@elastic-stack-elasticsearch-master-0 elasticsearch]# curl -XGET http://elastic-stack-elasticsearch-client:9200/_cluster/health?pretty
{
  "cluster_name" : "elasticsearch",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 7,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 1,
  "active_shards" : 2,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

• I verified that the elasticsearch svc is properly configured in the jaeger instance:

NAME                                    TYPE           CLUSTER-IP       EXTERNAL-IP                                                                PORT(S)                      AGE
elastic-stack-elasticsearch-client      ClusterIP      10.100.106.216   <none>                                                                     9200/TCP                     14d
elastic-stack-elasticsearch-discovery   ClusterIP      None             <none>                                                                     9300/TCP                     14d

```yaml
Name: marco-polo-jaeger
Namespace: marco-polo
Labels: app.kubernetes.io/instance=marco-polo
app.kubernetes.io/managed-by=Tiller
app.kubernetes.io/name=marco-polo
component=marco-polo-jaeger
helm.sh/chart=marco-polo-0.1.0
project=marco-polo
Annotations:
API Version: io.jaegertracing/v1alpha1
Kind: Jaeger
Metadata:
Creation Timestamp: 2019-02-27T15:08:07Z
Generation: 1
Resource Version: 4983384
Self Link: /apis/io.jaegertracing/v1alpha1/namespaces/marco-polo/jaegers/marco-polo-jaeger
UID: 7cee03b1-3aa1-11e9-bbf5-06c840c1a2d6
Spec:
Agent:
Image: jaegertracing/jaeger-agent:1.9
Options:
Resources:
Strategy:
Volume Mounts:
Volumes:
All In One:
Image:
Options:
Resources:
Volume Mounts:
Volumes:
Collector:
Image: jaegertracing/jaeger-collector:1.9
Options:
Resources:
Size: 1
Volume Mounts:
Volumes:
Ingress:
Enabled: false
Resources:
Security:
Volume Mounts:
Volumes:
Query:
Image: jaegertracing/jaeger-query:1.9
Options:
Resources:
Size: 1
Volume Mounts:
Volumes:
Resources:
Sampling:
Options:
Storage:
Cassandra Create Schema:
Datacenter:
Enabled:
Image:
Mode:
Dependencies:
Cassandra Client Auth Enabled: false
Cassandra Local Dc:
Cassandra Use Ssl: false
Elasticsearch Client Node Only: false
Elasticsearch Nodes Wan Only: false
Enabled: true
Image: jaegertracing/spark-dependencies
Java Opts:
Schedule: 55 23 * * *
Spark Master:
Es Index Cleaner:
Enabled: true
Image: jaegertracing/jaeger-es-index-cleaner
Number Of Days: 7
Schedule: 55 23 * * *
Options:
Es . Server - Urls: http://elastic-stack-elasticsearch-client.default.svc.cluster.local:9200
Secret Name:
Type: elasticsearch
Strategy: production
Ui:
Options:
Volume Mounts:
Volumes:
Status:
Events:

I see this error from the **jaeger-es-index-cleaner**, which I think it means that there are no indices created in elasticsearch:

```bash
raceback (most recent call last):
  File "/es-index-cleaner/esCleaner.py", line 88, in <module>
    main()
  File "/es-index-cleaner/esCleaner.py", line 49, in main
    filter_main_indices(ilo, prefix)
  File "/es-index-cleaner/esCleaner.py", line 64, in filter_main_indices
    ilo.filter_by_age(source='name', direction='older', timestring='%Y-%m-%d', unit='days', unit_count=int(sys.argv[1]))
  File "/usr/local/lib/python3.7/site-packages/curator/indexlist.py", line 500, in filter_by_age
    stats_result=stats_result
  File "/usr/local/lib/python3.7/site-packages/curator/indexlist.py", line 358, in _calculate_ages
    self._get_name_based_ages(timestring)
  File "/usr/local/lib/python3.7/site-packages/curator/indexlist.py", line 281, in _get_name_based_ages
    self.empty_list_check()
  File "/usr/local/lib/python3.7/site-packages/curator/indexlist.py", line 228, in empty_list_check
    raise exceptions.NoIndices('index_list object is empty.')
curator.exceptions.NoIndices: index_list object is empty.

Aside from that, I don't know if I need to configure anything more or how to do a simple smoke test.
Any help or guidance to a proper documentation is greatly appriciated

Thank you
Omer

[jpkrohling]

curator.exceptions.NoIndices: index_list object is empty.

I've seen this kind of problem happening when there is no data in your cluster, but @pavolloffay would be the right person to answer this part.

As a general rule, if the collector and query pods are in a healthy state, it means they did connect to ES. Once you store the first span, all the indexes will be created and the jobs should then work.

One thing did catch my attention:

    Options:
      Es . Server - Urls:  http://elastic-stack-elasticsearch-client.default.svc.cluster.local:9200

Looks like something is converting dashes into spaces. Could you please confirm the option is stored as es.server-urls?

[pavolloffay]

The error from index cleaner is there because there are no jaeger indices in Elasticsearch. Jaeger creates indices only if there are any data to store.

Once you report data to Jaeger the index cleaner will not exit with error code but instead it will print that there are no indices to delete.

[pavolloffay]

If we think that ES cleaner should exit with 0 if there are no jaeger indices open the issue against the main repository

[jungopro]

hi @jpkrohling, thanks for your reply
I'm using helm to generate the yaml for the jaeger instance. the template looks like this:

{{ if .Values.jaeger.deploy }}
{{- $serviceName := printf "%s-%s" .Values.global.project .Values.jaeger.name -}}
apiVersion: io.jaegertracing/v1alpha1
kind: Jaeger
metadata:
  name: {{ $serviceName }}
  labels:
    app.kubernetes.io/name: {{ include "marco-polo.name" . }}
    helm.sh/chart: {{ include "marco-polo.chart" . }}
    app.kubernetes.io/instance: {{ .Release.Name }}
    app.kubernetes.io/managed-by: {{ .Release.Service }}
    component: {{ $serviceName }}
    {{- if .Values.jaeger.extraLables }}
    {{- range $key, $value := .Values.jaeger.extraLables }}
    {{ $key }}: {{ $value }}
    {{- end }}
    {{- end }}
    {{- if .Values.global.extraLables }}
    {{- range $key, $value := .Values.global.extraLables }}
    {{ $key }}: {{ $value }}
    {{- end }}
    {{- end }}
spec:
  strategy: {{ .Values.jaeger.strategy }}
  storage:
    type: {{ .Values.jaeger.storageType }}
    options:
      es:
        server-urls: {{ .Values.jaeger.storageURL }}
  ingress:
    enabled: false
{{ end }}

the generate template (helm template) looks like this:

templates/jaeger.yaml
---
# Source: marco-polo/templates/jaeger.yaml
apiVersion: io.jaegertracing/v1alpha1
kind: Jaeger
metadata:
  name: marco-polo-jaeger
  labels:
    app.kubernetes.io/name: marco-polo
    helm.sh/chart: marco-polo-0.1.0
    app.kubernetes.io/instance: release-name
    app.kubernetes.io/managed-by: Tiller
    component: marco-polo-jaeger
    project: marco-polo
spec:
  strategy: production
  storage:
    type: elasticsearch
    options:
      es:
        server-urls: http://elastic-stack-elasticsearch-client.default.svc.cluster.local:9200
  ingress:
    enabled: false

and the previous output is from the command kubectl describe jaegers.io.jaegertracing marco-polo-jaeger:

$ kubectl describe jaegers.io.jaegertracing marco-polo-jaeger
Name:         marco-polo-jaeger
Namespace:    marco-polo
Labels:       app.kubernetes.io/instance=marco-polo
              app.kubernetes.io/managed-by=Tiller
              app.kubernetes.io/name=marco-polo
              component=marco-polo-jaeger
              helm.sh/chart=marco-polo-0.1.0
              project=marco-polo
Annotations:  <none>
API Version:  io.jaegertracing/v1alpha1
Kind:         Jaeger
Metadata:
  Creation Timestamp:  2019-02-27T15:08:07Z
  Generation:          1
  Resource Version:    4983384
  Self Link:           /apis/io.jaegertracing/v1alpha1/namespaces/marco-polo/jaegers/marco-polo-jaeger
  UID:                 7cee03b1-3aa1-11e9-bbf5-06c840c1a2d6
Spec:
  Agent:
    Image:  jaegertracing/jaeger-agent:1.9
    Options:
    Resources:
    Strategy:
    Volume Mounts:  <nil>
    Volumes:        <nil>
  All In One:
    Image:
    Options:
    Resources:
    Volume Mounts:  <nil>
    Volumes:        <nil>
  Collector:
    Image:  jaegertracing/jaeger-collector:1.9
    Options:
    Resources:
    Size:           1
    Volume Mounts:  <nil>
    Volumes:        <nil>
  Ingress:
    Enabled:  false
    Resources:
    Security:
    Volume Mounts:  <nil>
    Volumes:        <nil>
  Query:
    Image:  jaegertracing/jaeger-query:1.9
    Options:
    Resources:
    Size:           1
    Volume Mounts:  <nil>
    Volumes:        <nil>
  Resources:
  Sampling:
    Options:
  Storage:
    Cassandra Create Schema:
      Datacenter:
      Enabled:     <nil>
      Image:
      Mode:
    Dependencies:
      Cassandra Client Auth Enabled:   false
      Cassandra Local Dc:
      Cassandra Use Ssl:               false
      Elasticsearch Client Node Only:  false
      Elasticsearch Nodes Wan Only:    false
      Enabled:                         true
      Image:                           jaegertracing/spark-dependencies
      Java Opts:
      Schedule:                        55 23 * * *
      Spark Master:
    Es Index Cleaner:
      Enabled:         true
      Image:           jaegertracing/jaeger-es-index-cleaner
      Number Of Days:  7
      Schedule:        55 23 * * *
    Options:
      Es . Server - Urls:  http://elastic-stack-elasticsearch-client.default.svc.cluster.local:9200
    Secret Name:
    Type:                  elasticsearch
  Strategy:                production
  Ui:
    Options:
  Volume Mounts:  <nil>
  Volumes:        <nil>
Status:
Events:  <none>

[jungopro]

@jpkrohling, one more _general_ question, if I may...

just to make sure my operator setup works, is there some demo app I can deploy that will generate data automatically to jaeger? something you recommend for a hello world scenario? I couldn't find one in the docs.
any recommendation or guidance?

thank you
Omer

[jpkrohling]

We have an example that we use to demonstrate the sidecar injection:

https://github.com/jaegertracing/jaeger-operator/blob/master/deploy/examples/business-application-injected-sidecar.yaml

You'll need to add a service and ingress route for it, though. Once you have it, every HTTP request should generate a few spans.