Jaeger-operator: Override sampling strategy file in Jaeger Collector

[shubhanshus]

I am trying to use sampling strategy with my jaeger collector. I am able to pass the sampling file to the collector and it shows up in the args list. But, the problem is jager-collector always picks up the default (sampling.strategies-file=/etc/jaeger/sampling/sampling.json) file. I tried to modify the default file but seems like that's not possible.

   Args:
      --kafka.producer.brokers=my-cluster-kafka-brokers.kafka:9092
      --kafka.producer.topic=jaeger-spans
      --log-level=debug
      --sampling.strategies-file=/etc/config/my-sampling.json
      --sampling.strategies-file=/etc/jaeger/sampling/sampling.json

Is there any way to make jaeger-collector pick the correct file?

Below is my jaeger configuration:

{
   "apiVersion": "jaegertracing.io/v1",
   "kind": "Jaeger",
   "metadata": {
      "name": "jaeger",
      "namespace": "observability"
   },
   "spec": {
      "agent": {
         "options": {
            "log-level": "debug"
         },
         "strategy": "DaemonSet"
      },
      "collector": {
         "options": {
            "kafka": {
               "producer": {
                  "brokers": "my-cluster-kafka-brokers.kafka:9092",
                  "topic": "jaeger-spans"
               }
            }, 
            "log-level":"debug",
            "sampling.strategies-file": "/etc/config/my-sampling.json"
         },
         "volumeMounts": [
            {
               "name": "config-volume",
               "mountPath": "/etc/config"
            }
         ],
         "volumes": [
            {
               "name": "config-volume",
               "configMap": {
                  "name": "sampling-configmap",
                  "items": [
                     {
                        "key": "my-sampling.json",
                        "path": "keys"
                     }
                  ]
               }
            }
         ]
      },
      "ingester": {
         "options": {
            "ingester": {
               "deadlockInterval": 0
            },
            "kafka": {
               "consumer": {
                  "brokers": "my-cluster-kafka-brokers.kafka:9092",
                  "topic": "jaeger-spans"
               }
            }
         }
      },
      "storage": {
         "cassandraCreateSchema": {
            "datacenter": "cluster",
            "mode": "prod",
            "image": "jaegertracing/jaeger-cassandra-schema:1.14"
         },
         "options": {
            "cassandra": {
               "keyspace": "jaeger_v1_cluster",
               "servers": "10.249.85.166",
            }
         },
         "type": "cassandra"
      },
      "strategy": "streaming"
   }
}

[objectiser]

This looks like a bug - the operator appears to be adding the default path even though the CR is overriding it.

Have you tried specifying your sampling strategies directly in the CR? As described here: https://www.jaegertracing.io/docs/1.14/operator/#defining-sampling-strategies

[shubhanshus]

@objectiser thanks for your input. I tried that too. The Jaeger-collector picks the default sampling file in that case too.

I can see the sampling configuration present in my Jaeger configuration.

Collector:
    Image:  jaegertracing/jaeger-collector:1.13
    Options:
      kafka.producer.brokers:  my-cluster-kafka-brokers.kafka:9092
      kafka.producer.topic:    jaeger-spans
      Log - Level:             debug
    Replicas:                  1
    Resources:
    Size:           0
    Volume Mounts:  <nil>
    Volumes:        <nil>
  Ingester:
    Image:  jaegertracing/jaeger-ingester:1.13
    Options:
      ingester.deadlockInterval:  0
      kafka.consumer.brokers:     my-cluster-kafka-brokers.kafka:9092
      kafka.consumer.topic:       jaeger-spans
    Replicas:                     1
    Resources:
    Size:           0
    Volume Mounts:  <nil>
    Volumes:        <nil>
  Ingress:
    Enabled:  <nil>
    Resources:
    Security:       none
    Volume Mounts:  <nil>
    Volumes:        <nil>
  Query:
    Image:     jaegertracing/jaeger-query:1.13
    Options:   <nil>
    Replicas:  1
    Resources:
    Size:           0
    Volume Mounts:  <nil>
    Volumes:        <nil>
  Resources:
  Sampling:
    Options:
      default_strategy:
        Param:  50
        Type:   probabilistic

However as you can see the collector is still picking up the default strategy file.

jaeger-collector:
    Container ID:  docker://fabc1ad6ed209c92b46825541e6cff026b60642e87aae8381d97450f0ea08987
    Image:         jaegertracing/jaeger-collector:1.13
    Image ID:      docker-pullable://jaegertracing/jaeger-collector@sha256:d482afba0b8482d218e343e3450cb621b994d6bf48f1821005283875a9dea377
    Ports:         9411/TCP, 14267/TCP, 14268/TCP, 14269/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      --kafka.producer.brokers=my-cluster-kafka-brokers.kafka:9092
      --kafka.producer.topic=jaeger-spans
      --log-level=debug
      --sampling.strategies-file=/etc/jaeger/sampling/sampling.json
    State:          Running
      Started:      Tue, 15 Oct 2019 11:38:13 -0700
    Ready:          True
    Restart Count:  0
    Readiness:      http-get http://:14269/ delay=1s timeout=1s period=10s #success=1 #failure=3
    Environment:
      SPAN_STORAGE_TYPE:           kafka
      COLLECTOR_ZIPKIN_HTTP_PORT:  9411
    Mounts:
      /etc/jaeger/sampling from jaeger-sampling-configuration-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from jaeger-token-cx5b2 (ro)

[objectiser]

@shubhanshus Yes that is correct - however the sampling strategies defined in the Jaeger CR will have been written to the sampling file /etc/jaeger/sampling/sampling.json.

Have you tested to see whether the sampling strategy you defined is being used?

[shubhanshus]

Yes, I tried that. It works perfectly fine and collector picks up the correct strategy defined in Jaeger configuration. But, that doesn't seems to be a viable option when we have lots of services.

[objectiser]

But, that doesn't seems to be a viable option when we have lots of services.

@shubhanshus Could you provide more details of why this is an issue?

[shubhanshus]

@objectiser sure, so what we were thinking of is using the sampling strategies file and updating our sampling strategies based on the application load and the number of services we have. This was under the assumption the jager-sampling configmap will pickup the configuration whenever there is a change.

But, as per my understanding right now we have to redeploy the jager-configuration all the time whenever we want to make any changes to sampling strategy. This is the reason why I think this is not a viable option. Please correct me if my understanding is incorrect or if you think there is a better way to achieve this.

[objectiser]

@shubhanshus Thanks, that makes sense.

Would you be interested in contributing a PR to fix this? Essentially if the sampling.strategies-file option is defined, then any sampling strategy defined within the CR should just be ignored - possibly logging a warning.

[shubhanshus]

Sure, I can pick this up. Though might need some support from the community as this would be my first contribution.

[jpkrohling]

Check the CONTRIBUTING.md at the root of the repository. If you have any questions or if you get stuck, let us know.