Jaeger: Remote sampling strategy not working. agent-service sampling strategy enpoint returns errors.

[mdimec4]

Hi,
I am testing Jaeger and I must say it is a great tool which had already helped me to detect some bugs.

Until now I have been testing using "const" sampler in my service and everything works great. But after I switched to the default which is implicitly set to "remote", then it stopped working.

After reading the documentation, I figured out that agent has an endpoint which is responsible for sampling strategy.

After running tcpdump I figured that whenever client library (jaeger-client-go) queries this endpoint, it gets an error response.

The same can be observed with curl:

root@53279b97b86e:~/vss# curl http://jaeger-agent:5778/sampling?service=vss-engine
tcollector error: tchannel error ErrCodeBadRequest: no handler for service "jaeger-collector" and method "SamplingManager::getSamplingStrategy"

Why would this not work?

This is how I init. Jaeger in my service.

func initJaeger() (io.Closer, error) {
    if en, ok := os.LookupEnv("JAEGER_ENABLE"); !ok || (en != "true" && en != "yes") {
        return nil, nil
    }
    cfg := jaegercfg.Configuration{
        Sampler: &jaegercfg.SamplerConfig{},
        Reporter: &jaegercfg.ReporterConfig{
            LogSpans: true,
        },
    }

    if sType, ok := os.LookupEnv("JAEGER_SAMPLER_TYPE"); ok {
        cfg.Sampler.Type = sType
    }
    if sP, ok := os.LookupEnv("JAEGER_SAMPLER_PARAM"); ok {
        if sParam, err := strconv.ParseFloat(sP, 64); err != nil {
            Log.Errorf("problem parsing Jaeger sampling parameter '%s': %s", sP, err)
        } else {
            cfg.Sampler.Param = sParam
        }
    }
    if sServerURL, ok := os.LookupEnv("JAEGER_SAMPLING_SERVER_URL"); ok {
        cfg.Sampler.SamplingServerURL = sServerURL
    }
    if agentHostPort, ok := os.LookupEnv("JAEGER_AGENT_HOST_PORT"); ok {
        cfg.Reporter.LocalAgentHostPort = agentHostPort
    }

    jLogger := jaegerlog.StdLogger
    jMetricsFactory := metrics.NullFactory
    closer, err := cfg.InitGlobalTracer(
        "vss-engine",
        jaegercfg.Logger(jLogger),
        jaegercfg.Metrics(jMetricsFactory),
    )
    if err != nil {
        return nil, err
    }
    return closer, nil
}

Docker compose for my service:

version: "2"
services:
  vserver3:
    container_name: vss-master
    image: eu.gcr.io/visionect-gce-infra/visionect-server-v3:release_4.3.0
    privileged: true
    cap_add:
      - MKNOD
      - SYS_ADMIN
    devices:
      - "/dev/fuse:/dev/fuse"
    restart: always
    links:
      - "postgres_db:postgres"
      - "telegrafCollector:telegraf"
      - "redis:redis"
      - "jaeger-agent:jaeger-agent"
    ports:
      - 8081:8081 # admin browser
      - 11113:11113
      - 32991:32991 # GW RPC
      - 11115:11115 #NM
      - 32989:32989 #NM RPC
      - 5559:5559 # broker 1
      - 5560:5560 # broker 2
      - 10997:10997
      - 9444:9444
    environment:
      - DB2_1_PORT_5432_TCP_ADDR=postgres
      - DB2_1_PORT_5432_TCP_USER=visionect
      - DB2_1_PORT_5432_TCP_PASS=visionect
      - DB2_1_PORT_5432_TCP_DB=koala
      - VISIONECT_SERVER_MASTER_HOST
      - VISIONECT_SERVER_DEPLOYMENT_KEY
      - JAEGER_ENABLE=true
      - JAEGER_AGENT_HOST_PORT=jaeger-agent:5775
      - JAEGER_SAMPLING_SERVER_URL=http://jaeger-agent:5778/sampling
    volumes:
      - /dev/shm:/dev/shm
  postgres_db:
    container_name: pdb
    image: postgres:latest
    restart: always
    ports:
      - 5432:5432
    environment:
      - POSTGRES_USER=visionect
      - POSTGRES_DB=koala
      - POSTGRES_PASSWORD=visionect
      - PGPASSWORD=visionect
    volumes:
      - ./pgdata:/var/lib/postgresql/data
  jaeger-agent: # taken from https://github.com/jaegertracing/jaeger/blob/master/docker-compose/jaeger-docker-compose.yml
    image: jaegertracing/jaeger-agent
    command: ["/go/bin/agent-linux", "--collector.host-port=jaeger-collector-test.vnct.xyz:14267"]
    ports:
      - "5775:5775/udp"
      - "6831:6831/udp"
      - "6832:6832/udp"
      - "5778:5778"
    restart: always

Jaeger services are deployed to kubernetes at google with cassandra storage (based on oficial Jaeger k8s template).

#
# Copyright 2017-2018 The Jaeger Authors
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
# in compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under the License
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
# or implied. See the License for the specific language governing permissions and limitations under
# the License.
#

apiVersion: v1
kind: List
items:
- apiVersion: extensions/v1beta1
  kind: Deployment
  metadata:
    name: jaeger-collector
    labels:
      app: jaeger
      jaeger-infra: collector-deployment
  spec:
    replicas: 1
    strategy:
      type: Recreate
    template:
      metadata:
        labels:
          app: jaeger
          jaeger-infra: collector-pod
      spec:
        containers:
        - image: jaegertracing/jaeger-collector:1.2
          name: jaeger-collector
          command:
            - "/go/bin/collector-linux"
            - "--config-file=/conf/collector.yaml"
          resources: # (miha) added resources
            requests:
              memory: 512Mi 
              cpu: 0.5
            limits:
              memory: 1Gi
              cpu: 1
          ports:
          - containerPort: 14267
            protocol: TCP
          - containerPort: 14268
            protocol: TCP
          - containerPort: 9411
            protocol: TCP
          volumeMounts:
          - name: jaeger-configuration-volume
            mountPath: /conf
          env:
          - name: SPAN_STORAGE_TYPE
            valueFrom:
              configMapKeyRef:
                name: jaeger-configuration
                key: span-storage-type
        volumes:
          - configMap:
              name: jaeger-configuration
              items:
                - key: collector
                  path: collector.yaml
            name: jaeger-configuration-volume
- apiVersion: v1
  kind: Service
  metadata:
    name: jaeger-collector
    labels:
      app: jaeger
      jaeger-infra: collector-service
    annotations: #(miha) request for internal IP
      cloud.google.com/load-balancer-type: "Internal"
  spec:
    ports:
    - name: jaeger-collector-tchannel
      port: 14267
      protocol: TCP
      targetPort: 14267
    - name: jaeger-collector-http
      port: 14268
      protocol: TCP
      targetPort: 14268
    - name: jaeger-collector-zipkin
      port: 9411
      protocol: TCP
      targetPort: 9411
    selector:
      jaeger-infra: collector-pod
    type: LoadBalancer #ClusterIP (miha)
    loadBalancerIP: 10.4.0.5 # (miha) resrved internal IP https://kubernetes.io/docs/concepts/services-networking/service/
- apiVersion: v1
  kind: Service
  metadata:
    name: zipkin
    labels:
      app: jaeger
      jaeger-infra: zipkin-service
    annotations: #(miha) request for internal IP
     cloud.google.com/load-balancer-type: "Internal"
  spec:
    ports:
    - name: jaeger-collector-zipkin
      port: 9411
      protocol: TCP
      targetPort: 9411
    selector:
      jaeger-infra: collector-pod
    type: LoadBalancer #ClusterIP (miha)
    loadBalancerIP: 10.4.0.6 # (miha) resrved internal IP https://kubernetes.io/docs/concepts/services-networking/service/
- apiVersion: extensions/v1beta1
  kind: Deployment
  metadata:
    name: jaeger-query
    labels:
      app: jaeger
      jaeger-infra: query-deployment
  spec:
    replicas: 1
    strategy:
      type: Recreate
    template:
      metadata:
        labels:
          app: jaeger
          jaeger-infra: query-pod
      spec:
        containers:
        - image: jaegertracing/jaeger-query:1.2
          name: jaeger-query
          command:
            - "/go/bin/query-linux"
            - "--config-file=/conf/query.yaml"
          resources: # (miha) added resources
            requests:
              memory: 128Mi 
              cpu: 256m
            limits:
              memory: 512Mi
              cpu: 0.5
          ports:
          - containerPort: 16686
            protocol: TCP
          readinessProbe:
            httpGet:
              path: "/"
              port: 16686
          volumeMounts:
          - name: jaeger-configuration-volume
            mountPath: /conf
          env:
          - name: SPAN_STORAGE_TYPE
            valueFrom:
              configMapKeyRef:
                name: jaeger-configuration
                key: span-storage-type
        volumes:
          - configMap:
              name: jaeger-configuration
              items:
                - key: query
                  path: query.yaml
            name: jaeger-configuration-volume
- apiVersion: v1
  kind: Service
  metadata:
    name: jaeger-query
    labels:
      app: jaeger
      jaeger-infra: query-service
    annotations: #(miha) request for internal IP
      cloud.google.com/load-balancer-type: "Internal"
  spec:
    ports:
    - name: jaeger-query
      port: 80
      protocol: TCP
      targetPort: 16686
    selector:
      jaeger-infra: query-pod
    type: LoadBalancer
    loadBalancerIP: 10.4.0.7 # (miha) resrved internal IP https://kubernetes.io/docs/concepts/services-networking/service/

[jpkrohling]

After reading the documentation, I figured out that agent has an endpoint which is responsible for sampling strategy.

Where did you see this in the documentation? We might need to fix it. Unless I missed something, this is not implemented yet, but it's on the roadmap.

[jpkrohling]

I might indeed have missed something: https://github.com/jaegertracing/jaeger/pull/720

[mdimec4]

Well, "remote" is the default for go library.

Documentation says https://jaeger.readthedocs.io/en/latest/client_libraries/

Remote (sampler.type=remote, which is also the default) sampler consults Jaeger agent...

and https://jaeger.readthedocs.io/en/latest/deployment/ has a table which says

5778 | HTTP | serve configs, sampling strategies

If it is supported, then maybe there is some problem with my setup.
If it is not implemented yet, then I found it strange that this is set as the default strategy for client libraries.

[mdimec4]

So if I understend correctly, I can now provide agent service with strategy file if I want to, but no Adaptive sampling
Looking at link you have provided and documentation I see that "Adaptive Sampling" is not yet implemented. My bad for assuming It is.

It is somehow confusing given that libs. default behaivor is "remote" sampler, which will not work if I use official k8s template.

[black-adder]

Hi there, we only started supporting remote sampling in jaeger 1.3. I added some documentation for collector sampling configuration just yesterday: https://jaegertracing.netlify.com/docs/sampling/#collector-sampling-configuration

Sorry for the confusion!

[mdimec4]

Oh I see thank you