Loki: Could not load jaeger tracer configuration

[voslartomas]

Describe the bug
Installed loki-stack, everything seems fine, but if we install jaeger afterwards and restart loki pod, it will say: cannot parse env var JAEGER_AGENT_PORT=udp://100.67.30.30:5775

To Reproduce
Steps to reproduce the behavior:

1. Install loki-stack via helm chart
2. Install jaeger via helm chart
3. Delete loki pod (it will be in crash)

Expected behavior
I would expect to be able to run loki and jaeger alongside each other.

Environment:

• Infrastructure: Kubernetes
• Deployment tool: Helm

Screenshots, promtail config, or terminal output

Could not load jaeger tracer configuration: cannot obtain reporter config from env: 
cannot parse env var JAEGER_AGENT_PORT=udp://100.67.30.30:5775: 
strconv.ParseInt: parsing "udp://100.67.30.30:5775": invalid syntax

[voslartomas]

Possibly it is, but I don't get why it is happening only on loki pods.

[cyriltovena]

I don't see any env vars in the helm chart did you add this one yourself ?

JAEGER_AGENT_PORT=udp://100.67.30.30:5775

Reading Loki main.go you need JAEGER_AGENT_HOST to activate tracing.

[voslartomas]

Nope, this is my settings for loki.

loki:
  persistence:
    enabled: true
    storageClassName:
    # existingClaim: loki
  resources:
    limits:
      cpu: 400m
      memory: 2048Mi
    requests:
      cpu: 200m
      memory: 1024Mi

Where did you find that main.go file, please?

[cyriltovena]

https://github.com/grafana/loki/blob/master/cmd/loki/main.go#L41

[daixiang0]

@voslartomas hi, please try refer to this PR setting, wait for your feedback.

[voslartomas]

@daixiang0 Hello, thanks for PR. Not sure how to test it out while using terraform, because grafana has this chart URL https://grafana.github.io/loki/charts, any way to use your branch in this way?

[steven-sheehy]

@voslartomas You can checkout @daixiang0's branch and run make helm to build the chart locally.

[acim]

This is not a problem of this chart, it is actually dependency https://github.com/jaegertracing/jaeger-client-go that fails:

https://github.com/jaegertracing/jaeger-client-go/blob/c58709fe19d793cdd40ad8bfa6e645a40a2eccf8/config/config_env.go#L186

If you install jaeger chart with name jaeger, environment variable JAEGER_AGENT_PORT will be automatically created by Kubernetes (as string) and jaeger-client-go expects integer (just the port number, not url) under the same name so it is going to fail. The easiest way to fix the problem would be to install jaeger under some different name, for example:

helm install incubator/jaeger --name jaeger-tracing

[weeco]

@acim I did not use the jaeger helm chart but I face the same problem:

Could not load jaeger tracer configuration: cannot obtain reporter config from env: cannot parse env var JAEGER_AGENT_PORT=udp://10.3.241.148:5775: strconv.ParseInt: parsing "udp://10.3.241.148:5775": invalid syntax

I do not specify the environment variable JAEGER_AGENT_PORT and describing the pod which throws this error has no such environment variable set either. I only specify these JAEGER_ variables:

      JAEGER_AGENT_HOST:                (v1:status.hostIP)
      JAEGER_SAMPLER_TYPE:             ratelimiting
      JAEGER_SAMPLER_PARAM:            7

Can you explain why Kubernetes would automatically set the JAEGER_AGENT_PORT?

[acim]

Try to change the name of the service from jaeger to something else, let's say jaeger-tracing. You should probably change deployment name as well. And here is about Kubernetes variables:

https://kubernetes.io/docs/concepts/containers/container-environment-variables/

You may also want to check this:

https://github.com/kubernetes/kubernetes/issues/60099

[weeco]

@acim Wow I did not know this. In my case I had a service called jaeger-agent and apparently this implicitly sets the environment variables JAEGER_AGENT_PORT

A list of all services that were running when a Container was created is available to that Container as environment variables. Those environment variables match the syntax of Docker links.

Thanks a lot.