Jaeger-operator: Update of spec.ui has no effect on jaegerUI

[mwieczorek]

I have a running Jaeger defined with some spec.ui.options (f.e. linkPatterns).
If I update it (f.e. add another element to linkPatterns array) it has no effect on the UI (I expect a new link will be present but it's not).
I understand it's because jaeger-query has to be restarted to get new config.

I think the restart should be enforced.

[jpkrohling]

I think we are using a config map to store the UI options, and pods should be restarted once a config map it uses has changed.

I'm marking this as a bug, but I don't think I can look into it in the next couple of weeks. If anyone wants to take a shot on this one, go for it!

[mwieczorek]

I can take it.
Adding annotation with configmap hash to jaeger-query deployment (so the pods will be automatically redeployed) - would it be a proper solution?

[jpkrohling]

Could you try first manually updating the config map and watching the deployment? I had the impression that pods should be recreated automatically by Kubernetes once config maps are changed. There should be no need to track it on our side, AFAIK.

[mwieczorek]

I think Kubernetes doesn't recreate pods if the content of configmap is changed. I've done a test:

1. I Installed jaeger operator
2. I added such CR:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simplest

I run kubectl get pods -w to see all the changes.

1. I applied new version of CR:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simplest
spec:
  ui:
    options:
      linkPatterns:
      - type: "tags"
        key: "hostname"
        url: https://console.cloud.google.com/
        text: "Open GCP console"

New pod was created, old terminated (because in deployment definition there was a new volume mounted, and new parameter added to jaeger command - https://github.com/jaegertracing/jaeger-operator/blob/master/pkg/config/ui/ui.go#L79)

1. I applied the third version:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simplest
spec:
  ui:
    options:
      linkPatterns:
      - type: "tags"
        key: "hostname"
        url: https://console.cloud.google.com/
        text: "Open GCP console"
      - type: "tags"
        key: "version"
        url: https://github.com
        text: "Open release in github"

The configmap was changed, but the pod was not recreated. When I go to the UI and from dev tools console I see that it's still 1 element:

[jpkrohling]

Thanks for confirming! Are you able to check whether the mounted volume has the updated configmap? If so, we might want to add a reload to Jaeger itself. It would be far nicer than kill the pod

[mwieczorek]

I tried to exec to pod but I guess it doesn't have any shell even. I assume if cm is updated then also it's reflected in filesystem inside pod (I don't see reason why not...). So reload should do the job.

But: I think the configmap sync can take some time so it's hard for operator to know when to reload - https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#mounted-configmaps-are-updated-automatically

[jpkrohling]

The operator wouldn't even be aware that the config map has changed: it would be done by Jaeger itself. Viper has a hook to watch for changes to a local config file, we could probably use that.

[jpkrohling]

Looks like we aren't using Viper for that part:

https://github.com/jaegertracing/jaeger/blob/e567725c36d8490d74b7f3481ea201d6f57d906f/cmd/query/app/static_handler.go#L117-L141

[jpkrohling]

@mwieczorek I published a few images with this change, including jpkroehling/jaeger-query:Hot-reload-UI-config and jpkroehling/all-in-one:Hot-reload-UI-config. Would you like to try it out?

[mwieczorek]

@jpkrohling Sure, I'll check it today. Thanks

[mwieczorek]

@jpkrohling I made the same test as in one of previous comment (only this time defining your image in spec.allInOne.image).

And still after updating Jaeger CR with additional linkPatterns element I still cannot see new link in the browser (the configmap was changed, I waited a few minutes so cm change will be synced, I opened jaeger ui in incognito mode to avoid any browser caching, etc)

[jpkrohling]

Do you see a message like reloading UI config in Jaeger's console? I'll try that out here as well, based on your example.

[jpkrohling]

Apparently, the config map file is a symlink to a symlink to the actual config:

# pwd
/etc/config
# ls -lah
total 16K
drwxrwxrwx 3 root root 4.0K Jul 26 12:01 .
drwxr-xr-x 1 root root 4.0K Jul 26 11:55 ..
drwxr-xr-x 2 root root 4.0K Jul 26 12:01 ..2019_07_26_12_01_20.186918881
lrwxrwxrwx 1 root root   31 Jul 26 12:01 ..data -> ..2019_07_26_12_01_20.186918881
lrwxrwxrwx 1 root root   14 Jul 26 11:55 ui.json -> ..data/ui.json

Watching the ui.json would not yield any events, as the symlink isn't changing. Need to go back to the drawing board :-)

[jpkrohling]

I just updated the PR in the main repo and it seems it works now. Here's the CR I'm using for testing:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: hot-reload
spec:
  allInOne:
    image: jpkroehling/all-in-one:Hot-reload-UI-config-3
  ui:
    options:
      linkPatterns:
      - type: "tags"
        key: "hostname"
        url: https://console.cloud.google.com/
        text: "Open GCP console"
      - type: "tags"
        key: "version"
        url: https://github.com
        text: "Open release in github"

Once that is created, just apply the same CR again changing the UI config. The pod should remain up but the config map should be updated. In a few moments, a message like the following should be shown (multiple times, depending on the number of events Kubernetes causes):

{"level":"info","ts":1564653399.612611,"caller":"app/static_handler.go:154","msg":"reloading UI config","filename":"/etc/config/ui.json"}
{"level":"info","ts":1564653399.6128676,"caller":"app/static_handler.go:154","msg":"reloading UI config","filename":"/etc/config/ui.json"}
{"level":"info","ts":1564653399.6130342,"caller":"app/static_handler.go:154","msg":"reloading UI config","filename":"/etc/config/ui.json"}
{"level":"info","ts":1564653399.6131775,"caller":"app/static_handler.go:154","msg":"reloading UI config","filename":"/etc/config/ui.json"}
{"level":"info","ts":1564653399.6132812,"caller":"app/static_handler.go:154","msg":"reloading UI config","filename":"/etc/config/ui.json"}

It's also possible that the following message will appear, as the actual file might be missing during the events:

{"level":"warn","ts":1564653399.6134164,"caller":"app/static_handler.go:148","msg":"the UI config file has been removed, using the last known version"}

As long as there's a reloading UI message after that, it should be fine.

[jpkrohling]

@mwieczorek did you have a chance to test it? If you also confirm that it works, I'll get someone to review my PR at the main repo.

[mwieczorek]

@jpkrohling Works great for me. Thanks.

[mwieczorek]

@jpkrohling linked PR is merged - should we close it?

[jpkrohling]

Yes. I'm closing this with the note that it will be available in the next release (1.14.0).