Linkerd2: CronJob does not terminate

[dabeck]

Bug Report

What is the issue?

After enabling linkerd automatic injection on namespaces having a cronJob the do not terminate anymore.

How can it be reproduced?

Enable automatic proxy injection on a namespace with cronjobs. The cronjob container itself terminates while the linkerd containers keeps the pod alive.

Environment

• Kubernetes Version: 1.13.5
• Cluster Environment: RKE (rancher kubernetes environment)
• Host OS: Container Linux
• Linkerd version: 2.3.0

[klingerf]

@dabeck Thanks for reporting this. It sounds like the same issue as #1869, and we don't have an immediate fix, unfortunately. There's work underway in Kubernetes (kubernetes/kubernetes#25908 and kubernetes/enhancements#753) to designate the proxy as a sidecar container, which would then exit when the main job container finishes, but that isn't available yet. In the meantime I recommend adding the linkerd.io/inject: disabled annotation on the job's pod spec, so that that pod won't be injected with the proxy. There's more info about that here: https://linkerd.io/2/tasks/automating-injection/#configuration

[dabeck]

@klingerf ah thanks for your response. I haven't seen the other issue. Sorry about that.
I did indeed set the linkerd.io/inject: disabled annotation unfortunately this did not stop linkerd from injecting the proxy into my cronJobs.

[klingerf]

@dabeck Yikes, that sounds like a bug. Can you share your cronjob config? Or at least an approximate version of the config that you're using? We'll see if we can reproduce this.

[dabeck]

I'm sorry, I should have read better. My annotation was at the cronjob template level while it actually has to live at the pod spec level. 🙈

[klingerf]

Ahh, yep, that's a common source of confusion. The auto-injector watches pods directly, rather than their parent objects, so it needs to be on the pod spec itself.

[dabeck]

I'm closing this as duplicate of #1869