Jaeger-operator: Noticing weird kubernetes pods being deployed and terminated immediately

[btai24]

When I use the annotation "sidecar.jaegertracing.io/inject": "true" , there are extra pods that get created that terminate right away. IN my example below, the two api pods that get created named api-5955f589d5-fkpfg and api-5955f589d5-rhhm6 are created and immediately terminated. the correct replicaset that ends up being persisted and deployed are the first 3 api pods (with ages 1s and 2s). Anybody run into this before?

$ k get po NAME READY STATUS RESTARTS AGE api-57ffc5c59b-bq2vb 0/2 ContainerCreating 0 1s api-57ffc5c59b-hvr9l 0/2 ContainerCreating 0 1s api-57ffc5c59b-jv5zx 0/2 ContainerCreating 0 2s api-5955f589d5-8fjml 2/2 Running 0 74m api-5955f589d5-fkpfg 1/2 Terminating 0 9s api-5955f589d5-rhhm6 1/2 Running 0 9s client-78d7747758-b8m2b 1/1 Running 0 74m client-78d7747758-wrmrp 0/1 Terminating 0 9s client-7c74cd6d89-ptn2z 0/1 ContainerCreating 0 1s client-7c74cd6d89-q2lxg 0/1 ContainerCreating 0 2s

[jpkrohling]

This is "expected". We inject sidecars by watching existing pods and checking its annotations/labels. The side effect is that we only get to see deployments that exist already. We'll probably move to using an admission webhook (#171), which would address the behavior you described here.

Other than the "annoying" side of this, is this causing problems to you?

[btai24]

@jpkrohling no problems being caused, that makes sense. I was reading the logs and it said as much but I just wanted to make sure it was expected behavior.

[btai24]

does this look normal, I feel like I should only have 3 Terminating pods because the replica set only is a replica set of 3:

NAME                             READY   STATUS             RESTARTS   AGE
api-5f45849c89-5wc6f             1/2     Terminating        0          30s
api-5f45849c89-7bg42             1/2     Terminating        0          30s
api-5f45849c89-cqnmt             1/2     Running            0          24s
api-5f45849c89-mf6zh             1/2     Running            0          24s
api-5f45849c89-sfcq8             1/2     Running            0          24s
api-5f45849c89-tnrvf             2/2     Terminating        1          30s
api-6c4478985d-lgnv2             0/1     Terminating        0          25s
api-6c4478985d-x72k4             0/1     Terminating        0          25s
api-76656c48bc-66h8x             2/2     Running            0          79m
api-76656c48bc-7wd54             2/2     Running            0          79m
api-76656c48bc-g422s             2/2     Terminating        0          79m

[jpkrohling]

This does not look normal. I would also expect to have only one update to the replicaset, causing 6 pods to be started (3 old, 3 new). You have 11, which looks strange, as I would expect multiples of 3.

cc @rubenvp8510 could you take a look at this one?
cc @objectiser it might be time for us to start working on admission webhooks, as I think the Operator SDK/kubebuilder has support for it now.

[btai24]

@jpkrohling thanks for the quick response, something that I was thinking about recently w/ this approach was that my api pods actually consume a lot of cpu on startup. I'm curious if those pods while they are only up for the few seconds before they start terminating are contributing to the cpu pressure I have during deployments on my worker nodes. Has this been brought up before?

[jpkrohling]

This hasn't been brought before, but I can see that this might indeed be the case, especially if there's not enough CPU available to start the new pods, so that the old ones get terminated only once they stabilize and the CPU usage goes down.

As a temporary workaround, you might want to add an init container that just waits a couple of seconds. This way, your "old" pod has a higher change of being killed before your API container starts.

As a status: we are moving to the Operator SDK 0.19.x in the near future (next weeks), and we should be moving to the admission webhooks approach right after that, if everything goes well. Because of that, I'll only look into a permanent solution to this ticket if we decide to revert from that decision.

Would you be willing to test the new approach once I get a PR ready?

[btai24]

Yes of course, we can test it when its ready

[jpkrohling]

@rubenvp8510 could you check whether this is still happening?

[jpkrohling]

By the way: we are not moving to the Operator SDK 0.19.x soon, as it's a really disruptive update.

[jpkrohling]

@btai24 would you like to give the linked PR a try?

[TBBle]

To make this issue more-searchable, one symptom of this issue is that the auto-generated --jaeger.tags argument from #648 will be removed from a deployment a few seconds after it is added. That's actually what causes the pods to be created and immediately terminated.

Should be fixed in Jaeger Operator 1.20.0