Consul-helm: [injector] consul-connect-injector-webhook fails the injection after restarted

[enuoCM]

Pod consul-connect-injector-webhook and a sample Pod A(with connect-inject true, has there containers: Service A, envoy-sidecar and consul-sidecar) both ran well on node A at beginning. For some reason, node A was down and never came back(all status of pod changed to Terminating), then consul-connect-injector-webhook and A were redeployed on node B. But the new Pod A had no connect injected, it had no sidecars.
Is this problem abnormal? How to keep A always be injected automatically? Thanks.
BTW, if delete Pod A again, new launched Pod A will be injected.

[whiskeysierra]

Most likely a race condition of the injector is not yet running.

[kschoche]

Hi @enuoCM ! Thanks for filling this issue!
Without additional logs/info my initial guess is that when your node went down which also was running the webhook, kubernetes redeployed your pod's deployment before redeploying the webhook deployment. Unfortunately this is a race condition which could happen.

We are currently working on a new feature for the next major release which will also bring HA to the webhook, so I suspect that if the above is the actual cause of your problem that it should be fixed by this.

[whiskeysierra]

That's going to be tricky with the webhook issuing self signed certificates, right?

[kschoche]

That's going to be tricky with the webhook issuing self signed certificates, right?

Yup, good pointl! Without going into too much detail in this thread, we will use the webhook-cert-manager for this.

[enuoCM]

@kschoche Thanks. Race condition should be the reason. Is it true that we can only deploy one consul-connect-injector-webhook with latest "0.30.0" version?

[ishustava]

Hey @enuoCM, yes currently you can have only a single instance of the webhook.

[enuoCM]

Hi @ishustava, is there any way to deploy more webhooks currently?

[ishustava]

Not at the moment, unfortunately, but we are working on fixing that right now, so hopefully you will be able to soon!

[whiskeysierra]

Shouldn't that be possible if one would use its own certificate, rather than letting the webhook generate its own?

[kschoche]

In addition to the cert issues, the health check controller and the cleanup controller would also need work to enable more than one copy of the webhook to run. These are also being addressed by the work I mentioned above.

[enuoCM]

Shouldn't that be possible if one would use its own certificate, rather than letting the webhook generate its own?

Hi, do you mean by following configuration in values.yaml?

   # The certs section configures how the webhook TLS certs are configured.
   # These are the TLS certs for the Kube apiserver communicating to the
   # webhook. By default, the injector will generate and manage its own certs,
   # but this requires the ability for the injector to update its own
   # MutatingWebhookConfiguration. In a production environment, custom certs
   # should probably be used. Configure the values below to enable this.
  certs:
    # Name of the secret that has the TLS certificate and
    # private key to serve the injector webhook. If this is null, then the
    # injector will default to its automatic management mode that will assign
    # a service account to the injector to generate its own certificates.
    secretName: null

    # Base64-encoded PEM-encoded certificate bundle for the
    # CA that signed the TLS certificate that the webhook serves. This must
    # be set if secretName is non-null.
    caBundle: ""

    # Name of the file within the secret for
    # the TLS cert.
    certName: tls.crt

    # Name of the file within the secret for
    # the private TLS key.
    keyName: tls.key

[thisisnotashwin]

Hey @enuoCM ,

As @kschoche and @ishustava have mentioned, at the moment, even setting those values for now will not strictly help with the above solution as certain components of the connect-injector webhook are not designed for HA. We are working on updating that behavior so that in the future, you will be able to run multiple instances of the same.
The values that you have highlighted will be deprecated then.
Unfortunately for now, the recommended solution would be to delete Pod A and redeploy it. This will get reconnected into the service mesh once that happens

[david-yu]

Closing as the connect webhook injector is now HA as of Consul Helm 0.32.0.