Ambassador: Add watch on volumes for TLSContext

[cakuros]

Background/Summary

The TLSContext allows for specifying the cert-chain, the ca-cert-chain, and the key for TLS. It only does this, however, when it is first initialized and does not update when the volumes change. This is most evident with Istio's worker certificate rotation, where the /etc/istio-certs volume is updated, but the TLSContext is not.

Issue/Request

TLSContext needs to update whenever the volumes it's pointing at are changed.

Replicator

Kubernetes Cluster: Kubeception 1.17
Ambassador Version: 1.6.2 via Helm

1. Follow the Istio how-to (excluding Prometheus and Distributed Tracing), but in the environment variables of the istio-proxy container, add
   <ul> <li>name: SECRET_TTL<br /> value: "0h5m0s"<br />

1. wait 5 mins to see 2020-08-24T16:13:24.532920Z info sds resource:default pushed key/cert pair to proxy in kubectl logs -n ambassador {{AMBASSADOR_POD}} -c istio-proxy.

2. try to get to a backend service in the mesh:
   curl -v https://{{AMBASSADOR_HOST}}/backend/

< HTTP/1.1 503 Service Unavailable
< content-length: 91
< content-type: text/plain
< date: Mon, 24 Aug 2020 16:14:35 GMT
< server: envoy
< 
* Connection #0 to host {{AMBASSADOR_HOST}} left intact
upstream connect error or disconnect/reset before headers. reset reason: connection failure* Closing connection 0

1. Restart the pod kubectl rollout restart deploy -n ambassador ambassador

2. Try again curl -v https://{{AMBASSADOR_HOST}}/backend/

< HTTP/1.1 200 OK
< content-type: application/json
< date: Mon, 24 Aug 2020 17:05:58 GMT
< content-length: 168
< x-envoy-upstream-service-time: 4
< server: envoy
< x-envoy-decorator-operation: quote.default.svc.cluster.local:80/*
< 
{
    "server": "quintessential-passionfruit-t11jxcdt",
    "quote": "A principal idea is omnipresent, much like candy.",
    "time": "2020-08-24T17:05:58.616336259Z"
* Connection #0 to host {{AMBASSADOR_HOST}} left intact
}* Closing connection 0

See also #2692 and #2587

[nicolasbelanger]

FYI, so far, I've tested 1m and 5m rotation without downtime.

@cakuros About the 24h rotation and your recent slack comment:

Hi @nicolasbelanger, We did some testing by trying to accelerate the cert rotation to 1 hour, which appears to have worked (as described here: https://www.getambassador.io/docs/latest/howtos/istio/#how-to-test-istio-certificate-rotation), but are in the process of testing the 24 hour case which it's possible has a different behavior from what we were expecting.

why would the behaviour be any different?

[puneetloya]

I have pingdom alert set for the ambassador mapped endpoints.
I have set "SECRET_TTL: 1h", at the end of every hour it has downtime for about a minute. Its like the delta time before the tls context updates with the latest cert I think.

[anyname2]

I am facing the same issue. The certificate is renewed by Istio, but is not picked up by Ambassador.

• Kubernetes 1.18
• Istio 1.7.3
• Ambassador 1.8.1

I have tried setting SECRET_TTL to 5m and 24h (default), both fail until ambassador is restarted.

By default Istio rotates certificates at half-time, but at most once every 5 minutes (all configurable).

• With TTL of 24h the certificates are rotated every 12h but are valid for 24h. (app becomes unavailable after 24h)
• With TTL of 5m, certificates are rotated every 5min and are valid for 5 min. (app becomes unavailable after 5min)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.