Ambassador: Watch ambassador-certs and ambassador-cacert secrets to avoid need to restart Ambassador

[adrienjt]

When setting up TLS termination following https://www.getambassador.io/user-guide/tls-termination, it took me a while to realize that I needed to restart Ambassador for it to detect the ambassador-certs secret I had created. The same way ambassador/kubewatch.py watches services for annotation changes, it could watch the ambassador-certs secret (and the ambassador-cacert secret while we're at it).

The watch loops would run concurrently. The official Kubernetes client isn't asyncio-friendly at the moment https://github.com/kubernetes-client/python/issues?utf8=%E2%9C%93&q=asyncio, but concurrent.futures with threads or raw threading would do the trick. If time allows, this multi-watch logic could be implemented upstream in the Kubernetes client: https://github.com/kubernetes-client/python/issues/30

In the meantime, a bold warning in the documentation would be helpful.

[richarddli]

Thanks for the bug report. This makes sense. If you're able to do a documentation PR on the TLS guide, that would be greatly appreciated.

[kevinohara80]

I'm curious if this is something that's being looked at as a roadmap item? We're trying to avoid having to build tooling around this or switch gateways. LetsEncrypt (via cert-manager) is going to rotate our cert every 90 days and automatically update ambassador-certs.

[kflynn]

@kevinohara80 Very much so! We've actually been sorting out the path forward on Ambassador's roadmap just recently, so I expect to have a better sense of exactly when this will get handled shortly.

[concaf]

@KowalczykBartek volunteered to work on this earlier today :tada:

[kflynn]

Done in RC6.