Cloudsql-proxy: Feature Request: Add health checks to proxy container

[arschles]

I am starting up the cloudsql proxy as a sidecar to my app, and I notice that my app fails non-deterministically. After some trial and error, it appears that the proxy may take some time to start listening on its port(s). I believe that a Kubernetes healthcheck would solve this issue. Does the proxy support healthchecks?

[AthenaShi]

Please check this for healthcheck using Kubernetes:

https://github.com/GoogleCloudPlatform/cloudsql-proxy/issues/119

[arschles]

Hi @AthenaShi - thanks for the reply and reference. Would you be open to a PR to add an HTTP endpoint that adds an endpoint that can be used as a health check?

[AthenaShi]

Ya, sure!

[ahmetb]

We might incorporate this in the official sample manifest.

In the meanwhile can you give this livenessProbe with exec a try: https://github.com/GoogleCloudPlatform/cloudsql-proxy/issues/119#issuecomment-338749457 and see if it fixes your intermittent issues?

[abstrctn]

Thanks @ahmetb, the livenessProbe you linked to is helpful for ensuring the proxy is listening on a given port.

However, I wonder if a "healthy" state should also involve testing the connection to the instance. A few cases where the netcat test would succeed, but an application could not actually connect:

• A typo in the -instances argument pointing at a nonexistant instance name
• The database instance going down for any reason
• The service account used by the proxy losing IAM permissions to access the instance

I'm not sure whether those are appropriate to test continually while running the proxy, but I know we'd find use from testing it within a readinessProbe, to ensure any edits to the configuration don't break connectivity. (This is especially useful if running the proxy as a service, when you don't have application code and user credentials to test the connection.)

[ahmetb]

livenessProbe you linked to is helpful for ensuring the proxy is listening

you can probably ship the mysql CLI with the image (or write a small script alongside your container) that makes a simple ping query to the database and exits. You can still use "exec" livenessProbe for this purpose and it's a somewhat common practice to test liveness of non-HTTP systems via Kubernetes.

[caquinovs]

@arschles out of curiosity, when you proxy may take some time to start listening on its ports, how much time have you observed so far? I'm starting to do some troubleshooting on my own pods, and I still have yet to confirm it, but cloudsql proxy logging indicates that it could be hours after container start.

[bcbrockway]

We were using https://github.com/GoogleCloudPlatform/cloudsql-proxy/issues/119#issuecomment-338749457 as our healthcheck but nc has been removed from the v1.16 release. What's the official recommendation now? Seems a bit crazy that we should have to create our own image to add livenessProbe support when using gce-proxy as a sidecar is a well-documented use-case.

[kurtisvg]

Update for anyone that's relying on a shell, we do have alpine and buster images with shells still included.

[kurtisvg]

Some additional thoughts on a path to execution for this:

The original issue mentions k8s probe, of which there are 3 different flavors:

1. Startup probes - indicates the container has completed start up and is ready for traffic
2. Readiness probes - indicates that the container is able to handle new traffic
3. Liveness probes - indicates the container is still in a valid state

This is probably relevant because it'll change how we want to judge "health". Are we "healthy" if the proxy is successfully receiving/acting on connections, or just if it's successfully reaching the database? What about for multiple instances?

Then it looks like there are 3 ways you can implement each probes:

1. Command - run a command in the container and validate from exit code
2. TCP - open a connection on the specified port, validate if connection was successful.
3. HTTP - send a HTTP request and validate from response

TCP probably isn't helpful since it only validates that the proxy is listening to connections, but we could potentially implement one (or both) of the other two:

• Command: could be implemented with a flag that tells the proxy to report it's status in a specific way (maybe a file or HTTP mentioned below) and then a second flag that can evaluate based on the same commands
• HTTP: would require exposing an additional port for an HTTP server to respond to a simple GET request