Cloudsql-proxy: Readiness and liveness probes in Kubernetes

[wayneashleyberry]

I'm struggling to setup a health check for the proxy container on Kubernetes. Some documentation or guidance around this would be great. We're having to delete pods quite regularly due to an app not being able to connect.

Tried defining a tcp liveness probe which didn't quite do the trick.

[AthenaShi]

So far we don't expose such thing unfortunately...

As a work around you can check by run cmd:
netcat -U /cloudsql/$INSTANCE_CONNECTION_STRING

As for the "an app not being able to connect" issue, if you think it might due to cloud sql proxy, please send an email with following information to [email protected] so I may be able to help out looking at some internal data.

• with a link to this page.
• describe the issue you have.
• the full name of your cloud sql instance.
• the command your run
• the full logs

[rosskevin]

I misunderstood netcat's location and options on the image.

I attempted the following:

livenessProbe:
  exec:
    command: ["nc", "/cloudsql/foo:us-central1:db=tcp:3306"]
  initialDelaySeconds: 5
  timeoutSeconds: 3

and it failed with:

Liveness probe failed: nc: bad address '/cloudsql/foo:us-central1:db=tcp:3306'

Has anyone else gotten nc to work as a probe?

[AthenaShi]

Would you please try:

livenessProbe:
  exec:
    command: ["nc", "127.0.0.1", "3306"]
  ...

The command I pasted before (netcat -U /cloudsql/$INSTANCE_CONNECTION_STRING) was for unix domain socket.

[rosskevin]

Thanks @AthenaShi, that does report success.

[AthenaShi]

Seems like there's no problem left for this thread. Close it.

[hews]

FYI I actually needed to run it slightly differently; that is I needed to run nc in zero-I/O (scan) mode:

livenessProbe:
  exec:
    command: ["nc", "-z", "127.0.0.1", "3306"]
  ...

[thijsterlouw]

The healthcheck using netcat in zero-I/O mode works, but has the annoying side-effect of spamming our stackdriver logs with errors:

New connection for "xxxx"
Client closed local connection on 127.0.0.1:3306

This can probably be turned off by adding the "-verbose=false" flag: https://github.com/GoogleCloudPlatform/cloudsql-proxy/commit/14a1606a343e46f58fb8a86fb8b575f492e09020
https://cloud.google.com/sql/docs/mysql/sql-proxy#reducing-output

[ahalberkamp]

Hi,

I have added the Liveprobe using netcat is basically working for me, but It is also spamming my logs with this message:

Writing data to local connection on 127.0.0.1:3306 had error: write tcp 127.0.0.1:3306-127.0.0.1:45075: write: broken pipe

Can anyone give me a hint how I could solve this problem?