cloudsql-proxy gets into bad state if error occurs while connecting

[kgreenek]

I have a systemd service that simply runs cloud_sql_proxy. If the network isn't available when the service is started, I see the following messages:

2019/06/12 10:10:27 Rlimits for file descriptors set to {&{8500 1048576}}
2019/06/12 10:10:40 errors parsing config:
    Get [... project url ommitted ...]: dial tcp: lookup www.googleapis.com on 127.0.1.1:53: server misbehaving
2019/06/12 10:10:40 Ready for new connections

I have the cloud_sql_proxy configured to accept local connections via a unix socket file, like so:

cloud_sql_proxy -dir="/tmp/cloudsql" -instances="${OUR_DB_INSTANCE}"

After the series of events above, the cloud_sql_proxy is in an unrecoverable state. It doesn't retry to connect, and the unix socket file doesn't exist, so my client application can't connect to it.

Expected behaviour:
The cloud_sql_proxy should either exit with an error code (so systemd can restart it), or it should retry when there is an error connecting.

My work-around for now is to create a "watchdog" service that restarts the cloud_sql_proxy service if the unix socket directory is empty after more than 10 seconds, but this is obviously a hack.

EDIT:
The socket file does actually exist, but it refuses connections (so my original hack didn't work) when cloud_sql_proxy is in this state. I get the following python exception when I try to connect to it:

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not connect to server: Connection refused

[bajb]

We are also experiencing a similar issue, however we have multiple instances we are connecting to via the "-instances_metadata=project/attributes/cloud-sql-proxy-instances" flag.

Our error log will look something like this:

11:18:34 using credential file for authentication; [email protected]
11:18:34 Ready for new connections
11:19:05 errors parsing config:
    Get https://www.googleapis.com/sql/v1beta4/projects/project-id/instances/instance-5?alt=json&prettyPrint=false: oauth2: cannot fetch token: Post https://oauth2.googleapis.com/token: net/http: TLS 
    handshake timeout
    Get https://www.googleapis.com/sql/v1beta4/projects/project-id/instances/instance-6?alt=json&prettyPrint=false: oauth2: cannot fetch token: Post https://oauth2.googleapis.com/token: 
    net/http: TLS handshake timeout
11:19:05 Listening on /tmp/cloudsqlproxy/project-id:us-central1:instance-1 for project-id:us-central1:instance-1
11:19:05 Listening on /tmp/cloudsqlproxy/project-id:us-central1:instance-2 for project-id:us-central1:instance-2
11:19:05 Listening on /tmp/cloudsqlproxy/project-id:us-central1:instance-3 for project-id:us-central1:instance-3
11:19:05 Listening on /tmp/cloudsqlproxy/project-id:us-central1:instance-4 for project-id:us-central1:instance-4

While SQL Proxy starts up, and looks like its running perfectly fine. if we attempt to connect to instance 5 or 6, the socket is unavailable, and we are unable to connect to those instances.

Restarting the pod is the only option at this point.

It would be great to have a flag to fail or retry on setup errors, so we know if the sql proxy is running, its connected to all the available instances.

[terev]

@kurtisvg https://github.com/GoogleCloudPlatform/cloudsql-proxy/blob/master/cmd/cloud_sql_proxy/proxy.go#L337 seems like this line should actually return the error instead of logging then ignoring it . that way the container can restart based on conditions defined by the operator

[kurtisvg]

This looks correct to me - it looks like it should bubble up to this log.fatal statement to terminate.

@terev - Do you want to submit a PR for this?

I think there are some errors that we should retry on instead of exit (such as hitting the API quota) but we can narrow down that behavior in a future PR.

[bajb]

I think you also need to account for an error response while looping instances:
cmd/cloud_sql_proxy/proxy.go:68

https://github.com/GoogleCloudPlatform/cloudsql-proxy/pull/311/files

Not sure if skipping the update or failing is the best option here. Assuming we could retry again after a period of time, keeping everything running would be the best option in our case.