Cloudsql-proxy: Occasional errors connecting to the cloudsql proxy

[josephtyler]

In my cloudsql logs I frequently see errors like this:

E  2018/06/27 06:18:50 New connection for "myproject:region:db"
E  2018/06/27 06:18:51 Client closed local connection on 127.0.0.1:5432

And in my application logs I see this:

OperationalError at /url
could not connect to server: Connection refused
        Is the server running on host "127.0.0.1" and accepting
        TCP/IP connections on port 5432?

I'm not really sure how to go about troubleshooting this. This error doesn't happen consistently, seems to happen in bursts of 10-15 a few times a day.

[kurtisvg]

@josephtyler Is this still an issue for you? Can you provide any more information or an example of the code you are using to connect?

[rahmonov]

I am having the same issue since we migrated to CloudSQL. Here is more info:

@josephtyler were you able to fix it?

[hfwang]

If it's happening sporadically and the error is connection refused, I'd expect to see the cloud sql proxy getting restarted or something else happening to it...

@josephtyler
What environment are you running in?

When you say cloudsql logs, do you mean the logs for the cloud sql proxy or for the instance itself? Is there nothing else in the logs?

Also, just to double check: you are running the cloud sql proxy and binding it to 127.0.0.1:5432 right?

@rahmonov
Are you able to share any logs from the cloud sql proxy?

[fernandonogueira]

This is also happening here.
The proxy says it is ready to accept new connections but my application is throwing a timeout exception.

I followed the instructions from this page to connect my java/spring application to an Cloud SQL instance.

https://cloud.google.com/sql/docs/postgres/connect-kubernetes-engine

---

Differently of whats happening with @josephtylerl, looks like the cloudsql proxy is not getting restarted. I just cannot always connect to it. After sometime running, the connections start to work.

[fernandonogueira]

Hello guys!

In fact my problem was quite different.
I needed to add a sleep before starting my application that was running in the same pod.

After that, the proxy started receiving connections from my Spring application.

Thanks!

[kurtisvg]

Going to go ahead and close this - it's been over 30 days since it was first opened and there isn't enough information to verify any issues.

[rahmonov]

@kurtisvg @hfwang I think I found out why I am having this errors:

When a pod is terminated during rollout, the CloudSQL proxy container is terminated first and only after some seconds my application container is terminated. So, before my container is terminated, it keeps sending requests to the CloudSQL container which is already down, resulting in such errors.

If I could somehow specify the order of termination for the containers inside a single pod, this would solve this issue, I think.

I am trying to do something with container life cycle events, will write here if I solve it.

[kurtisvg]

@rahmonov You may be interested in #178 as well

[rahmonov]

@kurtisvg Thank you! Before it is merged and released, I am solving this by adding a preStop lifecycle event:

lifecycle:
  preStop:
    exec:
      command: ["/bin/sleep", "300"]

I am just having my cloudsql proxy container sleep until my application is down (which is 30s by default)

[markvincze]

Hi @josephtyler,

Did you manage to find the root cause and a solution for the original issue?
I'm seeing the exact same problem (New connection for... message in the Cloud SQL Proxy logs, and a Connection refused error in my application), and it's definitely not happening either during the startup nor the termination of a pod, but during normal operation.

[javapapo]

+1 on the above version 1.15

[pslobo]

I'm seeing the exact same situation as @markvincze. In a 30s load test, I'll see loads of New connection for... despite using connection pooling in the code. This, plus the added overhead of establishing new connections that Cloud Proxy introduces means that I can't even max out the low cpu/memory limits I have on my pods before starting to get status 500 from my services due to DB connection issues.

Edit: Forgot to mention currently using 1.17

[elmarti]

@pslobo I'm finding the exact same issue - excluding Istio traffic helped somewhat with this, but even with 40rps, i'm finding that my DB connection goes away. Ample mem and CPU for both the app and the database.

[pslobo]

@elmarti we eventually had to give up on using the proxy and establish direct connections using the private IP. Performance has increased significantly at the expense of not encrypting traffic via proxy