Cloudsql-proxy: `connection reset by peer` periodically

[n-sviridenko]

Hi. I'm having this issue. Using cloudsql-proxy. The kubernetes deployment config:

...
        - name: cloudsql-proxy
          image: gcr.io/cloudsql-docker/gce-proxy:1.14
          command:
            - "/cloud_sql_proxy"
            - "-instances={{ .Values.cloudSql.instanceConnectionName }}=tcp:3306"
            - "-credential_file=/secrets/cloudsql/gcp-cloud-sql-client.json"
          securityContext:
            runAsUser: 2  # non-root user
            allowPrivilegeEscalation: false
          volumeMounts:
            - name: cloudsql-instance-credentials
              mountPath: /secrets/cloudsql
              readOnly: true
      volumes:
        - name: cloudsql-instance-credentials
          secret:
            secretName: cloud-sql-client-credentials
...

Periodically (every hour) having this error which triggers my pod to restart and throws timeouts to upstream services:

2020/01/27 10:03:10 Reading data from local connection on 127.0.0.1:3306 had error: read tcp 127.0.0.1:3306->127.0.0.1:57728: read: connection reset by peer
2020/01/27 10:03:10 Reading data from local connection on 127.0.0.1:3306 had error: read tcp 127.0.0.1:3306->127.0.0.1:57730: read: connection reset by peer
2020/01/27 10:03:10 Reading data from local connection on 127.0.0.1:3306 had error: read tcp 127.0.0.1:3306->127.0.0.1:57732: read: connection reset by peer
2020/01/27 10:03:10 Reading data from local connection on 127.0.0.1:3306 had error: read tcp 127.0.0.1:3306->127.0.0.1:57736: read: connection reset by peer
2020/01/27 10:03:10 Reading data from local connection on 127.0.0.1:3306 had error: read tcp 127.0.0.1:3306->127.0.0.1:57740: read: connection reset by peer

This would be ok, but it happens when our jobs are running so they fail and we have to re-run them manually afterwards..
How can we fix it?

[Carrotman42]

I do not think the log messages you provided are directly related to the pod restarting, they should not be anywhere close to a fatal error.

Are there different logs toward the end which show a different error message? What about a different log right before you see the "connection reset" messages?

[pie-r]

Hi, I have the same problem with given by cloud sql proxy.
read tcp 10.3.5.212:55978-> Pulbic ip address : read: connection reset by peer

[KUNAL13]

Hi , I am also experiencing same issue. And at the same time , there is a huge bump in memory resources. (2020/03/12 13:11:54 Reading data from local connection on 127.0.0.1:3306 had error: read tcp 127.0.0.1:3306->127.0.0.1:57266: read: connection reset by peer)

Please provide a solution for this.

[ewjmulder]

We are also experiencing this issue. Maybe it's related to Istio (which is running on our k8s cluster)?

[kurtisvg]

Hey folks - commenting on a closed/stale issue is generally considered a bad practice.

If you believe this is an issue with the proxy, please open a new issue and make sure to include all of the information to help us reproduce.

[ewjmulder]

Actually, for us it turned out to be the combination of the cloud sql proxy and the istio proxy that randomly failed under some load. After excluding sql traffic from istio, the cloud sql proxy is stable again.

[VengefulAncient]

We're also experiencing this, and we're not using Istio. Would like to know if anyone has any recent insight on this issue.

(As for "bad practices", this is the only thread that can be found on this issue that is directly related to Cloud SQL proxy. Let's not get caught up in the meta, spawning more threads is counterproductive, it just dilutes search results.)

[kurtisvg]

As for "bad practices", this is the only thread that can be found on this issue that is directly related to Cloud SQL proxy. Let's not get caught up in the meta, spawning more threads is counterproductive, it just dilutes search results.

In this case specifically, additional comments on this thread are actually counterproductive. The thread went stale after an initial request for more information was not followed up on, and almost every comment since then have been both sporadic (months apart) and haven't provided any new or useful information (other than essentially "me too!"). We want to help diagnose and solve problems folks run into while using the proxy, but copy/pasting part of an error message into a closed issue without including helpful information like proxy version, configuration, or environment isn't giving us much to work with. Opening a new issue gives you the opportunity to provide this information and let us resume having a productive conversation - feel free to even mention this issue as potentially related, so that anyone reading this issue in the future can find your new issue easily (and hopefully a more satisfying conclusion to their problem).

Now, some information about the error itself:

The original error message ("connection reset by peer") is a very generic networking level error that's caused when a remote server suddenly terminates the connection. I've only encountered it before when either a connection is closed suddenly/improperly or if a server is under critical load. There are essentially two peers the proxy interacts with:

If the error message lists a local address/port (e.i. 127.0.0.1:3306), then the problem is likely related to the connection between the proxy and your application. I would recommend checking your application for connection mishandling (make sure you are using a connection pooler!) and watching both app and proxy for high CPU usage.

If the error messages lists an external address/port (e.i. 10.x.x.x:55978), then the problem is likely related to the connection between the proxy and instance. I would recommend watching your Cloud SQL instance (or proxy container) for high CPU usage.

If anyone does hit this issue from a search result, and the above isn't enough information to help you - please feel free to open a new issue (with as much relevant information as possible) and we'll be happy to help weigh in if we can 🙂