Cloudsql-proxy: oauth2: cannot fetch token

[andersgb]

Our GKE based webapp is unable to connect to Google Cloud SQL right now. I have no idea why this suddenly started happening. We haven't touched the GKE deployment in several days.

Our logs show everything being OK until 14:30:12 CEST today (2017-05-25), when all connections fail with this error:
2017/05/25 15:58:37 couldn't connect to "****:europe-west1:****": Post https://www.googleapis.com/sql/v1beta4/projects/*****/instances/****/createEphemeral?alt=json: oauth2: cannot fetch token: Post https://accounts.google.com/o/oauth2/token: dial tcp: i/o timeout

We are running cloudsql-proxy through the image b.gcr.io/cloudsql-docker/gce-proxy:1.05
I can succesfully connect manually with cloud_sql_proxy from my machine.

I realize this might not be a cloudsql-proxy issue, but any pointers to help me out would be much appreciated.

[Carrotman42]

Agreed that this doesn't appear to be a cloudsql-proxy-specific issue, and I'm not quite sure how to go about debugging this other than asking you to SSH to your kubernetes nodes and confirm that the network is indeed broken (or at least that HTTP to accounts.google.com isn't working properly).

If SSH to the nodes doesn't work, it's possible a network issue is preventing some packets from flowing as expected (and SSH is probably failing for the same reason HTTP is). If it does work, I suggest doing some sanity curls to see what works and what doesn't:

curl google.com
curl github.com
curl asdf.com # not usually https, can test is port 443 is broken but 80 works (for some reason...)
curl accounts.google.com

[Carrotman42]

If it does turn out to be the network, moving your cluster to a different zone may fix things.

[andersgb]

Hm, strange, those sanity curls work just fine. I'll look into moving to another cluster and see if that helps.

[andersgb]

The cloudsql connections magically started working again around 23:07 CEST (on the same cluster). Dunno how this could happen, I'm in contact with Google Cloud support about it.

[Carrotman42]

Any updates here? I suspect there was just some transient network problem but it'd be good to confirm.

[andersgb]

We didn't really get to the bottom of it. The GCP support guy didn't find anything of note in the logs, neither did we. The main suspect is a Docker networking problem. "Some transient network problem" is probably the best description we have right now.

Probably a coincidence, but 5 days later we actually got a very similar issue, but this one seemed to affect more people. This time it was a stream error: stream ID 1; PROTOCOL_ERROR, see [1], but this issue also disappeared after a while.

We have now bumped the gke-proxy image to version 1.09 and have upgraded Kubernetes version on our GKE cluster nodes and we've gone 2 days straight without downtime now (hah!)

Anyway, thanks for the suggestions here, I'm fine with closing this issue as I don't think cloudsql-proxy is to blame for this.

[1] https://groups.google.com/forum/#!search/google$20cloud$20sql$20proxy$20stream$20error%7Csort:relevance/google-cloud-sql-discuss/gmrBvV6IIhk/cejzaKMNCAAJ

[Carrotman42]

Yep, that issue is just a coincidence. I've posted on that Cloud SQL Discuss post you linked to explaining the issue yesterday (and there's also https://github.com/GoogleCloudPlatform/cloudsql-proxy/issues/92 about it).

Glad to hear things are stable now. I do suspect that the original issue was simply related to docker or the networking in that zone. Please let me know if there's anything I can do.

[delj]

just encountered the same (hopefully-transient?) error trying to use cloudsql-proxy kubernetes pod sidecar to connect to a postgres DB hosted in google cloud SQL

edit: I was just working with a small test cluster, so re-creating the cluster in a different zone and region was fine and the problem went away. re-deploying or waiting 1 hour had no effect.

[manhjava]

I met problem again. Just re-create new node-pool and delete old node-pool and problem resolved

[FloraSaysGoodmorning]

Met this problem when running cloudsql proxy as sidecar, image name is gcr.io/cloudsql-docker/gce-proxy:1.11,
Have been seeing the error from time to time for 1 week now.
2018/12/12 04:24:10 couldn't connect to "my db": Post https://www.googleapis.com/sql/v1beta4/projects/myproject/instances/myinstance/createEphemeral?alt=json: oauth2: cannot fetch token: Post https://accounts.google.com/o/oauth2/token: dial tcp: i/o timeout, it goes away when I retry.

[rvora]

I am seeing this problem after upgrading GKE to 13.9 and cloud-sql-proxy image to 1.15.

Strange thing is if I 'kubectl exec -it -c /bin/sh' and just do "ps" and "kill " the cloud_sql_proxy process, Kubernetes restarts the container and everything starts working. This method has always worked for me though I can't be sure about the timing as it usually takes me several minutes to get to the kill step.

Why would an API not work the first time a container starts and then suddenly work?

Also, I changed the cloud_sql_proxy command to something like: "timeout 4 $cloud_sql_proxy_command && $cloud_sql_proxy_command" where $cloud_sql_proxy_command contains the full syntax with all the CLI flags.

Even with that, when I redeploy all the Pods the method works 60% of the time i.e. for about 60% of the Pods the second cloud_sql_proxy command works. But for about 40% even the second cloud_sql_proxy command fails with the errors OP posted here.

I am just totally confused and frustrated by the unpredictable nature of this cloud_sql_proxy.

[ctavan]

When using a VPC-native Kubernetes Engine cluster (which I believe is the default now) you no longer need the cloudsql-proxy but instead can just connect to the private IP of your CloudSQL instances, see https://cloud.google.com/sql/docs/postgres/connect-kubernetes-engine