Cloudsql-proxy: Container unable to validate certs to perform OAuth

[tmaiaroto]

I've been experience an intermittent issue (that's becoming more frequent), where cloud sql proxy can't auth via OAuth. It continues along and says it's ready for new connections, but it's not.

In the logs, I see something like this:

Get https://www.googleapis.com/sql/v1beta4/projects/xxxxx-11111/instances/staging?alt=json&prettyPrint=false: oauth2: cannot fetch token: Post https://oauth2.googleapis.com/token: dial tcp: i/o timeout

I've seen other people talk about this years ago now, but there was never any clear resolution in their discussions. It just seemed to magically go away.

I was presented with this issue again and had a chance to look into the container a bit. Outgoing connections are working, but not for SSL. Another container in the same pod can make outgoing HTTPS requests. So I don't think there's any sort of networking issue with the pod itself at least.

/ $ wget -O - https://oauth2.googleapis.com
Connecting to oauth2.googleapis.com (74.125.142.95:443)
wget: SSL/TLS certificate is not being validated!

Of course on Alpine:

/ $ wget --no-check-certificate -O - https://oauth2.googleapis.com/token
Connecting to oauth2.googleapis.com (74.125.142.95:443)
wget: can't execute 'ssl_helper': No such file or directory

I've mounted the ssl certs volume as described in a readme somewhere (it wasn't before actually and it still ran). I can see a list of certs in /etc/ssl/certs. This didn't help. There are certs in ca-certificates.crt as well. Is it perhaps an authority that needs to be updated?

I'm also running this as a sidecar. I just realized other readmes about running as a service. I'm unsure if that matters here. It just seems that the container that can't connect to HTTPS for the OAuth, which I'd assume would still be an issue whether it's a sidecar or a service.

Any ideas?

[kurtisvg]

What image/version are you using for the proxy? Can you provide more info on how you are deploying the proxy?

[tmaiaroto]

gcr.io/cloudsql-docker/gce-proxy:1.15 as a sidecar. It just runs like normal with credentials file. Not much to it really. Worked fine for a long time and now intermittently has issues.

[tmaiaroto]

Noticed the build.sh Alpine 3.5. Current seems to be 3.11, but more importantly there's a related issue: https://github.com/docker-library/official-images/issues/2773 ... For some the apk command to add certs and then update them seemed to work, but for others it did not.

[kurtisvg]

Any reason you haven't upgraded to 1.16?

[tmaiaroto]

I've since upgraded to 1.16. The timeout is new though, it hasn't happened before regardless of version. Still hasn't occurred in our production environment either. I'm not certain it's tied to a version because of how intermittent it has been. Also not sure the newer Alpine version would help, though I did see that related issue that was also intermittent. The other container in the pod can make HTTPS requests so the pod and networking should all be fine. It has GCP support stumped too.

[Carrotman42]

This seems similar to #357, which also was unable to get an oauth token. (there's a different specific error here, it appears that the request to https://oauth2.googleapis.com/token is failing here compared to a request to the metadata server failing in the other issue.)

Where is your pod running? Like, which cloud and which cloud product?

How is the proxy set up to authenticate? --credential_file?

What resources do you have allocated to the pod? You could try increasing the allocation of CPU/memory if it is relatively low.

If #357 really is related, perhaps there's a common root cause within Google's backend auth stack.

[kurtisvg]

The resource allocation is what I was suspicious of as well.

However 1.16 image switched to a distroless base image - so if alpine was the culprit I would expect that version bump to resolve the issue.

[tmaiaroto]

Everything is running in GKE, there's two separate projects. In the Secrets, there's the instance credentials JSON for a service account.

Unfortunately, it's a painfully slow trial and error. I'll see how 1.16 works as it was a recent update. Good to know the base image isn't going to be an issue. Again, I'm just searching for related issues between everything that's in use. Being able to tick items off the list is great.

Edit: also, there are no resource limits set currently. Is there a suggested minimum?

[kurtisvg]

@tmaiaroto is this issue resolved?

[tmaiaroto]

Can confirm, latest version worked. Or something else changed, but all good. Thanks!