Cloudsql-proxy: x509: certificate signed by unknown authority

[ybenzine]

Last night around 11:02 pm, our batch workers on GAE reported losing connection to MySQL multiple times in the middle of running some queries.

I checked our production MySQL instance for anything out of the ordinary but its CPU and memory usage was relatively low, as were the amount of connections. I was even able to use our web app which uses that same database when these errors started appearing, confirming that in fact our MySQL instance was up, running, and responding to requests. In other words, our MySQL instance was not under stress and everything was functioning correctly. The issue went away on its own around 11:20 pm with no action required on my part.

Looking at the logs, before every 'Lost connection to MySQL server during query' entry, there was this error:

2019-08-07 23:05:05.000 EDT
2019/08/08 03:05:05 couldn't connect to "our-production-env:us-somewhere:our-mysql-db": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "Google Cloud SQL Server CA")

These errors showed up a few times on our two GAE batch applications, immediately followed by "Lost Connection to MySQL". This hasn't happened since.

Any insight as to what may be responsible for this? I'm worried it will happen again and I have no real lead.

Thanks in advance!

[ybenzine]

It happened again tonight.

Moreover, I noticed this message appearing before the first failure; I'm not sure of its relevance but it seems connected:

2019-08-08 20:39:49.000 EDT
2019/08/09 00:39:49 Throttling refreshCfg(our-production-env:us-somewhere:our-mysql-db): it was only called 9.676532666s ago

[dwhoman]

I also had this happen on a Google GCE Ubuntu system starting at 2019/08/08 03:01:56 UTC lasting until 03:20:07, then again at 05:29:13 UTC to 05:34:34, and then again the next day, 2019/08/09, at 00:39:37 to 00:59:01 UTC.

It also happened on a GKE system at 2019/08/08 03:15:19 UTC, and 05:31:51 UTC to 05:32:18, and on 2019/08/09 at 00:39:47 UTC.

I had this happen on a physical computer, running the proxy, at 2019/08/08 03:12:10 UTC. It did not happen on 2019/08/09 for that computer when the GCE and GKE computers were experiencing issues.

Some of these times correlate with ybenzine's issues. Stopping and starting the cloud sql proxy client did not fix the issue. In addition to the errors listed by ybenzine, I got the following error:
googleapi: Error 409: The instance or operation is not in an appropriate state to handle the request., invalidState

[ybenzine]

Thank you, Devin. I searched my logs for invalidState and Error 409 but did not find a match. Possibly due to the fact you're using GCE and we're using GAE.

It happened once more last night, though this time only once for each GAE app.
The somewhat interesting bit is that it happened on both GAE apps at almost exactly the same time, within a second: 20:09:00.000 EDT and 20:09:01.000 EDT.

[kurtisvg]

@ybenzine - Is this App Engine Flexible?

@dwhoman - the googleapi error you are describing seems to indicate there was an issue with your Cloud SQL instance.

It might be worthwhile (for both of you) to open a private issue with your project and instance information so that the support team can look more closely.

[dwhoman]

I have not had connection problems since August 9th. The 409 error may have been due to a database restart that was done to try to remedy the situation. The restart finished 1.5 minutes before the 409 error. The GCE and the physical computer are using cloudsql-proxy version 1.14 and GKE is using 1.11.

[ybenzine]

@kurtisvg: yes, it is App Engine Flexible.

Thank you for the advice, I"ll open a private issue. FWIW, this issue hasn't popped up the past two days so hopefully it won't happen again 🤞

[ybenzine]

By the way @kurtisvg, is the link you shared for reporting a private issue the correct one? It seems like an internal link for Google employees, no? I certainly don't have an @google.com email address 😄

[kurtisvg]

Sorry about that - you can find all the Product Issue Trackers here. The Cloud SQL tracker should be here.

[huguesalary]

Hi, the same issues have been happening to me since Aug. 7th, and are happening right now.

Our SQL Proxy keep spitting out Throttling refreshCfg and our entire app is down.

[kurtisvg]

@huguesalary - Could you provide more info? Are you connecting from a specific location, using the proxy? Are you seeing the "x509: certificate signed by unknown authority" error as well?

I would also encourage you to open private issue as well, as it will allow the support team to take a look

[huguesalary]

Hi @kurtisvg,

We're not actually seeing x509: certificate signed by unknown authority, we're seeing Throttling refreshCfg(betabrand-zeppelin:us-central1:magento): it was only called 32.613847651s ago over and over.

We're using the SQL Proxy yes, from a k8s cluster located in us-central1-b, our DB is _currently_ located in us-central1-a.

This still happenign as of now and has been happening for the past 2 hours.

We do not have a support package and as such can't open a ticket with Google.

[kurtisvg]

@huguesalary - I believe you can open a private issue without a support package (private issues are different than a support ticket)

[broady]

Verification error means the server cert provided by the sqladmin service was invalid.

We should probably add some more logging to this codepath so we can figure out what went wrong.

[Carrotman42]

Random theory: I think this might happen if someone reset the SLL configuration of the instance: that would cause the cached version of the server cert to be incorrect for some period of time. There is indeed throttling to make sure that there's no thundering herd when this happens (and this throttling is evident in the ealier comment https://github.com/GoogleCloudPlatform/cloudsql-proxy/issues/297#issuecomment-523207974), but it should only last for 1 minute by default (https://github.com/GoogleCloudPlatform/cloudsql-proxy/blob/18df49e/proxy/proxy/client.go#L31).

[broady]

@Carrotman42 good theory.

I think the issue reported by @huguesalary is different than OP @ybenzine.

@ybenzine:
"crypto/rsa: verification error" from 11:02 - 11:20 (18 minutes)

@huguesalary:
throttled refreshCfg ("2 hours")

@huguesalary, could you open a separate issue, please?

If you could also share more detailed logs, it would be good to see why/whether refreshCfg actually failed. If you want to share privately, you can e-mail me logs at cbro at google dot com.

[curtbushko]

Any chance that someone has figured this out. We are seeing it on cloudsql proxy 1.15.

[art-tts]

@curtbushko I use the proxy as well, and under "Connections" in the cloudsql console ui, i clicked this button and it started working for me.

[kurtisvg]

Marking as closed - this issue has gone stale and we don't have concrete steps to reproduce. If anyone runs into this, please open a new issue and reference this one. Thanks!