Cloudsql-proxy: Reload service account keyfile periodically

[kevincvlam]

Hi,

We run the CloudSQL proxy in our kubernetes cluster as a deployment and sometimes we rotate the secret that is used to provide the credentials file for IAM authentication.

As a result the credentials loaded at start-up of the proxy become invalid and the proxy begins printing invalid credentials errors, but does not error out. What's the recommended way to handle this situation? Is there a way to have the proxy reload the credentials?

My understanding is that mounted secrets are updated automatically, so it's up to the application to respond accordingly:

Mounted Secrets are updated automatically When a secret being already consumed in a volume is updated, projected keys are eventually updated as well. The update time depends on the kubelet syncing period.

[https://stackoverflow.com/questions/37945800/update-kubernetes-secrets-doesnt-update-running-container-env-vars]

[kurtisvg]

Hey @kevincvlam, thanks for bringing this issue to our attention.

We discussed this issue this morning, and decided that currently the only way to reload the credentials would be to restart the container with the proxy inside. This is obviously not an ideal solution, so we are investigating ways we could handle this. Currently we are looking into the following:

1. Reload credentials hourly during SSL cert refresh
2. Attempt credentials reload upon receiving an invalid credentials error
3. Potentially exit with error code if failing to retrieve valid credentials after X minutes

We'll be using this issue to update our progress on this issue.

[kevincvlam]

Hey @kurtisvg, thanks for the quick reply, and looking forward to your solution!

Do you have any idea regarding when you expect the issue to be resolved?

[kurtisvg]

Unfortunately, I don't have any promises to make at the moment, just that it's in the queue and the team will get to it when we can. If you have any expertise in this area, we are open to contributions.

[markvincze]

Hey folks,

This is affecting us as well. The setup we have is that we store the service account key in a Kubernetes secret, which is mounted to the Cloud SQL Proxy sidecar. And we rotate the service account key every day, and replace the content of the secret.

And as far as I understand, when we change the content of the secret, that change is automatically propagated to the mounted file seen by the running proxy container. (probably this is the same setup @kevincvlam described?)

This is not handled by the proxy, so if the mounted key file changes, that's not picked up by the running proxy, right?
Do you have any update on the timeline when this improvement can be expected?

Thanks!

[JorritSalverda]

In our golang applications we handle the reloading of the key by reinitialising an instance of the _class_ that uses the service account key with code like this:

dnsService := NewGoogleCloudDNSService(*googleCloudDNSProject, *googleCloudDNSZone)

foundation.WatchForFileChanges(os.Getenv("GOOGLE_APPLICATION_CREDENTIALS"), func(event fsnotify.Event) {
    log.Info().Msg("Key file changed, reinitializing dns service...")
    dnsService = NewGoogleCloudDNSService(*googleCloudDNSProject, *googleCloudDNSZone)
})

See https://github.com/estafette/estafette-google-cloud-dns/blob/09eaf7f4123b6c4a012837f2415893219456d137/main.go#L81-L84 and https://github.com/estafette/estafette-foundation/blob/master/foundation.go#L104-L161 for implementation details.

Works like a charm and relies on the github.com/fsnotify/fsnotify libary, which doesn't bring in too many dependencies.

[dhduvall]

I made some changes that address the failures that I've been seeing. It's not comprehensive, and it's pretty hacktastic, but it's survived a day of having Vault rotate the service account keys from underneath it and the new keys mounted into the k8s container. There are three specific points where it can recover: when the credential file is missing or corrupt at startup; at first connection; and failure to rotate the ephemeral cert. I'm sure there are many other places it could fail, but those are the ones I've been running into.

I'm not going to submit a PR in this state, but I figured if anyone else had a need for this, they could take what I have. If it's within shouting distance of being acceptable, though, I can try to polish it up a bit.