cloudsql-proxy 🚀 - Upgrading from 1.14 to 1.16 causes authError, Message: Invalid Credentials

Hi @carlos-cubas - Can you provide more info so we can try to replicate? What does your config look like and what scenario are you using the proxy in?

kurtisvg on 30 Dec 2019

👍1

I have the proxy deployed as a sidecar in a pod that hosts a nodejs api server within GKE. I use the docker-image gce-proxy:1.16.

Changing it to 1.14 fixes the above error.

my config looks as follows:

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: example-server
  labels:
    app: example
  annotations:
    fluxcd.io/automated: 'true'
spec:
  selector:
    matchLabels:
      app: example
      tier: api
  replicas: 1
  template:
    metadata:
      labels:
        app: example
        tier: api
    spec:
      containers:
      - name: example
        image: gcr.io/example-dev/example-server:1.0.0
        imagePullPolicy: Always
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        env:
        - name: GET_HOSTS_FROM
          value: dns
          # Using `GET_HOSTS_FROM=dns` requires your cluster to
          # provide a dns service. As of Kubernetes 1.3, DNS is a built-in
          # service launched automatically. However, if the cluster you are using
          # does not have a built-in DNS service, you can instead
          # access an environment variable to find the master
          # service's host. To do so, comment out the 'value: dns' line above, and
          # uncomment the line below:
          # value: env
        - name: DB
          valueFrom:
            secretKeyRef:
              name: ex-db-prod
              key: DB
        - name: PASSWORD
          valueFrom:
            secretKeyRef:
              name: ex-db-prod
              key: PASSWORD
        - name: PORT_DB
          valueFrom:
            secretKeyRef:
              name: ex-db-prod
              key: PORT_DB
        - name: SERVER
          valueFrom:
            secretKeyRef:
              name: ex-db-prod
              key: SERVER
        - name: USER_DB
          valueFrom:
            secretKeyRef:
              name: ex-db-prod
              key: USER_DB
        - name: NODE_ENV
          value: PRODUCTION
        ports:
        - containerPort: 8080
      # Change <INSTANCE_CONNECTION_NAME> here to include your GCP
      # project, the region of your Cloud SQL instance and the name
      # of your Cloud SQL instance. The format is
      # $PROJECT:$REGION:$INSTANCE
      # [START proxy_container]
      - name: cloudsql-proxy
        image: gcr.io/cloudsql-docker/gce-proxy:1.16
        command: ["/cloud_sql_proxy", "-instances=centering-line-248621:us-central1:example-cloud-sql-dbe28180-prod=tcp:5432",
          "-ip_address_types=PRIVATE"]
                  #"-credential_file=/secrets/cloudsql/credentials.json"]
        # [START cloudsql_security_context]
        securityContext:
          runAsUser: 2  # non-root user
          allowPrivilegeEscalation: false
        # [END cloudsql_security_context]
        volumeMounts:
        - name: cloudsql-instance-credentials
          mountPath: /secrets/cloudsql
      # [END proxy_container]
    # [START volumes]
      volumes:
      - name: cloudsql-instance-credentials
        secret:
          secretName: ex-db-prod
    # [END volumes]

carlos-cubas on 30 Dec 2019

I'm unable to replicate, but I suspect it's due to this section:

      - name: cloudsql-proxy
        image: gcr.io/cloudsql-docker/gce-proxy:1.16
        command: ["/cloud_sql_proxy", "-instances=centering-line-248621:us-central1:example-cloud-sql-dbe28180-prod=tcp:5432",
          "-ip_address_types=PRIVATE"]
         #"-credential_file=/secrets/cloudsql/credentials.json"]

What are you using for account credentials? If you provide a service account, does it still have this problem?

kurtisvg on 3 Jan 2020

we are using a service account for credentials, and yes we still have this problem.

carlos-cubas on 4 Jan 2020

This is me just guessing and browsing history a bit.

* errors parsing config *

There are some changes roughly 4 months ago that change the way that the config is parsed.
https://github.com/GoogleCloudPlatform/cloudsql-proxy/blame/69852d3cd6a85e1a53617cee0737cc5978ca6db5/cmd/cloud_sql_proxy/proxy.go#L338

Appears that parse errors used to be simply logged, but now they cause an exit. this might be what's causing the problem. I will change my config command a bit see if i can get it to work w/o issues.

carlos-cubas on 4 Jan 2020

This is me just guessing and browsing history a bit.

* errors parsing config *

There are some changes roughly 4 months ago that change the way that the config is parsed.
https://github.com/GoogleCloudPlatform/cloudsql-proxy/blame/69852d3cd6a85e1a53617cee0737cc5978ca6db5/cmd/cloud_sql_proxy/proxy.go#L338

Appears that parse errors used to be simply logged, but now they cause an exit. this might be what's causing the problem. I will change my config command a bit see if i can get it to work w/o issues.

Please let us know what you find. If the only problem is the error being ignored before, it seems weird that you were still able to successfully connect to the instance but after having failed to parse the instance config.

kurtisvg on 7 Jan 2020

We had the same issue. Version 1.14 did not care if we had old entries in the configuration for databases which do not exist anymore. 1.16 would not start up as long as we listed old configurations. I cannot say if 1.16 crashes if you delete a database while running but this might be a pitfall, removing a sql db causing the proxy to stop working.

rettier on 20 Jan 2020

Yes, this change in behavior has some pretty serious implications.

The changes are in these two PRs:
https://github.com/GoogleCloudPlatform/cloudsql-proxy/pull/310
https://github.com/GoogleCloudPlatform/cloudsql-proxy/pull/311

which make up the v1.15 -> v1.16 diff:
https://github.com/GoogleCloudPlatform/cloudsql-proxy/compare/1.15..1.16

The issue is that now a single "parsing" error in a slice of InstanceConfigs, will result in the entire slice of connections being rejected. An additional issue is that "parsing" each connection involves interacting with the googleapi of the target project, which can be rejected if no ServiceAccount permission has been setup, or if permission has been removed.

So, without any change to your connections config, if permission is removed from one of the target configs, all will fail (at least for that setup method) after restart. The previous behavior would log it and continue to setup the remaining configs.

Honestly with this new behavior, it makes much more sense to just run a separate CloudSQL Proxy instance for each connection. It's certainly safer, as one misconfiguration will only break that one Proxy connection. The v1.15 behavior was much more intuitive, especially given the dynamic connection setup behavior via metadata, and this change should've been announced in the README, imo.

Until this is fixed or reverted, I have had to switch back to using 1.15 indefinitely. I'm willing to code and submit the revert/fix, if others agree it should be changed (even if just making the behavior selectable). I'll have to think/investigate if there is a way to also handle https://github.com/GoogleCloudPlatform/cloudsql-proxy/issues/288 in a better way.

chadnetzer on 4 Mar 2020

The issue is that now a single "parsing" error in a slice of InstanceConfigs, will result in the entire slice of connections being rejected. An additional issue is that "parsing" each connection involves interacting with the googleapi of the target project, which can be rejected if no ServiceAccount permission has been setup, or if permission has been removed.

There were already some configuration errors that would cause the proxy to fail on startup. This change made that behavior more consistent - and now users can more reliable trust that when the proxy starts up, it's configured correctly instead of waiting until the first time they connect and being unable to reach the database.

Additionally, this is compounded by the fact that _any_ error that occurs could cause the proxy to fail -- including transient ones. Maybe a better behavior is to retry on transient errors, but it's not trivial to determine which error is transient and which isn't. So it's better for now just err on the side of caution and surface any errors so the user has immediate feedback.

So, without _any_ change to your connections config, if permission is removed from one of the target configs, _all_ will fail (at least for that setup method) after restart. The previous behavior would log it and continue to setup the remaining configs.

Right - if someone haven't changed their config, why would they expect the proxy to change it's behavior - just start ignoring a portion of that config and stop serving traffic for a specific instance?

The v1.15 behavior was much more intuitive

I disagree. I could see how if you were aware of the previous behavior, the change is different. However, I think that most users expect from an executable is that if it starts up successfully and says "Ready to receive new connections" is that it is actually ready to receive new connections.

this change should've been announced in the README, imo.

It was announced in the release notes for this version.

I'm willing to code and submit the revert/fix, if others agree it should be changed (even if just making the behavior selectable).

There's currently an open PR (#340) to add a flag to re-enable similar behavior. Currently we are leaning towards _not_ merging it -- mostly because it adds significant friction to debugging the behavior. However, feedback on the issue is welcome.

kurtisvg on 4 Mar 2020

Right - if someone haven't changed their config, why would they expect the proxy to change it's behavior - just start ignoring a portion of that config and stop serving traffic for a specific instance?

If 100 configs are correct, but one remote project revokes their permission, I'd expect upon restart that 99 configs still work and 1 breaks. Not that 100 break; that's unintuitive and a bit customer-hostile, imo. It's also a regression, so someone who built this intended it to work the way I claim it should.

There's currently an open PR (#340) to add a flag to re-enable similar behavior. Currently we are leaning towards _not_ merging it -- mostly because it adds significant friction to debugging the behavior. However, feedback on the issue is welcome.

I'll take a look. I'm sure there is an agreeable solution going forward.

chadnetzer on 5 Mar 2020

A simple example of the configuration "breaking" would be someone deleting
a database that is otherwise "not used." If someone happens to have that
database in some config somewhere, things would keep working after the
database deletion until the next time the Proxy client restarted (which
could be random or coordinated). That's quite surprising IMO, and could
cause issues that are only detected long after the actual "change" (i.e.
the deletion of the "not used" database), making it hard and frustrating to
debug. That doesn't seem ideal.

I'm not arguing for the Proxy to continually double-check the config
though, either: IMO deleting a database shouldn't prevent a previously
working Proxy from working with other databases which still exist.

It also seems weird to me to throw our hands up and claim nothing works
when there are indeed connections which would work.

A compromise could be to only have the Proxy exit on config parse failed if
it couldn't find any valid instance configs since in that case the Proxy
knows it can't do anything useful in that case.

Carrotman42 on 5 Mar 2020

A simple example of the configuration "breaking" would be someone deleting
a database that is otherwise "not used." If someone happens to have that
database in some config somewhere, things would keep working after the
database deletion

To clarify it doesn't really "keep working" - the proxy is bound to that interface and received connections, but there will be errors galore in the logs when the refresh of the cert fails. If the user's application depends on this connection to the database, they'll get a timeout error. If they check the proxy logs at this time, they'll see errors about refreshes throttling that should provide them a reason why.

until the next time the Proxy client restarted (which
could be random or coordinated). That's quite surprising IMO

Maybe surprising to discover, but again - not unexpected (or even unintuitive) unless you were already familiar with the previous behavior. Specifically with tools like nginx, if I try to start up a process and it fails to bind the interface I've specified, it exits.

The proxy should be have similarly - if it fails to start up successfully as configured, it should exit.

and could cause issues that are only detected long after the actual "change" (i.e.
the deletion of the "not used" database), making it hard and frustrating to
debug. That doesn't seem ideal.

I don't think this is true. For example, given the scenario described above where the user deletes an instance and the proxy is restarted:

_With the previous behavior:_ The proxy skips that instance, logs the error, and enters the serve state. If the user started the proxy, they see "Ready for new connections" and think it's healthy. However, once the application attempts to connect they are only left with "CANNOT FIND FILE" or "CONNECTION TIMED OUT" errors, which don't identify the problem. If they triage and check the proxy logs, there aren't any errors correlated with the connection attempt. Instead they need to _know_ that this error will appear at the beginning of the proxy logs (which, depending on how long until the first connection attempt) may be lost to time.

_With the current behavior:_ The proxy exits. If the user restarted, it's clear something is now wrong. If it was an automated restart (like supervisor or k8s) it likely restarts a few more times and shows up as a failure. Upon checking the logs the message is something along the lines of "The instance does not exist or you do not have permission to access."

It seems to me that unless the user _never_ plans to use the database again it's preferable to surface the error immediately.

kurtisvg on 5 Mar 2020

To clarify it doesn't really "keep working" - the proxy is bound to that interface and received connections, but there will be errors galore in the logs when the refresh of the cert fails. If the user's application depends on this connection to the database, they'll get a timeout error. If they check the proxy logs at this time, they'll see errors about refreshes throttling that should provide them a reason why.

It seems you didn't understand what I meant. Obviously if someone tried to make the connection to the nonexistent database things wouldn't work for that database connection. My point, piggybacking off what @chadnetzer said, is that with this change then none of the other databases can work. This goes against a lot of production advice which is "try to get as much done as possible in the face of errors, don't just throw your hands up and crash". Just throwing your hands up causes cascading failures in general.

Carrotman42 on 6 Mar 2020

I’ll have a more comprehensive response this weekend, but in short, yes, our application using the Proxy already detects “down” connections, and handles it via policy the application admins setup. It’s normal to (for example) setup a new connection on the Proxy, and then wait until the remote DB owner adds service client access to that DB (though it needs a metadata “bump” to activate). Also, with the v1.15 behavior, the same authoritative metadata connection setup can be used in different environments that may not be equally permissioned for all the environments; this is hugely important for making our use-case both safe and easy. But it all works fine now with v1.15, and though some users may not be able to reach a particular “down” DB with the app, others users who don’t need that DB connection are unblocked and can continue to work.

With v1.16, this use case completely fails. Any “down” DB blocks all users upon restart or (potentially) metadata update, even those using unrelated DBs, and its a major outage. It’s simply not a tenable policy for our use case, and retooling to the new behavior removes most of the elegance of the “instances_metadata” option, and the dynamic setup behavior (without restarting the Proxy process) that it enables. That behavior was a key point when deciding to use the Proxy.

I do think an approach like the one in PR #340 that was mentioned, would satisfy those who need both use cases. In fact, we currently use both behaviors, just by running one instance as v1.15 and one as v1.16. But I don’t want to miss out on future improvements by staying at v1.15 indefinitely, or worse, have to maintain a fork where I revert the v1.16 behavior. So an approach like PR #340 would satisfy us and probably others (though I want to test it a bit, I might have some suggestions for slight improvements before merging).

Edit: Just noticed I started this with “in short”, but it ended up pretty long. :grin:

chadnetzer on 6 Mar 2020

I agree --skip_failed_instance_config would at least eliminate the issue. It still seems to me that it'd be a good flag to default to "true" for production situations; on the other hand, maybe it is better off as "false" for development situations.

Carrotman42 on 6 Mar 2020

With v1.16, this use case completely fails. Any “down” DB blocks all users upon restart or (potentially) metadata update, even those using unrelated DBs, and its a major outage.

I will clarify my position in that that behavior should only apply during the setup phase of the proxy. If I've misunderstood and the proxy is exiting after it enters the "serving" state when the metadata value is refreshed, I would consider that a bug (and oversight on my part). The proxy shouldn't be exiting once it's reached a stable state, especially if it's recoverable.

kurtisvg on 6 Mar 2020

You have to assume that the Proxy could restart at any time, e.g. by an on-host event.

Carrotman42 on 6 Mar 2020

You have to assume that the Proxy could restart at any time, e.g. by an on-host event.

I do assume this - this is why the previous behavior is so dangerous. The proxy could restart at any time and suppressing this error rather than surfacing it leads to a flaky and inconsistent environment. But it's safe to assume that the (re)start is the result of some action (automated or otherwise) and the only way to provide feedback on that action is unfortunately binary (run or exit).

I think we'd be having a different argument here if the proxy was able to successfully bind to these interfaces and provide feedback on failed connection attempts instead, but unfortunately it isn't (and likely won't ever be able to due to the Postgres socket suffix requirement). But since the proxy's current strategy means a decision is made to which instances are supported at the start (and won't be changed later), we owe it to the user to provide that feedback. At this point there is nothing the user can do to change that (unless they restart the proxy).

This is also why serving from a static list of instances is a different scenario than serving from a mutable list of instances. With a polled list, the decision on which instances are supported is re-evaluated periodically. This means that the feedback is both surfaced periodically (via the logs, when re-evaluating the metadata) and recoverable (neither which is true from the default behavior).

kurtisvg on 6 Mar 2020

I do assume this - this is why the previous behavior is so dangerous. The proxy could restart at any time and suppressing this error rather than surfacing it leads to a flaky and inconsistent environment.

But it's not an inconsistent environment: before the proxy restarted, trying to connect to a recently-deleted database would just end with error logs; after the proxy is restarted the same error logs will just happen. I wouldn't call this suppressing the error; the application logic trying to talk to the database would still fail and the user can tell if their app is broken because of the deleted database.

If the proxy crashes on startup when it is configured to connect to a database which no longer exists, it will go from just logging errors for connections to that database (pre-restart) to not being able to start back up (post-restart). To me this is the inconsistent situation, where a random restart would break all other existing databases.

Am I misunderstanding?

But it's safe to assume that the (re)start is the result of some action (automated or otherwise) and the only way to provide feedback on that action is unfortunately binary (run or exit)

I disagree strongly with this statement. Hosts break down all the time; OOM conditions cause transient restarts; zones can have temporary blips where many VMs are forced to restart. Even on restarts related to upgrades (for version N+1 where both version N and N+1 crash if there is a deleted database in the configs), the normal "rollback to the previous known version" best practice wouldn't fix anything, and would cause more confusion. (Hmm, version N was working and version N+1 stopped working, but now version N is not working?)

Carrotman42 on 6 Mar 2020

But it's not an inconsistent environment: before the proxy restarted, trying to connect to a recently-deleted database would just end with error logs; after the proxy is restarted the same error logs will just happen. I wouldn't call this suppressing the error; the application logic trying to talk to the database would still fail and the user can tell if their app is broken because of the deleted database.

The feedback from the user's application is neither clear nor actionable; The user will receive a "Connection refused" message over TCP or "File not found" error via unix socket. These are just symptoms of a root problem, and not even necessarily indicate that the proxy is the source of the problem. It could just as easily be a networking or file problem (particularly in a system with docker or k8s, where the networking/file space is not as immediately clear as when running on a VM).

If the user checks the proxy, it's not immediately clear that there is a problem - it's still running, and the only message to indicate that something is wrong is at the very start of the logs - potentially way outside the relative time window the application error occurred. Unless the user already knows that the proxy made that decision, it's a bit of a needle in a haystack situation.

If the proxy crashes on startup when it is configured to connect to a database which no longer exists, it will go from just logging errors for connections to that database (pre-restart) to not being able to start back up (post-restart). To me this is the inconsistent situation, where a random restart would break all other existing databases.

Am I misunderstanding?

This is consistent with how the proxy handles other start up related errors - if the environment around it has changed and it is no longer able to setup as the user is asking, it exits. If they've provided the credential file flag and it's no longer present / readable, should the proxy default to using ADC instead? It feels to me that the answer is pretty obviously no - that's not what the user has asked, and doing so would be confusing and potentially lead to a hard to debug situation.

At least in this scenario the issue is easy to diagnose - the proxy being the source of the problem is obvious (it's not running) and the reason for exit is (hopefully) clearly stated and easily discoverable.

But it's safe to assume that the (re)start is the result of some action (automated or otherwise) and the only way to provide feedback on that action is unfortunately binary (run or exit)

I disagree strongly with this statement. Hosts break down all the time; OOM conditions cause transient restarts; zones can have temporary blips where many VMs are forced to restart.

Each time the proxy starts _something_ has taken an action to start it (again, even if that something is automated). In the scenario of disaster recovery, something is monitoring the state of the proxy (k8s, supervisord, etc) that detected the proxy had failed and taken an action to restart it. Hopefully, when that something is unable to do what is being asked of it, it's also passing feedback somewhere - maybe triggering a page, reporting a failed state, etc.

Even on restarts related to upgrades (for version N+1 where both version N and N+1 crash if there is a deleted database in the configs), the normal "rollback to the previous known version" best practice wouldn't fix anything, and would cause more confusion. (Hmm, version N was working and version N+1 stopped working, but now version N is not working?)

Okay, but with the current behavior the user inspects the logs from the proxy and the reason for failure is immediately clear. And in the _other_ scenario, where there the configuration change is swapping out the name of the database (potentially typed incorrectly), the canary roll out just caught an incorrect configuration before any traffic was switched over.

With the previous behavior, the application failing to connect is the first indicator that something has gone wrong. As I said above, it's not immediately obvious as to _why_ and the person debugging has to follow the breadcrumbs.

kurtisvg on 6 Mar 2020

Marking this bug as closed - while there has been some good discussion, but I think most of it is tangential to the original issue (which I think has been resolved)

kurtisvg on 8 Apr 2020

We have been getting this error lately in our case we are using a service account key for authentication and the issue happens randomly. In our case we are using the proxy on a Golang API so we are not relying on the proxy as a sidecar container and instead we are directly using the proxy Go package to communicate with CloudSQL.

github.com/GoogleCloudPlatform/cloudsql-proxy/proxy/dialers/mysql

2020/06/12 20:18:50 failed to refresh the ephemeral certificate for [INSTANCE]
before expiring: googleapi: Error 401: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential.
See https://developers.google.com/identity/sign-in/web/devconsole-project.
More details:
Reason: authError, Message: Invalid Credentials

alejandro-delgado-garcia on 12 Jun 2020

Hi @alejandro-delgado-garcia - please open a new issue as this is a different issue that just happens to be throwing the same error.

kurtisvg on 12 Jun 2020

👍1

@kurtisvg 1.16 also breaks essential workflows of using gcloud auth login to have credentials to use the proxy locally.

lozandier on 1 Dec 2020

@lozandier I don't think the v1.15 docker image even has gcloud installed, so I'm not really sure what you mean by "essential workflow". Regardless, it also sounds like a different issue than this one, so I'd suggest opening a new issue with more information on what you would like supported since this one is closed.

kurtisvg on 1 Dec 2020

I got same error .
so, the recomend solution for production situations is adding --skip_failed_instance_config ?
even thought, there is no --skip_failed_instance_config on the GCP document.

SpaceNet on 6 Jan 2021

I got same error .
so, the recomend solution for production situations is adding --skip_failed_instance_config ?
even thought, there is no --skip_failed_instance_config on the GCP document.

solved. It's my bad.
I did not set serviceAccount on deployment yaml.

just needed setting KSA ( I'm using 1.17, not 1.16 )

    spec:
      serviceAccountName: <YOUR-KSA-NAME>

SpaceNet on 6 Jan 2021

Cloudsql-proxy: Upgrading from 1.14 to 1.16 causes authError, Message: Invalid Credentials

All 27 comments

Related issues