Cloudsql-proxy: Error when attempting to deploy with Workload Identity

[thebigredgeek]

Bug Description

Following the latest instructions here, I am left with a pod that fails to start with the following log output:

2020/06/22 21:13:09 current FDs rlimit set to 1048576, wanted limit is 8500. Nothing to do here.
2020/06/22 21:13:09 errors parsing config:
    googleapi: Error 401: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
More details:
Reason: authError, Message: Invalid Credentials

Example code (or command)

Deployment yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  selector:
    matchLabels:
      run: api
  template:
    metadata:
      labels:
        run: api
    spec:
      containers:
      - name: api
        image: api
        ports:
        - containerPort: 8080
        env:
        - name: POSTGRES_HOST
          valueFrom:
            configMapKeyRef:
              name: postgres-connection-info
              key: host
        - name: POSTGRES_DB
          valueFrom:
            configMapKeyRef:
              name: postgres-connection-info
              key: database
        - name: REDIS_URL
          valueFrom:
            configMapKeyRef:
              name: redis-connection-info
              key: url
        - name: POSTGRES_USER
          valueFrom:
            secretKeyRef:
              name: postgres-connection-info
              key: user
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-connection-info
              key: password
      - name: cloud-sql-proxy
        # It is recommended to use the latest version of the Cloud SQL proxy
        # Make sure to update on a regular schedule!
        image: gcr.io/cloudsql-docker/gce-proxy:1.17
        command:
          - "/cloud_sql_proxy"

          # If connecting from a VPC-native GKE cluster, you can use the
          # following flag to have the proxy connect over private IP
          # - "-ip_address_types=PRIVATE"

          # Replace DB_PORT with the port the proxy should listen on
          # Defaults: MySQL: 3306, Postgres: 5432, SQLServer: 1433
          - "-instances=keybox-281016:us-central1:development=tcp:5432"
        # securityContext:
        #   # The default Cloud SQL proxy image runs as the
        #   # "nonroot" user and group (uid: 65532) by default.
        #   runAsNonRoot: true

How to reproduce

1. Follow the instructions mentioned above

Environment

1. OS type and version: Mac OS 10.15.4
2. Cloud SQL Proxy version (./cloud_sql_proxy -version): 1.17

[kurtisvg]

@thebigredgeek Can you confirm that you've correctly enabled Workload Identity on your cluster? If using an existing cluster you need to enabled it and migrate your workloads.

[thebigredgeek]

Yes, I enabled it when I created the cluster.

[kaaloo]

Thanks for posting this @thebigredgeek. I had a running cluster that I had setup maybe two weeks ago with Workload Identity for the CloudSQL Proxy but I tore it down yesterday to test some changes in our provisioning scripts and I've been experiencing the same issue since. I also had trouble initially but it finally worked when I added this iam policy binding. However, maybe that was a fluke. In any case that "workaround" is no longer working.

gcloud projects add-iam-policy-binding ${PROJECT_ID} \
  --role roles/cloudsql.client \
  --member "serviceAccount:${PROJECT_ID}@${PROJECT_ID}.iam.gserviceaccount.com"

[thebigredgeek]

thanks @kaaloo ! I'd try this, but I want to make sure to stick to documented pathways for now. I'm guessing there is a solution here, perhaps the docs are out of date?

[kurtisvg]

I'm unable to reproduce, so any additional information here would be helpful. I created a new GKE cluster (release channel - 1.16.8-gke.15) w/ Workload Identity enabled.

1. I created a KSA:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-proxy-sa

1. I bound it to an existing GSA:

gcloud iam service-accounts add-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:$MY_PROJECT.svc.id.goog[default/my-proxy-sa]" \
  $MY_GSA@$MY_PROJECT.iam.gserviceaccount.com

1. I annotated my service account:

kubectl annotate serviceaccount \
   my-proxy-sa \
   iam.gke.io/gcp-service-account=$MY_GSA@$MY_PROJECT.iam.gserviceaccount.com

1. Finally, I deployed an application:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tabs-vs-spaces
spec:
  selector:
    matchLabels:
      app: tabs-vs-spaces
  template:
    metadata:
      labels:
        app: tabs-vs-spaces
    spec:
      serviceAccountName: my-proxy-sa
      containers:
      - name: tabs-vs-spaces
        image: $MY_IMAGE
      - name: cloud-sql-proxy
        # It is recommended to use the latest version of the Cloud SQL proxy
        # Make sure to update on a regular schedule!
        image: us.gcr.io/cloudsql-docker/gce-proxy:1.17
        command:
          - "/cloud_sql_proxy"

          # If connecting from a VPC-native GKE cluster, you can use the
          # following flag to have the proxy connect over private IP
          # - "-ip_address_types=PRIVATE"

          # Replace DB_PORT with the port the proxy should listen on
          # Defaults: MySQL: 3306, Postgres: 5432, SQLServer: 1433
          - "-instances=$MY_CONNECTION_NAME=tcp:5432"
        securityContext:
          # The default Cloud SQL proxy image runs as the
          # "nonroot" user and group (uid: 65532) by default.
          runAsNonRoot: true

The logs from my sidecar show the proxy operating successfully.

[kaaloo]

It seems there is some caching of hidden state on Google Cloud. I created a new project and created everything from scratch with my scripts using a service account key instead of workload identity because in my previous project I was testing using a service account key but seeing the same error from the Cloud SQL Proxy. I'll also test workload identity again now that I now this behavior can be due to Google Cloud.

[kurtisvg]

@kaaloo This seems pretty unlikely, but the most helpful thing you could do would be to open a new issue on the IssueTracker for GKE with your project and cluster information so that our support team can investigate and gather information.

I'm going to mark this as closed since there doesn't seem to be any evidence the problem is in the proxy.

[wilhelmi]

@thebigredgeek did you make any progress on this? I am running into the same problem. I have a cluster where workload identity is working fine for a number of pods, just not the cloudsql proxy container.

[danny-molamola]

Got the same issue here. Workload identity works in my Kubernetes Config Connector set up, but got the same 401 error for Cloudsql proxy container.

2020/07/02 06:21:11 current FDs rlimit set to 1048576, wanted limit is 8500. Nothing to do here.
2020/07/02 06:21:12 errors parsing config:
googleapi: Error 401: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
More details:
Reason: authError, Message: Invalid Credentials

[kurtisvg]

Reopening for visibility since there are a couple of different reports. It seems unlikely this problem is specific to the proxy since Workload Identity is part of the ADC pattern.

Useful steps to take here if you are experiencing this issue:

1. If you are getting a different error message than "Error 401: Request had invalid authentication credentials.", please open a new issue (even if you believe the issue is related to Workload Identity)
2. Leave detailed information about your environment, so we can see if there are any common variables:
   
   
   
   • k8s version (kubectl version):
   
   
   • cloud_sql_proxy version:
   
   
3. Open a private issue on the GKE issue tracker with your project and cluster_id, and timestamps of when you encountered the issue. Filing an issue will let the support teams take a closer look at relevant logs and help escalate to the backend teams if possible.

[Jojoooo1]

Also having the same issue with similar setup.

kubectl version: Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.13-gke.1", GitCommit:"688c6543aa4b285355723f100302d80431e411cc", GitTreeState:"clean", BuildDate:"2020-07-21T02:37:26Z", GoVersion:"go1.13.9b4", Compiler:"gc", Platform:"linux/amd64"}

cloud_sql_proxy: gcr.io/cloudsql-docker/gce-proxy:1.17

[NicolasFruyAdeo]

Hi, also having the same issue.

kubectl version:
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.8", GitCommit:"ec6eb119b81be488b030e849b9e64fda4caaf33c", GitTreeState:"clean", BuildDate:"2020-03-12T21:00:06Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.13-gke.1", GitCommit:"688c6543aa4b285355723f100302d80431e411cc", GitTreeState:"clean", BuildDate:"2020-07-21T02:37:26Z", GoVersion:"go1.13.9b4", Compiler:"gc", Platform:"linux/amd64"}

cloud_sql_proxy: eu.gcr.io/cloudsql-docker/gce-proxy:1.17

[NicolasFruyAdeo]

In my case, this was a misconfiguration on my side. The Kubernetes Service Account that was binded to the Google Service Account isn't the default Kubernetes Service Account. As such, in my Kubernetes deployment file, I needed to specify the Kubernetes Service Account assigned to the pod.
I had missed this detail from the complete example configuration file in the documentation.

I hope this helps someone else.

[kurtisvg]

The Kubernetes Service Account that was binded to the Google Service Account isn't the default Kubernetes Service Account

@NicolasFruyAdeo Can you clarify what you mean by this? Is it you just missed adding the service account to your k8s object like this:

    spec:
      serviceAccountName: <YOUR-KSA-NAME>

I noticed that while the example in this project does include a a snippet of this under step 5, it's missing from the Cloud Docs version. Would this have helped you notice this step?

[NicolasFruyAdeo]

@kurtisvg, Yes exactly that.

The pages are the same, the Cloud Docs page shows that too, however it has been updated yesterday, has it just been added?
Anyway, that's what I missed, with hindsight, I don't know how I missed it...

[kurtisvg]

@NicolasFruyAdeo Thanks for following up - we did in fact push an update to the page after I noticed it was missing.

Since I've still been unable to replicate this and we haven't gotten a verifiable issue on the issue tracker, I'm hoping this was the same cause. I'm going to leave this open for another week or two, and if there aren't any more reports I'll close the issue for now.

[NicolasFruyAdeo]

It feels like it. I noticed there is no mention of the service account name in the original post either.

[Emmanuel35]

Hi
Dont forget to specify namespace. If you use another for your application, your service-account is not on default.

kubectl annotate serviceaccount \
   my-proxy-sa \
   --namespace YOURNAMESPACE \
   iam.gke.io/gcp-service-account=$MY_GSA@$MY_PROJECT.iam.gserviceaccount.com

With that, proxysql can connect to your SQL service.

[hazcod]

Does this also work with IAM authentication?
2021/04/19 09:18:22 errors parsing config: Get "https://sqladmin.googleapis.com/sql/v1beta4/projects/xxx/instances/europe-west1~mydb?alt=json&prettyPrint=false": compute: Received 403 `Unable to generate access token; IAM returned 403 Forbidden: The caller does not have permission This error could be caused by a missing IAM policy binding on the target IAM service account. For more information, refer to the Workload Identity documentation: https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#creating_a_relationship_between_ksas_and_gsas

    (I run it with `-enable_iam_login`)

[enocom]

@hazcod Yes, workload identity works with IAM authn. Let's talk more in #738.