Cloudsql-proxy: 1.20.1 image crashes with: errors parsing config: exec: "gcloud": executable file not found in $PATH

Created on 5 Mar 2021 · 5Comments · Source: GoogleCloudPlatform/cloudsql-proxy

Bug Description

Our testing GKE cluster just pulled in the gcr.io/cloudsql-docker/gce-proxy:1.20.1-alpine image (we see the same problem with the buster variant), and all deployments are failing with:

2021/03/04 23:22:07 errors parsing config:
Get "https://sqladmin.googleapis.com/sql/v1beta4/projects/<project id>/instances/<instance>?alt=json&prettyPrint=false": exec: "gcloud": executable file not found in $PATH

Example code (or command)

Auth via workload identity.

      - name: cloud-sql-proxy
        # It is recommended to use the latest version of the Cloud SQL proxy
        # Make sure to update on a regular schedule!
        image: gcr.io/cloudsql-docker/gce-proxy:1.20.1-alpine
        command:
        - "/cloud_sql_proxy"
        # If connecting from a VPC-native GKE cluster, you can use the
        # following flag to have the proxy connect over private IP
        - "-ip_address_types=PRIVATE"
        # Replace DB_PORT with the port the proxy should listen on
        # Defaults: MySQL: 3306, Postgres: 5432, SQLServer: 1433
        - "-instances=<project>:<region>:<name>=tcp:5432"
        securityContext:
          # The default Cloud SQL proxy image is based on distroless, which
          # runs as the "nonroot" user (uid: 65534) by default.
          runAsNonRoot: true

Environment

OS type and version: GKE 1.18.15-gke.1102 with cos_containerd
Cloud SQL Proxy version (./cloud_sql_proxy -version): 1.20.1

Possibly related

#649

p1 bug

Source

mlazowik

👍6

All 5 comments

Thanks for opening the issue. We're working on a fix.

So far, I was able to reproduce the error with the distroless version of the image using Docker locally and with Cloud Shell. Haven't tried it with GKE, but when I run the following command I get the same error.

docker run -it --rm -v ~/.config/:/root/.config/ {latest image} /cloud_sql_proxy -instances={cloud sql instance name}

Using the previous version of the image works fine.

So far, we think the issue is differing error behavior between go/exec and the execabs version of exec

shubha-rajan on 5 Mar 2021

👍1

got the same error here, and I can see from my cloud build log the only difference between my previous success build is the binary file size changed for cloudsql-proxy

i'm using name: gcr.io/cloud-builders/npm:node-10.10.0 as build image, basically download the binary and start the proxy. I then used the proxy to connect to my cloud sql for database migration.

my log:

converted 'https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64' (ANSI_X3.4-1968) -> 'https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64' (UTF-8)
--2021-03-05 11:35:51--  https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64
Resolving dl.google.com (dl.google.com)... 173.194.197.93, 173.194.197.136, 173.194.197.190, ...
Connecting to dl.google.com (dl.google.com)|173.194.197.93|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14816582 (14M) [application/octet-stream]
Saving to: './ops/cloud_sql_proxy'

     0K .......... .......... .......... .......... ..........  0% 30.4M 0s
    50K .......... .......... .......... .......... ..........  0% 36.1M 0s
 14450K .......... .........                                  100% 79.5M=0.2s

2021-03-05 11:35:51 (58.5 MB/s) - './ops/cloud_sql_proxy' saved [14816582/14816582]

2021/03/05 11:35:52 current FDs rlimit set to 1048576, wanted limit is 8500. Nothing to do here.
2021/03/05 11:35:52 errors parsing config:
    Get "https://sqladmin.googleapis.com/sql/v1beta4/projects/xxxxxxx/instances/us-central1~cotreat?alt=json&prettyPrint=false": exec: "gcloud": executable file not found in $PATH

neekey on 5 Mar 2021

We suspect that #649 was a red herring, and that it was actually #648 that inadvertently caused us to start trying to rely on gcloud credentials even if the initial gcloud token fails. We've got a couple of solutions in the pipeline, but are trying to make sure we've run a couple of longer tests at the moment.