Cloudsql-proxy: "Set in config.json failed: permission denied" on container startup

[MB-Jan]

Bug Description

On startup of the container I receive the following error:
Error: failed to start container "myapp-sql": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "chdir to cwd (\"/home/nonroot\") set in config.json failed: permission denied": unknown

This is the container configuration that I use in my kubernetes network:

- name: "myapp-sql"
          image: asia.gcr.io/cloudsql-docker/gce-proxy:1.17
          imagePullPolicy: Always
          command:
          - /cloud_sql_proxy
          - -dir=/cloudsql
          - -instances=myinstance:myinstance=tcp:0.0.0.0:3306
          - -credential_file=/credentials/credentials.json
          - -term_timeout=10s
          ports:
          - name: port-myapp-sg-db
            containerPort: 3306
          volumeMounts:
          - mountPath: /cloudsql
            name: cloudsql
          - mountPath: /credentials
            name: service-account-token

Before the error described above, I see this in the Kubernetes event list

Successfully pulled image "asia.gcr.io/cloudsql-docker/gce-proxy:1.17

How to reproduce

This error appears in the event log when I apply the configuration with:

kubectl apply -f kubernetes/mydeployment.yaml

I got the event log with

kubectl get events --sort-by=.metadata.creationTimestamp

Environment

1. OS type and version: MacOS locally and on GKE Container-Optimized OS (cos)

1. Cloud SQL Proxy version (./cloud_sql_proxy -version): 1.17

[philipsparrow]

Yes I've been seeing this also. My workaround is to allow the container to run as root.

This is not a good solution - there should be no reason why cloudsql-proxy needs to run as root (or for it to need a writable root filesystem (which I think was also an issue here) and it's bad from a security standpoint where people have a PodSecurityPolicy to prevent containers running as root

[kurtisvg]

Trying to reproduce now. Just to confirm - this works fine with 1.16 and just started with 1.17?

[philipsparrow]

Trying to reproduce now. Just to confirm - this works fine with 1.16 and just started with 1.17?

Yes that's correct.
I think it's because the container needs to be run as the nonroot user, which I figured out (from https://github.com/GoogleContainerTools/distroless/issues/235#issuecomment-503733486) is user ID: 65532

I tried setting the container to runAsNonRoot and it failed with the following error:
Error: container has runAsNonRoot and image has non-numeric user (nonroot), cannot verify user is non-root
So my cleanest workaround so far is to specify runAsUser: 65532 in the container's securityContext

[philipsparrow]

I've just heard anecdotal evidence that this issue is in GKE 1.14 and not in 1.15. Some of our clusters are a little behind

[kurtisvg]

I'm unable to replicate, even in GKE 1.14. Trying a few different things, but currently:

K8s versions:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.10-dispatcher", GitCommit:"f5757a1dee5a89cc5e29cd7159076648bf21a02b", GitTreeState:"clean", BuildDate:"2020-02-06T03:29:33Z", GoVersion:"go1.12.12b4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.10-gke.27", GitCommit:"145f9e21a4515947d6fb10819e5a336aff1b6959", GitTreeState:"clean", BuildDate:"2020-02-21T18:01:40Z", GoVersion:"go1.12.12b4", Compiler:"gc", Platform:"linux/amd64"}

Current deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: tabs-vs-spaces
  labels:
    app: tabs-vs-spaces
spec:
  template:
    metadata:
      labels:
        app: tabs-vs-spaces
    spec:
      containers:
        - name: tabs-vs-spaces
          image: MY_IMAGE
          ports:
            - containerPort: 8080
        - name: cloudsql-proxy
          image: asia.gcr.io/cloudsql-docker/gce-proxy:1.17
          command: ["/cloud_sql_proxy",
                    "-instances=MY_INSTANCE_NAME=tcp:5432",
                    "-credential_file=/secrets/cloudsql/proxy-acct.json"]
          volumeMounts:
            - name: my-secrets-volume
              mountPath: /secrets/cloudsql
              readOnly: true
      volumes:
        - name: my-secrets-volume
          secret:
            secretName: csql-proxy-acct

@MB-Jan Are you setting a custom user or anything that isn't down in your manifest?

@philipsparrow Did you have a user specified in the security context before this? There is a nonroot user set in the Dockerfile, which should be used by default.

[philipsparrow]

My apologies @kurtisvg I do have a user set in the podSecurityContext:

      securityContext:
        runAsUser: 2000
        fsGroup: 2000

So the container is trying to use UID 2000 which conflicts with the image's use of UID 65532.
This wasn't an issue before so I presume there's something about the new base image that conflicts.

[kurtisvg]

My apologies @kurtisvg I do have a user set in the podSecurityContext:

      securityContext:
        runAsUser: 2000
        fsGroup: 2000

So the container is trying to use UID 2000 which conflicts with the image's use of UID 65532.
This wasn't an issue before so I presume there's something about the new base image that conflicts.

We did switch to using the nonroot distroless base image in #321. The goal was to provide this security for folks by default :)

I actually didn't realize this would cause issues for users already using non-root, so I'll update the Releases page with a quick note.

[philipsparrow]

Thanks. I'm not sure why this requires a specific user ID to be set, I doubt there's anything in /home/nonroot that needs to be accessed but the workaround of specifying the userID doesn't seem unreasonable to me if it's documented (although I do see it as a workaround since cloudsql probably shouldn't care what user it is run as).

[kurtisvg]

Thanks. I'm not sure why this requires a specific user ID to be set, I doubt there's anything in /home/nonroot that needs to be accessed but the workaround of specifying the userID doesn't seem unreasonable to me if it's documented (although I do see it as a workaround since cloudsql probably shouldn't care what user it is run as).

So to be clear, if you _don't_ specify a user, it also uses nonroot right? So surely it's not really a workaround if the default behavior is secure?

If not I'm happy to consider reverting back to base.

[MB-Jan]

@kurtisvg Thank you for your swift response. I certainly did not expect this so quickly!
I have the following context in my manifest and assume that it is the same reason what @philipsparrow is experiencing.

securityContext:
        runAsUser: 65534
        fsGroup: 65534

I temporarily tested it without this security context and the error seems to be gone, but I need it for my network to function well. It would be amazing if we can get this working with custom users and groups. For now, I am sticking with version 1.16, which works fine.

Thank you!

[kurtisvg]

@MB-Jan Can you provide some more context on why you need a specific user for your network to function well? Why can't you just use the default nonroot account for the proxy container?

[philipsparrow]

Thanks. I'm not sure why this requires a specific user ID to be set, I doubt there's anything in /home/nonroot that needs to be accessed but the workaround of specifying the userID doesn't seem unreasonable to me if it's documented (although I do see it as a workaround since cloudsql probably shouldn't care what user it is run as).

So to be clear, if you _don't_ specify a user, it also uses nonroot right? So surely it's not really a workaround if the default behavior is secure?

The default behaviour is secure, however if you specify

      securityContext:
        runAsNonRoot: true

Then Kubernetes doesn't know that nonroot user is not the name of the root user with ID 0. So K8s only trusts user IDs rather than user names. So the only way to use the new version securely is to specify runAsUser: 65532 in the container's securityContext. I think this may be a limitation with the base image, or perhaps your image is specifying user nonroot somewhere.

@MB-Jan I'm curious to understand why you'd need a particular linux user for your network (the answer is off-topic, but I'm still curious) but could you set the podSecurityContext to

securityContext:
        runAsUser: 65534
        fsGroup: 65534

and then override the user ID in the SQL proxy by setting the container securityContext to:

        securityContext:
          runAsNonRoot: true
          runAsUser: 65532
          runAsGroup: 65534

Or does the user of this particular container also need to be ID 65534?

[kurtisvg]

@philipsparrow Thanks for the additional context. I was able to replicate a similar error just by adding runAsNonRoot: true. I agree this probably isn't ideal - I think there are a couple of different potential options to make the situation better. I opened #386 with a summary - please feel free to provide any additional feedback on the potential solutions.

[MB-Jan]

@MB-Jan I'm curious to understand why you'd need a particular linux user for your network (the answer is off-topic, but I'm still curious) but could you set the podSecurityContext to

@philipsparrow Thank you for your thorough feedback. I am still a Kubernetes novice, but when I tried a different user and group, one of the commands that I run for one other (custom) image failed to execute. However, I was unaware that I can specify the securityContext on a container level. Your suggestion to add the following code to the Cloud SQL proxy container worked perfectly:

securityContext:
          runAsNonRoot: true
          runAsUser: 65532
          runAsGroup: 65534

Thank you very much! From my end this topic can be marked as resolved and/or closed.