Skaffold: kaniko builds should work with GKE workload identity

[jlewi]

Expected behavior

My expectation is that skaffold kaniko builds would work with GKE workload identity

Actual behavior

skaffold kaniko pods end up setting the environment variable GOOGLE_APPLICATION_CREDENTIALS.

As a result, kaniko pods will not try to use workload identity but instead look for the GCP secret to be provided in the location specified by the secret.

Information

• Skaffold version: v1.1.1
• Operating system: ...
• Contents of skaffold.yaml:

apiVersion: skaffold/v2alpha1
kind: Config
metadata:
  name: label-microservice
build:
  artifacts:
  - image: gcr.io/issue-label-bot-dev/bot-worker
    # Set the context to the root directory. 
    # All paths in the Dockerfile should be relative to this one.
    context: ..
    # Automatically sync python files to the container. This should avoid
    # the need to rebuild and redeploy when the files change.
    # TODO(https://github.com/GoogleContainerTools/skaffold/issues/3448): We use manual sync
    # because inferred sync doesn't work
    #
    # This only works if we autorestart the program on changes.
    sync:
        manual:
        - src: 'py/code_intelligence/*.py'
          dest: '/'
        - src: 'py/label_microservice/*.py'
          dest: '/'
    kaniko:
      dockerfile: Label_Microservice/deployment/Dockerfile.worker
      buildContext:
        gcsBucket: issue-label-bot-dev_skaffold-kaniko
      cache: {}
  cluster:    
    # TODO(jlewi): This should be changed for each developer; or maybe we should create a reusable one?
    namespace: kaniko
    resources:
      requests:
        cpu: 8
        memory: 16Gi

deploy:
  kustomize:
    path: deployment/overlays/dev

Steps to reproduce the behavior

1. a clonable repository with the sample skaffold project
2. skaffold <command>
3. ...

[tejal29]

@jlewi Thanks for opening this issue. At this time, we don't have a clear understanding of workload identity. I will look into what (if any) pod config is required for kaniko pod and then follow up here in skaffold.

Thanks

[jlewi]

@tejal29 Thanks. If you are using Google Cloud Client libraries to get credentials you shouldn't need to do anything. The client libraries should attempt to contact the metadata server to get credentials if GOOGLE_APPLICATION_CREDENTIALS isn't set.

[balopat]

This should be relatively easy to implement.
We can add a skipGcpCredentials flag to build.artifact.kaniko.

[tstromberg]

@balopat - seems reasonable. Alternatively, if we just remove GOOGLE_APPLICATION_CREDENTIALS from kaniko, shouldn't it do the right thing with newer libraries?

[chanseokoh]

I think you still want GOOGLE_APPLICATION_CREDENTIALS for a way to supply GCR credentials on a non-GKE cluster?

And does kaniko use Google Cloud Client libraries to get credentials?

[nkubala]

cc @tejal29 do you know? also, is anyone interested in fixing this one?

[jlewi]

Workload identity also depends on being able to set the K8s service account in order to control the GCP identity that is used.

On GKE/GCP it is increasingly common to disallow exporting of service account keys. So this is becoming a blocker.

[jlewi]

Per: GoogleContainerTools/kaniko#968 kaniko works just fine with workload identity. So the issue is just properly configuring the kaniko pod so that WI works. Specifically

1. Don't set GOOGLE_APPLICATION_CREDENTIALS
2. Make the K8s service account configurable

[jlewi]

I just tried using the latest v2beta6 spec (https://skaffold.dev/docs/references/yaml/)
And skaffold 1.13.2

It looks like we can set the K8s service account on the pod.

To disable GOOGLE_APPLICATION_CREDENTIALS; it looks like we could just add an if statement here
https://github.com/GoogleContainerTools/skaffold/blob/46529503e1e8e754644895ccd06bd6aea8cbc128/pkg/skaffold/build/cluster/pod.go#L132
to not set if pullSecretPath is the empty string.

Likewise I think we could add an if statement to addSecretVolume to not add the secret volume if the secret is the empty string.

https://github.com/GoogleContainerTools/skaffold/blob/46529503e1e8e754644895ccd06bd6aea8cbc128/pkg/skaffold/build/cluster/pod.go#L164

It looks like there might a new code path related to using Tekton
https://github.com/GoogleContainerTools/skaffold/blob/e756368334d0d997a6c7c295b2c7389e31588dcb/pkg/skaffold/generate_pipeline/tasks.go#L87

• It looks like the Kaniko secret is currently hard coded in that case.