Cloud-builders: kubectl image ignores env.CLOUDSDK_CONTAINER_CLUSTER when used in multiple steps

[yungchin]

This may be one of those "don't do that, then" bugs, but just in case it's helpful: if you interact with multiple GKE clusters during the same build, like so

  - name: gcr.io/cloud-builders/kubectl
    args: ['create', '-f', 'this.yaml']
    env:
      - 'CLOUDSDK_COMPUTE_ZONE=europe-west1-c'
      - 'CLOUDSDK_CONTAINER_CLUSTER=my-first-cluster'
  - name: gcr.io/cloud-builders/kubectl
    args: ['create', '-f', 'that.yaml']
    env:
      - 'CLOUDSDK_COMPUTE_ZONE=europe-west1-c'
      - 'CLOUDSDK_CONTAINER_CLUSTER=my-other-cluster'

then I currently find that that.yaml is deployed to my-first-cluster instead of my-other-cluster.

[ebrake]

I am also facing this issue: Updating CLOUDSDK_CONTAINER_CLUSTER does not get applied with multiple steps.

[Philmod]

Hey!

I made a quick test, and the environment variables seem to work as expected:

steps:
- name: gcr.io/cloud-builders/kubectl
  entrypoint: 'bash'
  args: ['-c', 'echo hello $$CLOUDSDK_CONTAINER_CLUSTER']
  env:
    - 'CLOUDSDK_CONTAINER_CLUSTER=my-first-cluster'
- name: gcr.io/cloud-builders/kubectl
  entrypoint: 'bash'
  args: ['-c', 'echo hello $$CLOUDSDK_CONTAINER_CLUSTER']
  env:
    - 'CLOUDSDK_CONTAINER_CLUSTER=my-other-cluster'

Logs:

Starting Step #0
Step #0: Already have image (with digest): gcr.io/cloud-builders/kubectl
Step #0: hello my-first-cluster
Finished Step #0
Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/kubectl
Step #1: hello my-other-cluster
Finished Step #1

[ebrake]

@Philmod Hm! Are you able to run commands against the cluster after the environment is updated? I am attempting to set an image for kubernetes in a separate cluster, but the initial CLOUDSDK_CONTAINER_CLUSTER is still being used.

[Philmod]

I am now wondering if the state is kept globally (i.e. across steps) because of this line... cc @skelterjohn

[stevencorona]

Had this same issue— the kubeconfig file gets written to the cloud build workspace directory the first time gcloud container clusters get-credentias is called and then is skipped on subsequent calls (IIRC, it was because of that line). I solved it by setting KUBECONFIG in the subsequent steps env, i.e:

steps:
- name: gcr.io/cloud-builders/kubectl
  entrypoint: 'bash'
  args: ['-c', 'echo hello $$CLOUDSDK_CONTAINER_CLUSTER']
  env:
    - 'CLOUDSDK_CONTAINER_CLUSTER=my-first-cluster'
- name: gcr.io/cloud-builders/kubectl
  entrypoint: 'bash'
  args: ['-c', 'echo hello $$CLOUDSDK_CONTAINER_CLUSTER']
  env:
    - 'CLOUDSDK_CONTAINER_CLUSTER=my-other-cluster'
    - 'KUBECONFIG=/workspace/othercluster-kubeconfig'

[ebrake]

@stevencorona Decent!!

[rajatvig]

This issue can be closed now as it is fixed.

[mercuriete]

Please reopen because this was reverted.

I cant deploy to differents clusters.

https://github.com/GoogleCloudPlatform/cloud-builders/pull/467

EDIT: I can confirm that the KUBECONFIG workaround from @stevencorona works properly.

[imjasonh]

Confirming this is reproducible:

In a project with two clusters:

 steps:
 - name: gcr.io/cloud-builders/kubectl
   env:
   - CLOUDSDK_CONTAINER_CLUSTER=standard-cluster-1
   - CLOUDSDK_COMPUTE_ZONE=<zone>
   args: ['get', 'namespaces']
 - name: gcr.io/cloud-builders/kubectl
   env:
   - CLOUDSDK_CONTAINER_CLUSTER=standard-cluster-2
   - CLOUDSDK_COMPUTE_ZONE=<zone>
   args: ['get', 'namespaces']

gcloud builds submit --no-source succeeds, but only lists namespaces in the first cluster; the second step doesn't re-get credentials for that cluster, so it lists the first cluster's namespaces again.

[imjasonh]

Actually, closing this in favor of #465 which is the real bug: context is currently persisted across steps, even if the envs are set on the second step. In fixing that bug, we also need to take care to continue to support regional clusters.

I have a PR I'll merge today, and which we'll hopefully be able to release soon.