Cloud-builders: How do I run kubectl after image push?

[DocBradfordSoftware]

It would seem that a CI/CD pipeline would require the ability to deploy an image into the gke cluster after it was built/tested/pushed.

Without this, it seems like Builder is incomplete.

[skelterjohn]

At the moment, you will have trouble using kubectl from within the container builder service, since the credentials used do not have the proper scopes. We will address this issue soon.

For pushing before running another step, you can always run docker push as its own step (either instead of or in addition to listing the image in the images field).

steps:
- name: 'gcr.io/cloud-builders/docker'
  args: ['build', '-t', 'gcr.io/$PROJECT_ID/foo', '.']
- name: 'gcr.io/cloud-builders/docker'
  args: ['push', 'gcr.io/$PROJECT_ID/foo']
- name: 'gcr.io/cloud-builders/kubectl'
  args: ['run', 'foo-pod', '--image=gcr.io/$PROJECT_ID/foo']
images: ['gcr.io/$PROJECT_ID/foo']

(for an upcoming kubectl step that is waiting on the scope issue)

[DocBradfordSoftware]

Yes, that would work. Any idea if the scopes have to be added to the cluster and will it be able to be done after the fact.
I Built the gke/jenkins integration and I had to create a new cluster with the proper scope.

[skelterjohn]

The scopes I'm referring to, which are used to create credentials for your builder service account (https://cloud.google.com/container-builder/docs/permissions), don't apply to any resources outside of the container builder service. So, scopes you use with your cluster are unrelated.

What's missing at this point is the ability to use the GKE cluster API to list your clusters and get the auth secrets necessary for talking to the cluster master. Once those secrets are acquired, you can use kubectl to talk to the cluster master directly (and schedule pods, etc).

[bussyjd]

I am very interested in this capability.
When can we expect to have the scope issue solved? Thanks a lot.

[Philmod]

The other solution is to create a Google Cloud Function that listens to the cloud-builds PubSub feed, and update the appropriate deployment of your Kubernetes cluster.

In this example, I have two repositories that map to two different deployments in a same namespace.

[DocBradfordSoftware]

Well, that's pretty cool, @Philmod, that should work in the interim.

[Philmod]

There is a second solution: store your GKE credentials in GCS that can be pulled and used as part of the build to deploy to Kubernetes. Here is how you could pull and use the credentials in the build.

[DocBradfordSoftware]

To expand on the above solution, I think these steps will work at this time.
In addition to the above link, this thread was useful Here. In particular a comment by jlowdermilk.

You begin by going to IAM > Service accounts and creating a creating a service account and downloading the json key file. ex. [email protected]

Then you add this service account to IAM and give it project editor access (the same as cloud builder, the difference is that later on this account will have the missing scopes).

1. On a machine with gcloud:

export CLOUDSDK_CONTAINER_USE_APPLICATION_DEFAULT_CREDENTIALS=true
gcloud container clusters get-credentials mycluster
kubectl config view --minify --flatten > kubeconfig

1. Copy these this file and the json key file to Cloud Storage. I removed all privleges except 'Owner' and then granted the service account ([email protected]) read access. Not that this matters because this whole process is not very secure.

2. Then add a build step to cloudbuild.yaml

- name: 'gcr.io/cloud-builders/docker'
  args: ["push", "gcr.io/$PROJECT_ID/imagename:$BRANCH_NAME-$REVISION_ID"]
- name: 'gcr.io/cloud-builders/gcloud'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    gcloud components install kubectl
    gsutil cp gs://owner_secrets/cloud_build/$PROJECT_ID-cloudbuild.json .
    gsutil cp gs://owner_secrets/cloud_build/$PROJECT_ID-kubeconfig .
    export GOOGLE_APPLICATION_CREDENTIALS=$PROJECT_ID-cloudbuild.json
    export KUBECONFIG=$PROJECT_ID-kubeconfig
    kubectl set image deployment/deploy_name imagename=gcr.io/$PROJECT_ID/imagename:$BRANCH_NAME-$REVISION_ID --namespace=$BRANCH_NAME

That seemed to work for me at this time.

[hamx0r]

I found this example pretty helpful too. I'm using a custom cloudbuild.yaml which builds and pushes using gcr.io/cloud-builders/docker (a la @skelterjohn 's suggestion), and then copies my kubernetes objects (yaml files) and kubeconfig file (also yaml) to a Bucket, and then from there to /workspace. Now, I can use gcr.io/google_containers/hyperkube:v1.6.1 build steps to run kubectl commands where my credentials are available per the kubeconfig file. This snippet assumes all my kubeconfig and Kubernetes object YAML files are in a "kubernetes" folder/bucket:

- name: gcr.io/cloud-builders/gsutil
  args: ['cp', 'kubernetes/*', 'gs://${PROJECT_ID}/kubernetes']
  id: 'copy-local'
  waitFor: ['-']
- name: gcr.io/cloud-builders/gsutil
  args: ["cp", "gs://${PROJECT_ID}/kubernetes/*", "/workspace/"]
  id: 'copy-workspace'
  waitFor: ['copy-local']

- name: "gcr.io/google_containers/hyperkube:v1.6.1"
  env: ["KUBECONFIG=/workspace/kubeconfig"]
  entrypoint: "/hyperkube"
  args: ["kubectl", "apply", "-f", "/workspace/worker-deployment.yaml"]
  id: 'config-worker'
  waitFor: ['copy-workspace']
- name: "gcr.io/google_containers/hyperkube:v1.6.1"
  env: ["KUBECONFIG=/workspace/kubeconfig"]
  entrypoint: "/hyperkube"
  args: ["kubectl", "set", "image", "deployment", "worker", "worker=gcr.io/myrepo/worker"]
  waitFor: ['config-worker', 'push-worker']

[Philmod]

Hey!

Good news, the up-scoping feature is now live.

That means that you can add permission for Container Engine to your Cloudbuild service account, and then you can use kubectl directly in your build step:

steps:
- name: 'gcr.io/cloud-builders/docker'
  args: ["build", "-t", "gcr.io/my-project/frontend", "."]
- name: 'gcr.io/cloud-builders/docker'
  args: ["push", "gcr.io/my-project/frontend"]
- name: 'gcr.io/cloud-builders/gcloud'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    gcloud components install kubectl
    gcloud container clusters get-credentials <my-cluster> --zone <my-zone>
    kubectl set image deployment/frontend frontend=gcr.io/my-project/frontend

Enjoy!

[dlsniper]

Awesome! Thank you!

[DocBradfordSoftware]

This is good news, Thanks.

I assume that the minimum permission to add would be "Container Engine Developer" or is there something else?

[Philmod]

Depends what you try to do, but for the example I posted, "Container Engine Developer" permission is indeed sufficient.

[DocBradfordSoftware]

If we wanted to use hyperkube, because it was a lot faster, would the kubeconfig be located at /root/.kube/config?

- name: "gcr.io/google_containers/hyperkube:v1.6.1"
  env: ["KUBECONFIG=/root/.kube/config"]
  entrypoint: "/hyperkube"
  args: ["kubectl", "apply", "-f", "/workspace/worker-deployment.yaml"]

[Philmod]

I'm not sure about hyperkube.

But I would guess you still need to authenticate to your cluster somehow. That's why I'm using:
gcloud container clusters get-credentials <my-cluster> --zone <my-zone>

[skelterjohn]

hyperkube is the binary you use to run a local kubernetes cluster, right? In that case I bet it doesn't need authentication. If it does, the gcloud container clusters get-credentials command won't be able to fetch it - that command only talks to the GKE service which won't know about your hyperkube instance.

Regarding the example step you pasted, is that how hyperkube works? Do you really run hyperkube kubectrl ...? I suspect you run hyperkube by itself, and then run kubectl in another shell/process.

I don't have the time to try this myself, but a starting point might be...

- name: 'gcr.io/cloud-builders/docker'
  args: ['run', '-d', '--name=hyperkube', 'gcr.io/google_containers/hyperkube:v1.6.1']
- name: 'gcr.io/cloud-builders/gcloud'
  entrypoint: 'bash'
  args:
  - -c
  - |
    gcloud components update kubectl -q
    kubectl ....

I don't know how you tell kubectl to talk to the hyperkube, so you'd need to add that in the script in the second step.

[hamx0r]

@DocBradfordSoftware @skelterjohn If you refer to my comment from on Apr 27, I link to a full working example of how to pass credentials to hypercube. It's done by passing a YAML file to hyperkube via Environment variable:

env: ["KUBECONFIG=/workspace/kubeconfig"]

[skelterjohn]

Very nice, thank you. It looks like for hyperkube the right thing to do is indeed to run kubectl from within that image.

Though, in your example I don't understand why you first copy into GCS and then into /workspace. Why not cp kubernetes/* /workspace? Or better yet, forget the copy and set KUBECONFIG=/workspace/kubernetes/kubeconfig?

[DocBradfordSoftware]

@skelterjohn the reason we copy from gs to /workspace is because ~/.kube/config doesn't exist in the image. and until yesterday

gcloud container clusters get-credentials <my-cluster> --zone <my-zone>

wouldn't get the correct credentials to modify our deployments.
So now we just have to find .kube/config in order to pass it to hyperkube.

[byates-va]

This is really awesome @Philmod @skelterjohn. I had been using credentials mounted via GCS in the past but that was complicated to get setup, this makes it a breeze.

Are there any plans to add kubectl to the components installed by default on the cloud builder gcloud image, or any other image containing kubectl that is in your image cache? That would keep things nice and speedy, this approach is a bit slow having to always install kubectl.

The one thing that is out of your control that is needed next is finer grained IAM control for the service account to kubernetes, for example to make sure this service account can only access a certain namespace. We understand that is coming (maybe not exactly as I described, but some level of finer grained control), and once that is in IAM this approach should be able to simply use it.

[bendory]

Yes, there are plans for this, landing soon...

[hamx0r]

For those curious, I compared CloudBuild times with hyperkube vs Philmod's up-scoped implementation. Hyperkube is about 1min faster for me.

My build does this:

1. Builds 4 Docker images
2. Pushes all 4
3. Updates kubernetes config for only 2 of them
4. Sets Kubernets image for those same 2

It is written so the 4 build/push tasks happen in parallel, using waitFor . It takes ~1:15 with hyperkube (which pulls the hyperkube image twice in parallel). Using kubectl directly with the new implementation takes ~2:15 and installs kubectl twice (in parallel).

On the upside, the new method results in a much more concise cloudbuild.yaml as I can combine the kubectl apply... and kubectl set image... lines in the same build step, and completely remove the steps which copy repo files to the bucket then again to the local workspace.

[brtnshrdr]

In case people are curious, here is my cloudbuild.yaml file that builds my image, pushes it to docker registry, then updates kubernetes:

My kubernetes file references my image like this:
- image: gcr.io/my-google-project/my-project:${TAG}
And is then set up with the following cloudbuild.yaml

steps:
- name: 'gcr.io/$PROJECT_ID/envsubst'
  args: ['/workspace/kubernetes.yaml', '/workspace/kubernetes.yaml']
  env:
  - 'TAG=$TAG_NAME'
- name: 'gcr.io/cloud-builders/docker'
  args: ['build', '--build-arg', 'PROJECT_ID=$PROJECT_ID', '--cache-from', 'gcr.io/$PROJECT_ID/my-project:latest', '-t', 'gcr.io/$PROJECT_ID/my-project:$TAG_NAME', '-t', 'gcr.io/$PROJECT_ID/my-project:latest', '.']
- name: 'gcr.io/cloud-builders/docker'
  args: ['push', 'gcr.io/$PROJECT_ID/my-project:$TAG_NAME']
- name: 'gcr.io/cloud-builders/kubectl'
  env: ['CLOUDSDK_COMPUTE_ZONE=us-central1-a', 'CLOUDSDK_CONTAINER_CLUSTER=my-cluster']
  args: ['apply', '-f', '/workspace/kubernetes.yaml']
images: ['gcr.io/$PROJECT_ID/my-project:latest']

The envsubst Dockerfile:

FROM alpine:3.6
MAINTAINER Nick Richardson <[email protected]>
RUN apk --update add gettext-dev

ADD envsubst-file.sh /
RUN chmod +x /envsubst-file.sh

ENTRYPOINT ["/envsubst-file.sh"]

The envsubst-file.sh:

#!/bin/sh

FILENAME=$1
FILENAME_NEW=$2

if [ ! $FILENAME ]
then
  echo 'No filename argument provided'
  exit -1
fi
if [ ! $FILENAME2 ]
then
  FILENAME_NEW=$FILENAME
fi
echo "Processing $FILENAME ..."

envsubst < $FILENAME > $FILENAME-bak
echo "Finished modifying file"
mv $FILENAME-bak $FILENAME_NEW

[mixerka]

@brtnshrdr Thanks, it works for me.

You should remove images section from cloudbuild.yaml:
images: ['gcr.io/$PROJECT_ID/my-project:latest']
because you push image manually. With that section push will be triggered twice.

[brtnshrdr]

@mixerka I push both the $TAG_NAME and the LATEST image. I mostly push the latest image so it shows up in the cloud builder console. I use kubernetes with the TAG_NAME image, but use the LATEST image just for information purposes.

[mixerka]

@brtnshrdr Oh, I get it now, thanks.

[skelterjohn]

It's reasonable to leave the images field complete, even when you do push with explicit build steps, because it will be very quick (since all the layers are already uploaded) and it will cause cloudbuild to track the digests of the pushed images.

[rimusz]

Yes, I do leave them too, so I can see images in slack notification as well.

[nick4fake]

It is really weird that this extremely basic feature is missing from in the GCB. I always thought that cloud builder is best suited for to build and publish to GKE.

[mixerka]

@nick4fake why do you say something is missing? What do you mean? GCP is decent tool to deploy to kubernetes (GKE or not).