Cloud-builders: kubectl does not support `wait`

[ernsheong]

Starting Step #9
Step #9: Already have image (with digest): gcr.io/cloud-builders/kubectl
Step #9: Running: kubectl wait --for=condition=Complete jobs/migrate --timeout=30s
Step #9: Error: unknown command "wait" for "kubectl"
Step #9: 
Step #9: Did you mean this?
Step #9: edit
Step #9: taint
Step #9: 
Step #9: Run 'kubectl --help' for usage.
Step #9: unknown command "wait" for "kubectl"
Step #9: 
Step #9: Did you mean this?
Step #9: edit
Step #9: taint
Step #9: 
Finished Step #9

Similar to https://github.com/GoogleCloudPlatform/cloud-builders/issues/313

[ernsheong]

Looks like there's the community-supported https://github.com/GoogleCloudPlatform/cloud-builders-community/tree/master/kubectl_wait_for_job as a workaround, but this was the workaround for prior to when kubectl wait existed.

[ernsheong]

Why does cloud-builders not supply non-default kubectl versions with tags?

[vinayan3]

I just used the underlying command which is

until kubectl get jobs "$@" -o jsonpath='{.status.conditions[?(@.type=="Complete")].status}' | grep True ; do sleep 1 ; done

I hope they give us the latest kubectl with wait... that'd be nice.

[imjasonh]

The kubectl image uses whatever version is made available by the Cloud SDK, which I believe is required to support the versions of Kubernetes that GKE supports, which lags a bit. We've heard this feedback from other channels and would love to find a better solution, for now bundling what gcloud provides has been expedient, but obviously there are downsides.

[ernsheong]

I wonder why we can't just do:

gcr.io/cloud-builders/kubectl:some-version-tag. This applies to all other builders too. I can't just do gcr.io/cloud-builders/go:1.11, I am pretty much guessing what I would get.

[imjasonh]

Not supporting that has been a decision within our team about the amount of support burden we're comfortable taking on. Our team is not comprised of kubectl experts and Go experts (though we write mostly Go) and Node/.Net/Java experts, though we support builders for those languages. It's hard to stay on top of each version release for every tool, and so we have decided mostly not to try to be perfect.

Lately we've been hoping to offload support for these tools to the official images themselves where possible (where those images exist and are well supported by language experts, often released alongside the official tool release).

Getting there takes some time and care and there's a lot left to do. My hope is that by the end of 2019 you won't need to care at all about the gcr.io/cloud-argo/* images, and we will have good documented support for using official builders hosted on Dockerhub.

Back on topic, kubectl is a notable exception, since AFAIK there is no official kubectl image on Dockerhub, and our image has a custom endpoint to better support GKE (since that's what most of our users use). There will need to be extra work to support kubectl well.

[ernsheong]

Thanks for the response! Staying lean and agile and not trying to be all things to all people 😃

I guess I'll leave this issue open till supported kubectl catches up, and for passers-by to know.

[imjasonh]

I found bitnami/kubectl which seems to be the most well-supported kubectl Docker image. It won't have the GKE-specific support that gcr.io/cloud-builders/kubectl has, but it could be extended to add that if you need it.

docker run bitnami/kubectl wait --help indicates that the :latest release (1.13.3-r8) includes the wait command.

[apando]

Btw, I think this has now been resolved, or at least it worked once I actually ran gcloud components update.

[fubar]

Yes, looks like the image is now running v1.11 which is where the wait command got introduced (https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md#sig-cli-1).