Brigade: Determine minimum compatible version of Kubernetes

[krancour]

We've used 1.18 - 1.20 throughout Brigade 2 development, but we don't rightfully know what the minimum compatible version of k8s is.

This issue only calls for some research and experimentation to make that determination.

@willie-yao this might be something you can play with as time permits.

cc: @carolynvs-- thanks for raising this issue.

[willie-yao]

k8s 1.17.0 keeps crashing 20 minutes in

API Server logs:
2021/06/09 16:53:45 Starting Brigade API Server -- version v2.0.0-alpha.5 -- commit 78d30cf
2021/06/09 16:53:50 error adding indexes to events collection: context deadline exceeded

Scheduler logs:
2021/06/09 16:55:00 Starting Brigade Scheduler -- version v2.0.0-alpha.5 -- commit 78d30cf panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x6165a5]

Observer logs:
2021/06/09 16:55:01 Starting Brigade Observer -- version v2.0.0-alpha.5 -- commit 78d30cf 2021/06/09 16:56:39 error checking Brigade API server connectivity: error invoking API: Get "https://brigade2-apiserver.brigade2.svc.cluster.local/v2/ping": GET https://brigade2-apiserver.brigade2.svc.cluster.local/v2/ping giving up after 5 attempt(s): Get "https://brigade2-apiserver.brigade2.svc.cluster.local/v2/ping": dial tcp: lookup brigade2-apiserver.brigade2.svc.cluster.local on 10.96.0.10:53: read udp 10.244.0.5:44701->10.96.0.10:53: read: connection refused

The issues seem to be caused by the scheduler

EDIT: The same issue persists in version 1.16.1

[willie-yao]

1.18.0 works fine

[willie-yao]

1.17.5 works fine

[willie-yao]

1.17.2 results in infinite crashing loop, making 1.17.5 the oldest kind node version on dockerhub to work with brigade

API server logs:
2021/06/09 16:38:54 Starting Brigade API Server -- version v2.0.0-alpha.5 -- commit 78d30cf
2021/06/09 16:38:59 error adding indexes to events collection: context deadline exceeded

Scheduler logs:
\2021/06/09 16:39:46 Starting Brigade Scheduler -- version v2.0.0-alpha.5 -- commit 78d30cf panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x6165a5]

[krancour]

@willie-yao can you please get logs from whatever's crashing on 1.17.0 and 1.17.2? It would be good if we could pinpoint why we're incompatible. It's also possible the logs reveal something that's easily fixed to ensure broader compatibility.

[krancour]

After more research, coredns wasn't coming up clean when using a newer kind CLI and older kind image.

For such scenarios, this hack moved us past this issue, presenting us with a healthy single-node k8s cluster on which to test Brigade 2:

https://stackoverflow.com/questions/53075796/coredns-pods-have-crashloopbackoff-or-error-state/53414041#53414041

We've already found Brigade 2 works fine on a healthy k8s 1.17.2 and @willie-yao will resume the hunt for an older version of k8s that we are legitimately not compatible with.

[willie-yao]

As an update to the previous comment, using this hack actually caused a dns issue when running a job with brigade. I resumed testing with minikube instead of kind, which has better support for older k8s versions.

The last compatible k8s version is 1.16.0, as using anything below 1.16.0 causes this issue when installing the helm chart for Brigade2:
Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(Deployment.spec.template.spec.containers[0]): unknown field "startupProbe" in io.k8s.api.core.v1.Container

This is because startupProbes were added in k8s version 1.16, thus making 1.16 the minimum compatible version of k8s for Brigade2. cc @krancour to close the issue

[krancour]

Thanks for doing the legwork on this @willie-yao.

I agree that 1.16.0 is the minimum version that supports Brigade 2-- unless we were to give up on using the startup probe, which I do not want to do.