Service-catalog: Run e2e tests in pipeline

[jberkhahn]

Run the e2e tests in the service catalog pipeline.

@mszostok had some thoughts on how to run it all in docker

[mszostok]

Yep, I've already started some work on this.

I will post the POC results because right now I know how to do it but I also want to know the best practices so I will try to validate my approach with someone from the sig-testing. In current pipelines, jobs which are using the kind are based on gcr.io/k8s-testimages/kubekins-e2e image and then building the kind each time the job is executed. I want to optimize that and ask sig-testing to create a new image where kind will be already preinstalled and thanks to that the building phase of kind can be just skipped. That will additionally help to make the whole build faster.

/dibs

[MHBauer]

/assign @mszostok

[mszostok]

Update

I've discussed that with sig-testing and for now, they do not have an official image with preinstalled kind binary.

The final solution that I've ended up looks as follow:

1. Install kind (kubernetes-in-docker) and Helm.
2. Provision Kubernetes cluster using kind.
   kind create cluster
3. Build Service Catalog images from sources and copy them into the cluster.
   make service-catalog-image
kind load docker-image service-catalog:canary
4. Deploy Service Catalog into the cluster.
   helm install charts/catalog
5. Execute e2e tests.
   go test -v ./test/e2e/...
6. Delete Kubernetes cluster.
   kind delete cluster

cons:

• installing helm and kind causes that build is longer. But for now, it's around 10min so it still not to bad. (prow job execution)

pros:

• can be executed both on local and Prow. The dependencies in both scenarios are always the same thanks to that when e2e passing locally then on CI they should also work.

Maybe in the future, we can think about releasing our docker image with test-infra tools like kind, helm, etc.

Here is the draft pr: https://github.com/kubernetes-sigs/service-catalog/pull/2690

[MHBauer]

prow job looks like it just stops partway through. I don't see it finish, success or no success.

[mszostok]

@MHBauer when u click Raw build-log.txt then u will be redirected here: https://storage.googleapis.com/kyma-prow-logs/logs/post-test-sc-e2e-job/1162418742185955328/build-log.txt

at the end of script u will see:

SUCCESS! -- 5 Passed | 0 Failed | 0 Pending | 0 Skipped --- PASS: TestE2E (49.06s)

u can also checkout that PR: https://github.com/kubernetes-sigs/service-catalog/pull/2690

and execute make test-e2e to run them locally

[mszostok]

Note: u will also see sth like [1m in raw logs. This is an ANSI escape codes for colors. They are working only on terminals. Colors are automatically added by ginkgo and I do not know if it can be disabled. In raw logs, it makes it a little bit unreadable

[MHBauer]

ginkgo colors can be omitted with the -noColor flag.

[MHBauer]

that's weird, it works now to view both the logs. wonder what broke it the first time.

[MHBauer]

since the colors come through fine, you can leave that in, I think.

[MHBauer]

I'm happy with it. Glad that kind exists now.

[mszostok]

that's weird, it works now to view both the logs. wonder what broke it the first time.

Prow is a synonym of weird things 😄

kind is a quite new thing, on our(Kyma) pipelines we are using the minikube for the same thing but I spoke with guys on KubeCon and in k8s ecosystem kind is the recommended way of running the e2e tests. From my perspective kind is easier to install, it creates cluster faster than minikube, and allows u to create a local cluster with multi nodes (minikube is always a single node)

but in general, I wanted to be idiomatic and that is the main reason why I've chosen the kind instead of minikube.

[mszostok]

The issue will be resolved after merging that PR: https://github.com/kubernetes/test-infra/pull/13945/files (adding ci pipeline)