Cri-o: Converting tests from BATS to Ginkgo

[jawnsy]

Starting a discussion here. I'd like to propose converting the integration tests from BATS to Ginkgo. There are a number of benefits:

1. Easier for new contributors to get started
2. IDE support (vim-go, vscode-go, etc)
3. Fancier output messages (including colors)
4. Optional JUnit reports (would replace the existing TAP output), so no loss of functionality from a CI perspective (Jenkins can generate HTML reports from both)

[jawnsy]

Was discussing this in passing with @mrunalp the other day. Maybe we can just start an experiment alongside our current tests? I've used Ginkgo/Gomega quite successfully in the OpenShift Online team

[mrunalp]

Yeah we can try converting a few and see how it goes.

On Dec 24, 2016, at 9:21 AM, Jonathan Yu [e-mail] wrote:

Was discussing this in passing with @mrunalp the other day. Maybe we can just start an experiment alongside our current tests? I've used Ginkgo/Gomega quite successfully in the OpenShift Online team

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[runcom]

I'm not happy with bats either at this point even if it does its job quite good. I'm fine with this.

[Crazykev]

@xlgao-zju also did some similar job in k8s, you may want see this.

[jawnsy]

@Crazykev @xlgao-zju I looked through the changes in https://github.com/kubernetes/kubernetes/pull/35266 but I don't think this would replace the existing integration tests, since these exercise the command-line interface code for ocic. Are we okay with integration tests that only exercise the CRI interface, since that's the only "supported" method of using cri-o? (As far as I know, ocic is mainly for local development/testing)

[mrunalp]

I think going directly over CRI is fine. It will make this reusable for other projects as well.

On Dec 26, 2016, at 11:08 AM, Jonathan Yu [e-mail] wrote:

@Crazykev @xlgao-zju I looked through the changes in kubernetes/kubernetes#35266 but I don't think this would replace the existing integration tests, since these exercise the command-line interface code for ocic. Are we okay with integration tests that only exercise the CRI interface, since that's the only "supported" method of using cri-o? (As far as I know, ocic is mainly for local development/testing)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[Crazykev]

@jawnsy glad you find that, I am also agreed with @mrunalp . BTW, CRI validation test has been promoted to P0, I believe it will land soon.

[xlgao-zju]

Agree with @mrunalp too. As @Crazykev said, CRI validation test has been promoted to P0. And maybe it will be put in a separate repo which is reusable for other projects.

[resouer]

We talked about this during CRI-O online meeting, kubernetes sub-repos containing CRI verify tests & client tools will be created soon.

Ref: https://docs.google.com/spreadsheets/d/1-hADEbGEUrW04QP4bVk7xf1CWuqbFU6ulMAH1jqZQrU/edit?ts=585c5905#gid=1369358219

CRI validation test suite   lantaol@google, feiskyer@hyper  P0      
CRI util / tool feiskyer@hyper  P2  http://hypercontainer.io/   In a separate repo, a common tools / utils for CRI

[runcom]

should we go ahead however and start moving our tests to ginkgo? Or will be that unnecessary due to https://github.com/kubernetes-incubator/cri-o/issues/296#issuecomment-272307503? I have a branch where I started setting up a side-by-side-bats-ginkgo test suite so if this should be implemented regardless we can start with my branch I guess.

[mrunalp]

@runcom I think we should start converting anyways. We can move them upstream when ready. We can first push the ginkgo setup/teardown with a few tests and then start converting the rest.

[runcom]

@runcom I think we should start converting anyways. We can move them upstream when ready. We can first push the ginkgo setup/teardown with a few tests and then start converting the rest.

alright, I paused my work from this morning (my tz) but alright, I'll start next week again. Basically we just need a setup/teardown in ginkgo to setup the ocid server before running anything, I'll tackle that on monday.

...unless anyone was already working on this? @jawnsy want to take a stab at this? :innocent:

[mrunalp]

I spent some time looking into ginkgo today. I think k8s has a nice setup. We can do it similarly.

[jawnsy]

@runcom Yeah, absolutely, hope to give it a shot this weekend.

[xlgao-zju]

We have https://github.com/kubernetes-incubator/cri-tools now :)

[runcom]

closing in favor of kubernetes-incubator/cri-tools