Community: Add docs on running specific node test suite

[karan]

I'm trying to run a specific test suite remote.

For ex if I want to run https://testgrid.k8s.io/sig-node-containerd#containerd-node-features, I can't figure out how to run just that. The prow job for the test has --test_args=--nodes=8 --focus="\[NodeFeature:.+\]" --skip="\[Flaky\]|\[Serial\]" but those seem to run other tests (make test-e2e-node REMOTE=true FOCUS="\[NodeFeature:.+\]" SKIP="\[Flaky\]|\[Serial\]")

We should document better what exactly FOCUS is and what its relation to test suites is. Examples of running specific test suites locally/remotely would be helpful as well.

[karan]

/sig node

/area node-e2e

/kind documentation

[MHBauer]

I think this is one of the special tests that requires a gcloud setup to run vms.

Looking at the prowjob config, I see the image-config-file used, which contains a list of images to run the tests in.

[karan]

I think image is unrelated to the question here. Unfortunately, https://github.com/contributing-to-kubernetes/gnosis/tree/master/stories/e2e-node-tests don't explain how focus works either.

[MHBauer]

Okay, my shot at it, the yaml definition of the job
we see --test_args=--nodes=8 --focus="\[NodeFeature:.+\]" --skip="\[Flaky\]|\[Serial\]"
that would be PARALLELISM=8 FOCUS="\[NodeFeature:.+\]" SKIP="\[Flaky\]|\[Serial\]"

I'll try running it and report back.

[MHBauer]

I'm going to make a pr to try and improve the doc at https://github.com/kubernetes/community/blob/master/contributors/devel/sig-node/e2e-node-tests.md

[MHBauer]

I blather on for quite a bit, but I got some good ideas on improvements to that doc, and other docs.
https://youtu.be/aoqnFd1Wd9E?t=1355
The focus + skip runs the same 34 tests as seen on the server.

[alejandrox1]

/assign
Couple things we should document and i'll give a quick explanationhere..

Starting with https://testgrid.k8s.io/sig-node-containerd#containerd-node-features
This is an e2e [prow] job.
An e2e job runs e2e tests.
Each [gray] rowon testgrid corresponds to an e2e test or a stage of the e2e job (e.g., created a VM, downloaded some files).

All e2e tests are writen using Ginkgo and that is where the FOCUS and SKIP come into play, http://onsi.github.io/ginkgo/ see "The Spec Runner" in those docs.

There is additional complexity with node e2e tests because the image configuration files DO matter.
For example

--focus="\[NodeFeature:.+\]" --skip="\[Flaky\]|\[Serial\]"

RUns all e2e tests labeled with "[NodeFeature:*]" and would skip any tests labeled with "[Flaky]" or "[Serial]".
Two examples of the e2e tests that would run are

• https://github.com/kubernetes/kubernetes/blob/0e2220b4462130ae8a22ed657e8979f7844e22c1/test/e2e_node/security_context_test.go#L155
• https://github.com/kubernetes/kubernetes/blob/0e2220b4462130ae8a22ed657e8979f7844e22c1/test/e2e_node/security_context_test.go#L175

Smetimes, image configuration files will further chose tests based on their tests filed.
For example, https://github.com/kubernetes/test-infra/blob/4572dc3bf92e70f572e55e7ac1be643bdf6b2566/jobs/e2e_node/benchmark-config.yaml#L23

For https://testgrid.k8s.io/sig-node-containerd#containerd-node-features this is not the case because the image config used in the prow job doesn't specify any particular tests https://github.com/kubernetes/test-infra/blob/master/jobs/e2e_node/image-config.yaml (prow job config https://prow.k8s.io/prowjob?prowjob=ac6647f7-a98f-11ea-83f2-268265686cc2).

---

Example time, to run one particular e2e tests, for example https://github.com/kubernetes/kubernetes/blob/0e2220b4462130ae8a22ed657e8979f7844e22c1/test/e2e_node/security_context_test.go#L175 ,one could do

--focus="should not show its pid in the non-hostpid containers \[NodeFeature:HostAccess\]" --skip="\[Flaky\]|\[Serial\]"

To run all node e2e tests related to the "HostAccess" feature one could similarly do

--focus="\[NodeFeature:HostAccess\]" --skip="\[Flaky\]|\[Serial\]"

---

If you wanna see an example that dives deeper into how all that works this is the post to read https://github.com/contributing-to-kubernetes/gnosis/tree/master/stories/e2e-tests

[alejandrox1]

I think this would be a good issue for new contributors.
I'm going to save it for new contributors to give this a try :smiley:

/help wanted
/good-first-issue

[kscharm]

Hi @alejandrox1 I'm a new contributor to k8s and would like to take this issue. I'm not too familiar with the testgrid framework, but I think this would be a good learning experience to get my feet wet.

[kscharm]

/assign

[alejandrox1]

/remove-help
/lifecycle active