Che: Add ability to create OpenShift-specific selenium tests of Eclipse Che

[dmytro-ndp]

Description

Provide @Openshift type-level annotation to mark _OpenShift_-specific selenium test classes, and --openshift parameter of webdriver.sh to include such tests into the test execution.
_Docker_-specific selenium tests will be executed by default.

This issue is an implementation of requests https://github.com/eclipse/che/issues/7761 and https://github.com/eclipse/che/issues/7669.

UPDATE: it was decided to use _TestNG groups_ mechanism to mark infrastructure-specific tests instead the annotations as more flexible and simple.

OS and version:
Eclipse Che 6.x

[garagatyi]

@dmytro-ndp @riuvshin @skabashnyuk @tolusha let's consider other options. If we add --openshift we might want to add --k8s, --k8s-moby, etc at some point. This is something that doesn't scale.
On the other hand, the idea that we can annotate tests and apply a filter based on the annotation is useful.
Want I want to suggest is to create an annotation based filter to include/exclude tests. This may provide a lot of opportunities, such as:

1. Allow to reuse Che tests for a product build on top of Che but which may remove/add/replace some parts of Che.
2. Run some subsets of tests to do a quick test of a certain area of Che.
3. Include/exclude some test, based on a certain setup of Che. This use-case can cover the case described in the description of this issue.

If we do that we can mark issues with different annotations and play with filters to cover all described cases.
Examples:
./webdriver.sh --filter=+ws_lifecycle to check WS start/stop use cases.
./webdriver.sh --filter=+all,-openshift_infra to check all tests, except those that are specific to OpenShift infrastructure. Here +all is a special tag that includes all the tests.
./webdriver.sh --filter=+git to check git functionality
./webdriver.sh --filter=+git,+github to check Git and Github related functionality
As an improvement, we can add parenthesis to filters syntax in the future.

[tolusha]

@garagatyi
I guess the most easiest way here is to run tests inside package
./webdriver.sh --test=org.eclipse.che.selenium.debugger.** for debugger tests
./webdriver.sh --test=org.eclipse.che.selenium.git.** for git tests

[skabashnyuk]

+1 for packages

[tolusha]

@dmytro-ndp
Can we run suite instead of adding a new annotation?

OpenshiftSuite.xml

  ...
  open shift specific tests
  ...

  <suite-files>
    <!-- that is the path where sub-suite file being looked at -->
    <suite-file path="src/test/resources/suites/CheSuite.xml"/>
  </suite-files>

[dmytro-ndp]

@tolusha this approach is certainly the simplest way to INCLUDE tests in the test execution if we have only one configuration to filter the tests. Unfortunately, we have already had different Che configs with different test sets: openshift-specific, docker-specific, multiuser-specific, ets. And we can't provide common suite _CheSuite.xml_ without duplications among the _config-specific_ suites.

But there is another solution possible:

• to have single _CheSuite.xml_ for all the Che configs;
• to mark tests which are specific for certain Che config by special type-level annotation @RequiredCheConfig(CheConfigParameter[]), where
  CheConfigParameter is enum with values {OPENSHIFT, MULTIUSER, DOCKER, SINGLEUSER, ...};
• to define config of being tested Che by using specific parameters of _webdriver.sh_: --openshift, --multiuser, ..., where default Che config parameters are (docker, singleuser);
• to exclude tests, which are not relevant to current config parameters, from the test execution at the start of testing, as it has already been done in CheSeleniumTestHandler class for _Mutliuser Che_ specific tests marked by @Multiuser annotation.

[tolusha]

My suggestions:

public @interface RequiredTestEnvironemnt {
    Infrastructure infrastructure() default Infrastructure.DOCKER;
    boolean singeuser() default true;
}

[dmytro-ndp]

_Test Environment_ is not the same as _configuration of product to being tested_.
RequiredTestEnvironment annotation from example above could be treated as tests requirement to have _selenium webdriver_ be run in docker infrastructure and with single test user only. That is why we need to have strict separation of _test environment_ from _configuration of product to be tested_, and taking it in mind example above could be rewrote as follow:

public @interface RequiredCheConfiguration {
    CheInfrastructure infrastructure() default CheInfrastructure.DOCKER;
    boolean singeuser() default true;
}

[skabashnyuk]

@tolusha Can we use testng groups for that?

[tolusha]

I think no.

[tolusha]

@dmytro-ndp
If we use RequiredCheConfiguration in Codenvy tests it will look weird to see Che prefix.
It is not a Che configuration, it is an environment where Che is run.

[dmytro-ndp]

@tolusha: we should differ selenium tests itself configuration from the configuration of product which selenium tests executes on. Our tests package path starts from "org.eclipse.che", as well as they have been situated inside the Eclipse Che repo :-)
Anyway, we could use Product word instead of Che in class name:

public @interface RequiredProductConfiguration {
    Infrastructure infrastructure() default Infrastructure.DOCKER;
    boolean singeuser() default true;
}

[dmytro-ndp]

@skabashnyuk: thank you for suggestion to use _testng groups_. It's quite promising way to filter the tests without usage of additional annotations like @RequiredProductConfiguration and @Multiuser.

[dmytro-ndp]

Documentation can be found here.