Vscode-extension-for-zowe: Setup automated regression tests of Zowe Explorer in Theia

[crawr]

We want to setup automated regression tests to ensure the compatibility of the Zowe Explorer with Theia releases.

The basic process is as follows:

• Build and Run a Theia browser example with the Zowe Explorer vsix installed as a plugin
• Use Selenium to emulate user interaction with the Zowe Explorer in the Theia browser
• Combine Selenium with Mocha and Chai tests, similar to what we already have under Integration tests

Here is a pipeline that serves as PoC:

In this pipeline,

• The regression test package is installed with node dependencies
• Selenium, Firefox, and geckodriver are installed
• Theia is cloned from source (Theia version can be parametrized)
• ZE vsix is copied from artifactory to the Theia plugins folder (vsix version can be parametrized)
• Yarn is installed and is used to build and run Theia
• A simple Mocha test is performed to check that the Zowe Explorer is loaded and that the Favorites node appears

Planned implementation:

• Create a new CI pipeline to perform the regression tests
  -- Open question: nightly build or after a commit to master branch or both?
• Use the already existing jenkins slave image in the Zowe environment that already supports Firefox, Selenium, and Node
• Perform initial test cases that do not require connection to the mainframe
  -- Check that all the three trees are loaded and that the Favorites node exists
  -- Check that new Connection profiles can be created and added to the trees
• Add more test cases after discussion and prioritization by the squad

[zFernand0]

Really cool stuff 😸

As you mentioned, more tests need to be included. I was wondering if you considered reusing the current set of "integration" tests that we already have?
Maybe we can properly split Integration and System tests in order to take advantage of this pipeline even sooner.

As for the frequency, I believe that at least on every PR this pipeline should be executed. I understand that it may slow things down a bit, however we can research how to speed up the process (e.g. parallel stages/pipelines, prebuilt environments or shared-volumes that already have Theia Build and installed)?

[crawr]

Yes, the plan is to definitely use the existing set of integration tests that we have. I think after the initial setup, the next step is to plan for the test cases we want to include.

Ideally, Theia shouldn't be built for every run. So having a separate image with Theia pre-built will speed things up. You are right that this will require separate discussion/research.

I think we can also setup the regression test pipeline in a separate pipeline, triggered by every PR.