Windowstemplatestudio: Improve testing of generated templates

[mrlacey]

The current testing of templates is a long and slow process. There are a few scripts to enable running parallelizing of some test execution but we can (and should) do better.

Need to do more to fully scope this out but the goals are:

• make tests faster to run (to avoid excuses for not running them)
• make tests easier to run.
• avoid developers relying on automated test execution at PR (or post PR) time
• Make it easier to test small subsets of templates. This may require reviewing data-bound test generation (which is slow anyway)

[mrlacey]

@sibille Please can you verify that the details in https://github.com/microsoft/WindowsTemplateStudio/blob/dev/docs/getting-started-developers.md#test-execution are correct in terms of which ExecutionSets are run where/when?

I want to make sure that any changes made for this issue don't break any of the automated test execution. Thanks.

[sibille]

@mrlacey, just checked, docs are up to date.

[mrlacey]

Adding "meta" executionsets.
These are named matching the groupings executed during the different pipeline operations but beginning with an underscore.

The aim is to make it easier in VS to verify all tests that will be executed on the server will pass.

[mrlacey]

Rather than list template IDs that do support both C# & VB, just list the ones that don't support VB.
This removes the need to add to the list every time a template is added which should mean this isn't missed/forgotten and testing is more likely to catch differences between languages

[mrlacey]

To improve the speed of the CI builds, I propose moving the GenerateAllPagesAndFeaturesAndCheckWithStyleCopAsyncWithOptionalLogin tests to just be part of the full test suite.

Thoughts?

[sibille]

I added this to the CI build to cover all templates with styles there, but am fine with moving it to the full test suite.

[mrlacey]

To improve the speed of the CI builds, I propose moving the GenerateAllPagesAndFeaturesAndCheckWithStyleCopAsyncWithOptionalLogin tests to just be part of the full test suite.

Thoughts?

I added this to the CI build to cover all templates with styles there, but am fine with moving it to the full test suite.

I'll add a simpler test of OptionalLogin (with just a single page) to the CI build and the version with all pages/features/etc. to the full build.

This was the only simple thing I could see to remove from the CI tests to try and avoid it frequently timing out for taking too long.

[mrlacey]

The biggest improvement I can see to reduce build times is to be more selective in what is tested. This could be done by only testing things that have changed.

I propose:

• changes to docs (*.md files) only trigger tests related to those files.
• changes to templates should only trigger testing of those templates.
  -- This can be sub-divided by language. (e..g only run VB templates files if a *.vb file is added or modified--& C# equivalent)
• support by platform (e.g. UWP/WPF/...) once supported
  -- When we support, splitting templates into different packages (after #234 is implemented) those packages can also be tested separately. (plans to move prism & caliburn.micro into separate packages will make testing of the main C# templates 40% faster.)

Having the ability to divide tests like this would make it easier to only run those impacted by changes. This would require some configuration on the CI servers but would potentially reduce what needs to be run for each PR.
All tests can/should still be run on a nightly basis.

[sibille]

@mrlacey, I think we could remove the BuildAllPagesAndFeaturesAsync test as generating and building is also covered by BuildAllPagesAndFeaturesThenRunTestsAsync.

We could also run the tests on the StyleCop tests, this would make the tests less unitary, but we'd save building the projects consumes a lot of time (aprox 2 min per project)

[mrlacey]

If you're happy to have tests verify more than one thing in a way to reduce total execution time, there are others that could be combined too.

The simplest ones to combine are:

• BuildEmptyProjectAndInferConfigAsync and BuildEmptyProjectAsync
• BuildAllPagesAndFeaturesAsync and BuildAllPagesAndFeaturesThenRunTestsAsync

BuildAllPagesAndFeaturesProjectNameValidationAsync and BuildAllPagesAndFeaturesRandomNamesAsync could also probably be combined with something else too.

[sibille]

Given the amount of tests and execution time I think we should test as much as we can on every test.

Will open a separate issue on milestone 3.6 for this.

[mrlacey]

Test changes merged to main branch so closing this. Follow on work on a separate issue