Stryker: Performance issues with angular app.

[randspy]

Summary

Hi,

I have performance issues when executing stryker on my computer. It can be very slow. The estimation for the project is around 1000 hours. So to get around I mutate only the file for the tests I would like to execute. I also focus the tests itself (seams to be a little faster, but maybe my mind make tricks on me). The test by them-self are fast. It takes 3s to execute 500 tests. It slows down with increasing number of mutators. If there are not many mutators in the tests selected (for example 7) it takes around 7 min, with 130 mutators it estimates it on like 2 hours (never waited to verify how long it takes in reality). But if I move those 2 hours tests with the executed code to new created repository it takes only 8 min. In the logs is written that the app launched 507 tests. Is it ignoring fdescribe?

14:16:55 (20324) INFO InputFileResolver Found 1 of 680 file(s) to be mutated.
14:16:55 (20324) INFO InitialTestExecutor Starting initial test run. This may take a while.
14:20:03 (20324) INFO InitialTestExecutor Initial test run succeeded. Ran 507 tests in 3 minutes 8 seconds (net 40 ms, overhead 22862 ms).
14:20:03 (20324) INFO MutatorFacade 107 Mutant(s) generated

Configuration the same as in https://github.com/stryker-mutator/stryker/issues/1415, except stryker, it has the newest version.

$ yarn list | grep stryker
├─ @stryker-mutator/[email protected]
├─ @stryker-mutator/[email protected]
│ ├─ @stryker-mutator/api@^1.1.0
│ ├─ @stryker-mutator/util@^1.1.0
├─ @stryker-mutator/[email protected]
│ ├─ @stryker-mutator/api@^1.1.0
│ ├─ @stryker-mutator/util@^1.1.0
├─ @stryker-mutator/[email protected]
│ ├─ @stryker-mutator/api@^1.1.0
├─ @stryker-mutator/[email protected]
│ ├─ @stryker-mutator/api@^1.1.0
│ ├─ @stryker-mutator/util@^1.1.0
├─ @stryker-mutator/[email protected]

Second thing. Visual Code Studio needs to closed when executing the tests, otherwise it does a lot of operations even when .stryker-tmp is in .gitingnore. It gets into no responding state. Don't know if this one is fixable.

I will appropriate all the help. You are doing a great job.
BR

[simondel]

Thanks for opening this issue! I don't think we'll override your fdescribe in this scenario, but I'm not 100% sure right now.

I do find something a bit odd:

It takes 3s to execute 500 tests.
But your logging says it takes 3 minutes:
INFO InitialTestExecutor Initial test run succeeded. Ran 507 tests in 3 minutes 8 seconds (net 40 ms, overhead 22862 ms).

How long does it take to run your tests using ng test?

[randspy]

As requested for headless chrome, most of that 89s is typescript compilation.

Finished in 11.779 secs / 2.629 secs @ 16:00:00 GMT+0100 (Romance Standard Time)

SUMMARY:
√ 497 tests completed
i 10 tests skipped
Done in 89.39s.

[markgaze]

I'm seeing similar performance issues, I can run my tests (217 in total) in just under 3 seconds but Stryker takes almost 2 hours to run the mutation. I have a similarly-sized React project (using TypeScript) that reported: INFO Stryker Done in 5 minutes 57 seconds. with a normal test run time of about 4 seconds.

I can provide more information if required/useful.

[nicojs]

As requested for headless chrome, most of that 89s is typescript compilation.

I'm pretty sure it is a combination of typescript compilation, angular compilation, sass transpiling, bundling and anything else Angular needs to run an application.

I'm afraid this is quite a normal performance for Angular. Every mutant needs to progress through the entire angular cli pipeline as well as the normal karma test time. We use parallel test runner to try and speed up this process, but it only helps performance linearly. We don't support coverage analysis for Angular at the moment, but that would not help with the pipeline time, only the test time.

We're thinking of introducing mutation switching to Stryker, but that will definitely take some time. In theory that would mean only transpiling all mutants once.

[randspy]

Hi,

Yes, angular does a lot of staff. The fact that each mutator needs to go those that process explains the performance.

Looking forward for mutation switching.

As performance goes typescript 3.4 will get some performance improvements with incremental flag. https://devblogs.microsoft.com/typescript/announcing-typescript-3-4-rc/

BR

[simondel]

Since there's not much we can do about this at the moment, I'm closing this issue.