Generator-jhipster: Improve test speed by using mocha's parallel flag

Created on 13 Sep 2020 · 10Comments · Source: jhipster/generator-jhipster

When doing some tests this morning, I decided to try the -p flag of Mocha to run tests in parallel.
The results:

Tests were over in 4 minutes
Only 40/5000+ tests failed (all CLI-related)

The error:

 Error: ENOENT: no such file or directory, uv_cwd
      at process.wrappedCwd (internal/bootstrap/switches/does_own_process_state.js:128:26)
      at process.cwd (node_modules/fs-extra/node_modules/graceful-fs/polyfills.js:10:19)
      at process.cwd (node_modules/graceful-fs/polyfills.js:10:19)
      at _prepareTempEnv (test/utils/utils.js:60:25)
      at testInTempDir (test/utils/utils.js:84:25)
      at Context.<anonymous> (test/cli/cli.spec.js:386:21)

Without the flag: Over 13 minutes

@jhipster/developers I think it's worth investigating.

If the investigation is inconclusive, we can still split our tests so as to have a majority of tests running in parallel, and a few others sequentially.

area tests

Source

MathieuAA

👍1

All 10 comments

BTW, solving this issue before JCode would be good, so that we don't wait too long for our test to be over.
Or maybe we could do this during JCode :)

MathieuAA on 13 Sep 2020

That is a huge improvement :+1:

atomfrede on 13 Sep 2020

https://github.com/mshima/generator-jhipster/commits/mocha_parallel
Fell free to test, but I am against merging it.

won’t benefit github CI (dual core)
will leave too many temp files. Proper cleanup after step should be added to tests. Same folder is used currently, cleanup happens on before step.

A easy workaround is to run only desired tests:
mocha test/...

I’ve already merged others fixes.

mshima on 13 Sep 2020

I tried to run tests in parallel when I was developing the CLI but it was
very buggy and flaky so gave up but probably we can split the tests into
multiple groups and run some of them in parallel. For folder cleanup may be
we can switch to generating folder with random names and clean after each
test

On Sun, 13 Sep 2020, 2:34 pm Marcelo Shima, notifications@github.com
wrote:

https://github.com/mshima/generator-jhipster/commits/mocha_parallel
Fell free to test, but I am against merging it.

won’t benefit github CI (dual core)

will leave too many temp files. Proper cleanup after step should be
added to tests. Same folder is used currently, cleanup happens on before
step.

A easy workaround is to run only desired tests:
mocha test/...

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jhipster/generator-jhipster/issues/12444#issuecomment-691666079,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAIOKF6ADTIOPWBTN456OPTSFS35XANCNFSM4RKQGFUQ
.

deepu105 on 13 Sep 2020

👍1

Splitting them by domain like JDL, client, server, etc., or even by subgen may be a good idea

MathieuAA on 13 Sep 2020

From 13mins down to 4/5 minutes

Wow - great work @MathieuAA!!

mraible on 15 Sep 2020

Thanks Matt!

MathieuAA on 15 Sep 2020

@MathieuAA I would prefer create PRs instead of committing directly to master.
There is ~37 uses of the parallel unsafe function, only ~8 of them will need independent fixes.

mshima on 15 Sep 2020

👍1

Great, then it may be the next steps

MathieuAA on 15 Sep 2020

Yes everyone since JDL is now in the main generator lets do PRs so that we
can keep track of changes.

On Tue, 15 Sep 2020, 6:36 pm Mathieu ABOU-AICHI, notifications@github.com
wrote:

Great, then it may be the next steps

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jhipster/generator-jhipster/issues/12444#issuecomment-692832715,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAIOKF6MLOBKROKMRG4M53LSF6JX7ANCNFSM4RKQGFUQ
.