Pact-js: Proposal: Support parallel tests

[jeffreysilver]

Software versions

• OS: Mac OSX 10.12
• Consumer Pact library: Pact 3.0.1
• Provider Pact library: n/a
• Node Version: 8.2.1
• Jest Version: 21.0.0

Expected behavior

1) The Pact Server is deterministic when running a test multiple times
2) If the Pact Server log receives and handles interactions correctly, the test suite should pass.

Actual behavior

1) The Pact Server fails to register certain events when running a test multiple times. Most of the time the test executes as expected, but sometimes certain requests get lost. Specifically, the _first_ interaction with the Pact Server results in an ECONNREFUSED error.
2) Often times a test executes as expected (as documented in the server-integration.log), but the test cases fail with the following Jasmine Timeout:

Note: The error is surfaced for both the request, and the verification step of all tests.
Note: Re-running the test after this failure resulted in a success.

Steps to reproduce

quotes-pact-test.js: https://gist.github.com/jeffreysilver/6adefb78e3f7242432923eaa3e546f27

Relevant log files

mock-server-integration.log: https://gist.github.com/jeffreysilver/0c5c01aa3d18cd2de5403298df9c86a4
^This log file is the result of the test executions that led to Jasmine Timeouts.

mock-server-integration.log 2: https://gist.github.com/jeffreysilver/2e8fe4cb91a93c2b65a1e3960b2f30f8
This log was the result of running the same test file. The first interaction was not registered due to an ECONNREFUSED that was surfaced in our error logging.

Debugging

./node_modules/@pact-foundation/pact-standalone/platforms/<platform>/bin/pact-mock-service
./node_modules/@pact-foundation/pact-standalone/platforms/<platform>/bin/pact-provider-verifier

The above commands run.

[mefellows]

Thanks for the report @jeffreysilver.

Logs 1
Very strange indeed, as the logs look perfect.

Logs 2
From those logs, everything does appear to be executing in the order intended, however as you say if the first request is "lost" then everything will fall apart.

Perhaps focussing on improving that issue could be a good starting point.

[jeffreysilver]

Update:

The non-determinism appears to arise from Jest's parallelized test running. Using npm test -- --runInBand has resolved our issue. Ideally, we would run our test suite in parallel. Do you have any ideas of why the parallelism might cause the test to flake?

[mefellows]

Ah, this makes sense!

Yes, the mock server is stateful as it needs to know if the expected calls are made in the right order.

We are working on fixing this issue but could take some time.

[bethesque]

We got a work around happening by using one mock service per thread. Ask in
the gitter channel.

On Nov 14, 2017 7:02 AM, "Matt Fellows" [e-mail] wrote:

Ah, this makes sense!

Yes, the mock server is stateful as it needs to know if the expected calls
are made in the right order.

We are working on fixing this issue but could take some time.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/pact-foundation/pact-js/issues/124#issuecomment-344040380,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAbPFLIKfitGI_tPKFXNc30atzK13qB1ks5s2KBagaJpZM4QU9dx
.

[bethesque]

Have a look at the usage instructions here
https://github.com/pact-foundation/pact-mock_service/blob/master/README.md#mock-service-usage

On Nov 14, 2017 8:01 AM, "Beth Skurrie" [e-mail] wrote:

We got a work around happening by using one mock service per thread. Ask
in the gitter channel.

On Nov 14, 2017 7:02 AM, "Matt Fellows" [e-mail] wrote:

Ah, this makes sense!

Yes, the mock server is stateful as it needs to know if the expected
calls are made in the right order.

We are working on fixing this issue but could take some time.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/pact-foundation/pact-js/issues/124#issuecomment-344040380,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAbPFLIKfitGI_tPKFXNc30atzK13qB1ks5s2KBagaJpZM4QU9dx
.

[mefellows]

Eventually this will be rolled into the pact js stuff, but for now if you really need parallel tests have a look into what Beth has suggested.

[mefellows]

Renamed this to capture it's new intent - supporting parallel tests.

I've also updated the readme with a note on async/parallel tests.

[mefellows]

I'll add full support for this in pact js tomorrow (or very soon). Just pushed a feature branch with working merge example.

[lirantal]

Nice stuff seeing this.
Please don't forget updating the documentation/wiki to make sure we understand how it works :-)

[christopher-francisco]

The readme section says:

Parallel tests
...
The good news is, parallel test execution is possible, you need to ensure that...


Note on Jest
...
Jest also can run tests in parallel, which is currently not supported as the mock server is stateful.

Are parallel test in Jest supported?

[bethesque]

The Jest docs need updating. Yes, you can run tests in parallel, but you need to create a mock service for each thread, and ensure the interactions are all merged into the same file. https://github.com/pact-foundation/pact-mock_service/blob/master/README.md#mock-service-usage

[mefellows]

@chris-fa I've updated the relevant bit in the readme with some more clarity on the matter. In short, we support parallel test execution with Jest.

Thanks for pointing out the error.