Cucumber-js: Intermittent publish failures

[charlierudolph]

Seen the following intermittently on CI:

1) Scenario: the publication banner is not shown when publication is done # features\publish.feature:114
   √ Before # features\support\hooks.ts:16
   √ Before # features\support\hooks.ts:20
   √ Given a report server is running on 'http://localhost:9987' # features\step_definitions\report_server_steps.ts:8
   √ And my env includes "CUCUMBER_PUBLISH_URL=http://localhost:9987/api/reports" # features\step_definitions\cli_steps.ts:15
   √ And a file named "features/a.feature" with: # features\step_definitions\file_steps.ts:10
       """
       Feature: a feature
         Scenario: a scenario
           Given a step
       """
   √ And a file named "features/step_definitions/steps.js" with: # features\step_definitions\file_steps.ts:10
       """
       const {Given} = require('@cucumber/cucumber')
       Given(/^a step$/, function() {})
       """
   √ When I run cucumber-js with arguments `--publish` and env `` # features\step_definitions\cli_steps.ts:29
   √ Then the error output does not contain the text: # features\step_definitions\cli_steps.ts:118
       """
       Share your Cucumber Report with your team at https://reports.cucumber.io
       """
   √ After # features\support\hooks.ts:110
   × After # features\support\hooks.ts:97
       Error: Last run errored unexpectedly. Output:
       .
       1 scenario (1 passed)
       1 step (1 passed)
       0m00.005s (executing steps: 0m00.000s)
       Error Output:
       ┌──────────────────────────────────────────────────────────────────────────┐
       │ View your Cucumber Report at:                                            │
       │ https://reports.cucumber.io/reports/f318d9ec-5a3d-4727-adec-bd7b69e2edd3 │
       │                                                                          │
       │ This report will self-destruct in 24h unless it is claimed or deleted.   │
       └──────────────────────────────────────────────────────────────────────────┘
       events.js:174
             throw er; // Unhandled 'error' event
             ^
       Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
           at ClientRequest.onclose (internal/streams/end-of-stream.js:57:15)
           at ClientRequest.emit (events.js:203:15)
           at Socket.socketCloseListener (_http_client.js:358:9)
           at Socket.emit (events.js:203:15)
           at TCP._handle.close (net.js:607:12)
       Emitted 'error' event at:
           at errorOrDestroy (internal/streams/destroy.js:107:12)
           at stream._final (_stream_writable.js:620:7)
           at sendRequest (C:\projects\cucumber-js\lib\formatter\http_stream.js:49:28)
           at stream_1.pipeline (C:\projects\cucumber-js\lib\formatter\http_stream.js:102:21)
           at internal/streams/pipeline.js:84:7
           at internal/util.js:370:14
           at ClientRequest.eos (internal/streams/pipeline.js:33:21)
           at ClientRequest.<anonymous> (internal/util.js:370:14)
           at ClientRequest.onclose (internal/streams/end-of-stream.js:58:23)
           at ClientRequest.emit (events.js:203:15)
           [... lines matching original stack trace ...]
           at TCP._handle.close (net.js:607:12)
           at World.<anonymous> (C:\projects\cucumber-js\features\support\hooks.ts:103:11)

[charlierudolph]

So far seen two instances of this on appveyor Node 10 and one instance on travis Node 12

[charlierudolph]

Another on appveyor Node 12

https://ci.appveyor.com/project/charlierudolph/cucumber-js/builds/35868261/job/7lrpwx471aemhe4y

[aslakhellesoy]

Thanks @charlierudolph we'll have a look at this. cc @vincentcapicotto @vincent-psarga

[aslakhellesoy]

Is this still an issue with 7.0.0 @charlierudolph?

[davidjgoss]

Still an issue with a fairly high % of CI builds.

[aslakhellesoy]

@aurelien-reeves and I tried to fix this a few months ago without success.

I think the pragmatic solution is to just disable these tests for Windows in CI.

[davidjgoss]

@aslakhellesoy are they definitely limited to just Windows?

[aurelien-reeves]

No, they are not limited to windows.
https://github.com/cucumber/cucumber-js/pull/1617 shows us that all build may be impacted.
Cf. https://github.com/cucumber/cucumber-js/commit/edc8f106edda0d001f1b31a82a5b4bd86f260fb5

I would suggest:

• deactivating the tests that are flacky
  
  
  
  • eventually executing those only when changes to specific files are pushed
  
  
• rerun the build after a first failure

[aslakhellesoy]

My bad - I thought it was Windows only.

Rerunning the build won't solve anything, they are just as likely to fail a 2nd time. I think we'd have a better chance with --retry. But even then it might end up failing.

We need 100% consistent test runs. I'd propose disabling the tests in CI. This code has low churn, and it might be sufficient to run them locally.

[aurelien-reeves]

My bad - I thought it was Windows only.

Rerunning the build won't solve anything, they are just as likely to fail a 2nd time. I think we'd have a better chance with --retry. But even then it might end up failing.

That was what I had in mind, using the rerun file

We need 100% consistent test runs. I'd propose disabling the tests in CI. This code has low churn, and it might be sufficient to run them locally.

I have a preference for that one too

[aurelien-reeves]

For information: the build seems to be more stable than it used to be.
The recent failures was not always related to that issue, but to dependency audit. But it seems to be have been fixed recently.

[davidjgoss]

I think it could be worth an experiment with --retry - happy to run that up and see what happens with e.g. 30 builds in a row?

[mattwynne]

Does this test actually integrate with the live cucumber reports server? Would it be possible to have it call a stub server instead, running locally? Or swap out a little adapter that wraps the calls to the remote reports endpoints?

[aslakhellesoy]

The test uses a local Node http server that we start and stop in every test.