Cypress: Cypress doesn't run all tests and reports as done

[lucker14-zz]

Current behavior:

From time to time Cypress doesn't run all the tests and reports as done, without any additional output (first image), it does not even log the test result table. In the mochawsome output, we can clearly see, that all the tests that are not shown in output were not really running. This behavior happens even with a single test (second image)

Desired behavior:

If Cypress doesn't finish all the tests it would be nice if it could return a non-zero return code. Or ideally, if it could process everything as we're used to.

Steps to reproduce: (app code and test code)

This behavior is not reproducible, it happens only from time to time.

Versions

Cypress: 3.4.0
OS: Dockerized Ubuntu 18.04 bionic
Docker: 18.6.3
ENV: Gitlab CI that runs on Kubernetes

Cypress setup

Cypress cmd: cypress run --spec <list of ~20 spec files>

cypress.json:

{
  "defaultCommandTimeout": 20000,
  "requestTimeout": 5000,
  "responseTimeout": 45000,
  "projectId": "REDACTED",
  "supportFile": "cypress/support/index.ts",
  "viewportHeight": 1024,
  "chromeWebSecurity": false,
  "viewportWidth": 1440,
  "videoCompression": false,
  "blacklistHosts": [
   <REDACTED>
  ],
  "reporter": "mochawesome",
  "reporterOptions": {
    "reportDir": "cypress/test-results",
    "overwrite": false,
    "html": false,
    "json": true
  }
}

plugins used:

"cypress": "~3.4.0",
"@cypress/webpack-preprocessor": "4.1.0",
"cypress-failed-log": "^2.5.1",
"cypress-pipe": "^1.4.0",
"cypress-plugin-retries": "^1.2.2",
"@4tw/cypress-drag-drop": "1.2.1",

[jennifer-shehane]

Please detail all of the setup involved in your project. What reporters you are using - how you have them set up to run, any reporterOptions, plugins / support files / cypress.json config. How are you running Cypress - ie cypress run, from Module API, etc.

This is unique somehow to your environment/settings, as this definitely does not happen for every user, so we need to know what is created this edge case.

[lucker14-zz]

@jennifer-shehane Thank you very much for such a fast answer, I updated the issue with more details about our configuration 🙂

[jennifer-shehane]

Does this still happen when removing the mochaawesome reporting? Thinking it may be isolated to something with mochaawesome, but this will be hard for us to track down without being able to run an example ourselves.

[lucker14-zz]

Hey @jennifer-shehane I've tried runs with the Mochawesome reporting turned off, but the problem still persists, obviously, I can't give you access to our development environment, and it's really hard to replicate this issue, but I've found some interesting logs while running the tests, hopefully, they can help 🙂

cypress:server:server Getting remote state: { auth: null, props: { port: '443', tld: 'info', domain: 'productboard' }, origin: 'https://qa-e2e-build-4d37e30d-2.productboard.info', strategy: 'http', visiting: false, domainName: 'productboard.info', fileServer: null } +611ms
  cypress:server:proxy handling proxied request { url: '/api/views/229236', proxiedUrl: 'https://qa-e2e-build-4d37e30d-2.productboard.info/api/views/229236', headers: { host: 'qa-e2e-build-4d37e30d-2.productboard.info', connection: 'keep-alive', 'content-length': '52', origin: 'https://qa-e2e-build-4d37e30d-2.productboard.info', 'x-csrf-token': '[REDACTED]', 'pb-embedded-mode': '1', 'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/3.4.0 Chrome/61.0.3163.100 Electron/2.0.18 Safari/537.36', 'content-type': 'application/json', accept: 'application/json', 'pb-rev': '4d37e30df504f2a05b1f3fffec43d01ff40013fa', 'x-referer': 'https://qa-e2e-build-4d37e30d-2.productboard.info/feature-board/229142-planning?rev=branch/cy-crash-test-2&disable_analytics', 'pusher-session-id': '544603211', 'pb-deploy-rev': 'branch/cy-crash-test-2', referer: 'https://qa-e2e-build-4d37e30d-2.productboard.info/', 'accept-encoding': 'gzip, deflate', 'accept-language': 'en-US', cookie: '[REDACTED]' }, remoteState: { auth: null, props: { port: '443', tld: 'info', domain: 'productboard' }, origin: 'https://qa-e2e-build-4d37e30d-2.productboard.info', strategy: 'http', visiting: false, domainName: 'productboard.info', fileServer: null } } +303ms
  cypress:server:stream_buffer appending chunk to buffer { bytesWritten: 0, chunkLength: 52 } +610ms
  cypress:network:agent addRequest called { isHttps: true, href: 'https://qa-e2e-build-4d37e30d-2.productboard.info/api/views/229236' } +611ms
  cypress:network:agent got family { family: 4, href: 'https://qa-e2e-build-4d37e30d-2.productboard.info/api/views/229236' } +0ms
  cypress:server:stream_buffer stream buffer writeable final called +1ms
  cypress:server:request received status code on request { requestId: 'request423', statusCode: 200 } +403ms
  cypress:server:request successful response received { requestId: 'request423' } +0ms
  cypress:server:proxy received response for { url: 'https://qa-e2e-build-4d37e30d-2.productboard.info/api/views/229236', headers: { server: 'Cowboy', date: 'Thu, 25 Jul 2019 20:29:18 GMT', connection: 'keep-alive', 'cache-control': 'no-cache, no-store', pragma: 'no-cache', expires: 'Fri, 01 Jan 1990 00:00:00 GMT', 'x-robots-tag': 'noindex, nofollow', 'content-type': 'application/json; charset=utf-8', p3p: 'CP="CAO PSA OUR"', vary: 'Accept-Encoding', 'content-encoding': 'gzip', 'set-cookie': [ REDACTED ], 'x-request-id': 'REDACTED', 'x-runtime': '0.080989', 'x-protected-by': 'Sqreen', 'strict-transport-security': 'max-age=31536000; includeSubDomains; preload', 'x-frame-options': 'sameorigin', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'x-download-options': 'noopen', 'x-permitted-cross-domain-policies': 'none', 'referrer-policy': 'strict-origin-when-cross-origin', 'content-security-policy': [REDACTED] worker-src \'self\' blob:; report-uri /csp_report', 'transfer-encoding': 'chunked', via: '1.1 vegur' }, statusCode: 200, isGzipped: true, wantsInjection: false, wantsSecurityRemoved: false } +100ms
2019-07-25T20:29:21.228Z cypress:cli Stopping Xvfb
Done in 256.15s.
+ exit 0

  Running: editor/portal_card_description_feature.spec.ts...                             (10 of 19) 
[301:0802/133747.501564:ERROR:zygote_host_impl_linux.cc(254)] Failed to adjust OOM score of renderer with pid 1223: Permission denied
[301:0802/133748.153663:INFO:CONSOLE(162422)] "%cDownload the React DevTools for a better development experience: https://fb.me/react-devtools", source: http://localhost:45663/__cypress/runner/cypress_runner.js (162422)
[301:0802/133748.382181:INFO:CONSOLE(118886)] "%cDownload the React DevTools for a better development experience: https://fb.me/react-devtools", source: http://localhost:45663/__cypress/runner/cypress_runner.js (118886)
[301:0802/133748.438967:INFO:CONSOLE(96850)] "Warning: Failed prop type: The prop `config.browsers` is marked as required in `App`, but its value is `undefined`.
    in App (created by Container)
    in Container", source: http://localhost:45663/__cypress/runner/cypress_runner.js (96850)
(node:471) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 preprocessor:close listeners added. Use emitter.setMaxListeners() to increase limit


[301:0802/133749.457624:ERROR:zygote_host_impl_linux.cc(254)] Failed to adjust OOM score of renderer with pid 1243: Permission denied
[301:0802/133750.274087:INFO:CONSOLE(162422)] "%cDownload the React DevTools for a better development experience: https://fb.me/react-devtools", source: https://qa-e2e-build-735a83ec-2.productboard.info/__cypress/runner/cypress_runner.js (162422)
[301:0802/133750.524132:INFO:CONSOLE(118886)] "%cDownload the React DevTools for a better development experience: https://fb.me/react-devtools", source: https://qa-e2e-build-735a83ec-2.productboard.info/__cypress/runner/cypress_runner.js (118886)
[301:0802/133750.579892:INFO:CONSOLE(96850)] "Warning: Failed prop type: The prop `config.browsers` is marked as required in `App`, but its value is `undefined`.
    in App (created by Container)
    in Container", source: https://qa-e2e-build-735a83ec-2.productboard.info/__cypress/runner/cypress_runner.js (96850)
[301:0802/133753.953603:INFO:CONSOLE(2)] "Event tracking disabled", source: https://cdnjs.cloudflare.com/ajax/libs/rollbar.js/2.2.7/rollbar.min.js (2)
  Portal card description editor
Done in 318.02s.
+ exit 0

[v-anyukov]

Any updates on this topic?

[jennifer-shehane]

This message is really peculiar. We recently updated how config.browsers is read in, as well as validation for config.browsers in Cypress 3.7.0.

Basically config.browsers is just an array of browsers that are found on the system - in order to decide which browser to run the tests within. This should never be undefined since Cypress ships with Electron browser by default, although it is theoretical that you could have overwritten this before 3.7.0 - it would be strange for this to randomly be undefined when the tests have already begun running.

If anyone is still experiencing this issue with the config.browsers warning, can you try updating to 3.7.0 to see if this is resolved.

[301:0802/133750.579892:INFO:CONSOLE(96850)] "Warning: Failed prop type: The prop `config.browsers` is marked as required in `App`, but its value is `undefined`.
    in App (created by Container)
    in Container", source: https://qa-e2e-build-735a83ec-2.productboard.info/__cypress/runner/cypress_runner.js (96850)

If it is not fixed in 3.7.0, a fully reproducible example would be needed for us to debug further.

[dinuemani]

looks like upgrading to 3.7.0 or 3.8.0, we are unable to run tests on chrome headless.. seeing below error..

" Error: WebSocket is not open: readyState 3 (CLOSED)

Because this error occurred during a 'before each' hook we are skipping the remaining tests in the current suite"

[jennifer-shehane]

@dinuemani Your issue does not look related to the original issue at all. Open a new issue with a reproducible example.

Unfortunately we have to close this issue due to inactivity. Please comment if there is new information to provide concerning the original issue and we can reopen.