Cypress: Firefox hangs sometimes when running in cypress/included:4.0.1

[bahmutov]

Cypress v4.0.1 installed in cypress/included:4.0.1 with Firefox 72

Running via docker-compose in repo https://github.com/cypress-io/cypress-example-docker-compose-included on CircleCI

There are 3 jobs in the Circle workflow https://circleci.com/gh/cypress-io/workflows/cypress-example-docker-compose-included/tree/master test-electron and test-chrome usually pass without any problems (they are running our Kitchensink tests against example.cypress.io), while FF usually hangs during one the specs (different specs)

You can search test-firefox among jobs here https://circleci.com/gh/cypress-io/cypress-example-docker-compose-included/tree/master

so it could be a crash inside the container, but we don't know

[rdelvallej32]

I'm seeing similar behavior with our tests using Gitlab CI. Chrome runs fine, whereas Firefox will hang indefinitely for random specs.

[bahmutov]

I have added DEBUG=cypress:server:browsers:firefox-util,cypress:server:util:process_profiler and this shows hanging, yet Cypress still shows memory profiles https://circleci.com/gh/cypress-io/cypress-example-docker-compose-included/15 Weird that ffmpeg is still there, it should have exited.

[bahmutov]

disabled video while running FF - still it hangs at some point https://circleci.com/gh/cypress-io/cypress-example-docker-compose-included/18

[bahmutov]

In any case, this project is a good test case. If needed, we can turn parallelization in Circle for this job to test FF on 10 machines to make sure the fix is reliable (once we find the fix)

[ZachMayry]

I'm actually get this error in chrome too

[tommueller]

We are also encountering this issue running our tests in a drone.io CI using Firefox.

[Hendric-Eckelt]

I have the same error on both Chrome and Firefox using Gitlab CI.

[jennifer-shehane]

@ZachMayry @Hendric-Eckelt There is not an error being displayed in the CIs, this is hanging on certain tests. If this is happening outside of Firefox, then this is a new issue as this issue is completely isolated to Firefox.

Please open an issue with a fully reproducible example.

[gwpetersenag]

I'm also seeing this on Firefox.
Running test on Jenkins in Docker.
Docker Image: cypress/base:8
Cypress v4.0.1
Firefox 73
Chrome test work perfectly fine.

[SebDuf]

I'm getting this on Firefox also; Cypress will hang around the second spec in a test run (no matter which).

Image: cypress/included:4.1.0

Here's some of the last logs before hanging:

cypress:server:util:socket_whitelist whitelisted socket closed, removing { localPort: 56390 } +15s
cypress:server:util:socket_whitelist whitelisted socket closed, removing { localPort: 44158 } +4ms
cypress:server:util:socket_whitelist whitelisted socket closed, removing { localPort: 57324 } +57s
cypress:server:util:socket_whitelist whitelisted socket closed, removing { localPort: 43308 } +1ms
cypress:server:util:socket_whitelist whitelisted socket closed, removing { localPort: 43312 } +1ms

[vivek12345]

I am seeing this error on our CI Pipeline too. I am running on chrome

FROM cypress/included:4.0.2

whitelisted socket closed, removing { localPort: 40924 }

[vivek12345]

@Bkucera Is there any solution to this problem?

[dalinarkholin]

@Bkucera @jennifer-shehane

I am seeing something similar, may be related to other issues as well, sorry if this is the wrong place.

Tests run fine locally.

Tests fail in CI.

Cypress 4.2.0
Cypress/included:4.2.0(have seen this in 4.0.2 as well as 3.8.3)
Browser: I have tried both electron and chrome.

No matter how I rearrange the tests(order wise) the first few pass, then once one fails from a ESOCKETTIMEDOUT. the rest of the tests fail.

Note that from run to run, the tests fail pretty consistently on the same test...unless I reorder them, then they start failing on a different test pretty consistently. The endpoint that fails is consistent unless I modify the test with cy.wait, cy.route, etc. Then another one usually fails, except now for the test that is failing I have covered every single XHR request with waits and still it fails on the one you see below.

Again, tests fun fine locally. I even tried switching to node@10 locally(what the docker image is based off of) and still they run fine.

At this point I am pretty certain It is either a networking issue, or a Cypress issue. I have had all the tests pass in CI before. But they have always been flaky. Now it seems like they are failing pretty consistently.

e2e_1.              |   cypress:server:util:socket_whitelist whitelisted socket closed, removing { localPort: 57512 } +5s
e2e_1               |   cypress:server:util:socket_whitelist whitelisted socket closed, removing { localPort: 57518 } +146ms
e2e_1               |   cypress:server:util:socket_whitelist whitelisted socket closed, removing { localPort: 36856 } +5s

cypress:server:request received an error making http request { timeout: 30000, retryIntervals: [ 0, 100, 200, 200 ], url: 'http://app-being-tested/api/v1/GridSettings?gridId=invoices', requestId: 'request351', retryOnNetworkFailure: true, retryOnStatusCodeFailure: false, delaysRemaining: [ 0, 100, 200, 200 ], err: Error: ESOCKETTIMEDOUT at ClientRequest.<anonymous> (/root/.cache/Cypress/4.2.0/Cypress/resources/app/packages/server/node_modules/request/request.js:816:19)     at Object.onceWrapper (events.js:299:28)     at ClientRequest.emit (events.js:210:5)     at Socket.emitRequestTimeout (_http_client.js:690:9)     at Object.onceWrapper (events.js:299:28)     at Socket.emit (events.js:210:5)     at Socket._onTimeout (net.js:468:8)     at listOnTimeout (internal/timers.js:531:17)     at processTimers (internal/timers.js:475:7) { code: 'ESOCKETTIMEDOUT', connect: false } }

cypress:proxy:http:error-middleware error proxying request { error: Error: ESOCKETTIMEDOUT at ClientRequest.<anonymous> (/root/.cache/Cypress/4.2.0/Cypress/resources/app/packages/server/node_modules/request/request.js:816:19)     at Object.onceWrapper (events.js:299:28)

[vivek12345]

Tried with Cypress:4.2.0, it still gets stuck with an error of whitelisted socket closed error.

[lipowiczNeurosys]

@bahmutov have you try to assign more shared memory for a docker container? I mean to run docker image with the following parameter i.e. docker run --shm-size=1g.
I think that I have a similar issue, but I found kind of workaround (at least in my local docker run).

[vivek12345]

@lipowiczNeurosys I have tried that too. Setting memory to 1G. Still gets stuck randomly with the following error

whitelisted socket closed, removing { localPort: 40924 }

[jruhe-adesso]

FF hangs in Azure Devops pipeline. Micorosoft hosted Build agent (on-premise). Config:
https://github.com/actions/virtual-environments/blob/master/images/linux/Ubuntu1804-README.md

[vivek12345]

Can someone please look into this issue?

[edarchimbaud]

Same here.

[dalinarkholin]

@jennifer-shehane @Bkucera Outside of a reproduction what is useful to help diagnose what is going on here? In our case, we have had to disable CI tests, which I am not okay with, so would love to see some movement on the tickets @vivek12345 pointed out all seem to be related.

[vivek12345]

Any updates on this? @Bkucera
We have to unfortunately disable e2e because no solution or fix is being provided for this.
A lot of us are facing this problem.
Can anyone atleast start looking into this?

[vanakema]

Having the same problem in chrome. Reference it in #6857

[michalziolek]

I also faced similar issue. Setting MOZ_FORCE_DISABLE_E10S=1 env variable solved the issue for me. This disables the multiprocess mode in Firefox

export MOZ_FORCE_DISABLE_E10S=1
cypress run --headless  --browser firefox 

[smfelsher]

I would like to confirm that setting MOZ_FORCE_DISABLE_E10S=1, according to @michalziolek's comment, also worked for me.

I am running both Chrome and Firefox browsers with the --headless option in a GitLab CI Docker. Firefox would hang until the CI job timed-out.

[kamranayub]

@lipowiczNeurosys increasing --shm-size=2g was the ticket for me in GitHub actions, see my research into this here: https://github.com/cypress-io/github-action/issues/104

Using MOZ_FORCE_DISABLE_E10S didn't seem to work in headed/headless mode, though I may not have put it at the right level (I tried on the uses: cypress-io/github-action@v2 step, not the job or container).

[jruhe-adesso]

FF hangs in Azure Devops pipeline. Micorosoft hosted Build agent (on-premise). Config:
https://github.com/actions/virtual-environments/blob/master/images/linux/Ubuntu1804-README.md

Works for me now in Azure Devops CI using Cypress 4.11 & Ubuntu 20.04 agents.