Cypress: Retries hangs on failed assertions
Current behavior:
Sometimes Cypress hangs to eternity on failed assertions while Retries functionality is active.
I described it once in this ticket:
https://github.com/Bkucera/cypress-plugin-retries/issues/64#issuecomment-675498787
Now I'm experiencing similar issue in 5.0.0 version.
It's probably a problem with .should('contain', xxx) assertion.
Test code to reproduce
Versions
5.0.0
zbigniewkalinowski
All 21 comments
Can you provide a reproducible example? We have quite a few tests we run through, so there鈥檚 likely something specific about the way the tests are written or run that will trigger this.
This will be hard to track down without that, especially if it only fails sometimes. Thanks!
jennifer-shehane
on 20 Aug 2020
Same happened for us. Usually our test suite finished in about 3 or 4 minutes max (success/failure). Nothing changed, we just enabled retries runMode: 3 and then our last CI cypress run lasted ran for 01:32:22 according to the Cypress Dashboard, but actually the Github Action was still running after 4h 34m.
It hits the first failure and then just hangs forever, it literally does nothing else:
(Attempt 1 of 4) test name omitted here
(Attempt 1 of 4) test name omitted here
(Attempt 2 of 4) test name omitted here
^ this is the very last line, there is nothing else. edit: The duplicate "Attempt 1 of 4" is also for the same test, which is weird.
this happened on all of our CI runners for parallel tests (each one just hit the first error and then the same thing like above happens).
edit: Just removed the retries from our cypress config, and the test execution finished again in 4 minutes.
crazyricardo
on 20 Aug 2020
Same is happening for us, both with the plugin and with 5.0.0.
I did an investigation to track down what was hanging, you can see the issue on cypress plugin retries for that, assuming it鈥檚 the same problem, but I wasn鈥檛 able to work out the exact problem only what was happening.
Some of the failed tests hang, whilst others are able to retry 3 times and then fail so I think it is something specific to certain tests. I鈥檝e already spent way too long this week debugging cypress but if I find a reproduction case I will post it.
lukeapage
on 20 Aug 2020
I'm unable to recreate this with a .should('contains', 'foo') assertion alone.
We will really need a reproducible example to be able to track this down - there could be a lot of things contributing, such as other defined hooks or things in the project support/plugins.
My test code
index.html
<html>
<body>
<div>bar</div>
</body>
</html>
spec.js
it('test', () => {
cy.visit('index.html')
cy.get('div').should('contain', 'foo')
})
cypress.json
{
"retries": {
"runMode": 3,
"openMode": 3
}
}
Runs 4 times and correctly ends the test.
jennifer-shehane
on 21 Aug 2020
- It only happens in run mode and I have a test case I can reproduce fairly consistently locally and on the build server.
- Its failing on the first attempt to retry (test runs once, fails and then stalls). But once I start removing code from the test it starts failing on 2nd retry.
- my test that is failing does not have
should('contain', 'foo')assertion in it. - no additional hooks
- I'm trying to make a test case but its very fragile - remove one assertion or another and it starts working or fails on 2nd retry - its like the length of the test makes a difference on whether retry will work
lukeapage
on 21 Aug 2020
Today my test hanged on click also.
zbigniewkalinowski
on 21 Aug 2020
Does anyone have a test:after:run defined in their tests anywhere? https://github.com/cypress-io/cypress/issues/2271
jennifer-shehane
on 21 Aug 2020
@jennifer-shehane no hooks are needed.
I made a reproduction here: https://github.com/cypress-io/cypress-test-tiny/pull/62
Does that reproduce for you? I know your on mac and I'm on windows but 馃
lukeapage
on 21 Aug 2020
Yah, it reproduces it when running via cypress run, not during cypress open.
jennifer-shehane
on 21 Aug 2020
Is that enough to investigate? I ask because this issue is still labeled "Needs information"
lukeapage
on 21 Aug 2020
It is, but it's going to take a lot longer to determine the cause with such a large example.
jennifer-shehane
on 24 Aug 2020
Same happened for us. Usually our test suite finished in about 3 or 4 minutes max (success/failure). Nothing changed, we just enabled retries
runMode: 3and then our last CI cypress run lasted ran for 01:32:22 according to the Cypress Dashboard, but actually the Github Action was still running after 4h 34m.It hits the first failure and then just hangs forever, it literally does nothing else:
(Attempt 1 of 4) test name omitted here
(Attempt 1 of 4) test name omitted here
(Attempt 2 of 4) test name omitted here^ this is the very last line, there is nothing else. edit: The duplicate "Attempt 1 of 4" is also for the same test, which is weird.
this happened on all of our CI runners for parallel tests (each one just hit the first error and then the same thing like above happens).
edit: Just removed the retries from our cypress config, and the test execution finished again in 4 minutes.
I meet the similar problem. after upgrade to 5.0.0, deprecate cypress-plugin-retries and use built-in retry function, my test cases fail all the time then hang for serveral hours when run github action.
print like below
(Attempt 1 of 3) tset1 name
(Attempt 2 of 3) test1 name
1) test1 name
........
(Attempt 1 of 3) tsetn name
(Attempt 2 of 3) testn name
n) testn name
but my code runs well on local machine. the only change is upgrade
(with "cypress open")
I run with "cypress run" on local machine again. And It hangs. Maybe it is not about github action.
Roger-Fang-CS
on 26 Aug 2020
@lukeapage I narrowed down the reproducible example to this single spec file. I think the issue is with your overwriting cy.get(). We really don't recommend doing this - you should be using overwrite to overwrite commands.
https://github.com/cypress-io/cypress-test-tiny/pull/62/files#diff-05da80f0f815af9c2b8279ce29b86790
Run via cypress run --headed --no-exit to see the test stall.
jennifer-shehane
on 26 Aug 2020
There鈥檚 only a single spec file and spec in that Pr isn鈥檛 there?
How did you narrow it down to overwriting get ? I can try refactoring that.
Unfortunately we got so many issues with overwrite that we don鈥檛 use it.
lukeapage
on 26 Aug 2020
Sorry I see what you did now. I will take a look later.
lukeapage
on 26 Aug 2020
@jennifer-shehane I pushed some commits that simplify it further. I've removed the overwrite of get and also I replaced the "waitForTimers" with a simpler pause - I added the commits in stages because a 1s pause is more likely that it might suddenly stop reproducing.. but for me I still reproduce with the simplified example.
https://github.com/cypress-io/cypress-test-tiny/pull/62/commits/ae67c76a371356a0abcf8696810a19b0c19abc0f
https://github.com/cypress-io/cypress-test-tiny/pull/62/commits/e52a63d77f62f78404249e7ed64e250f620b082b
https://github.com/cypress-io/cypress-test-tiny/pull/62/commits/739d2c8050b00f1623abeb8113de9fefec8736af
lukeapage
on 26 Aug 2020
@lukeapage K, thanks, thats helpful.
jennifer-shehane
on 27 Aug 2020
I think I found the source of the issue, hoping to get a patch out in the next release.
Bkucera
on 4 Sep 2020
what reproduces the bug is having a test that fails and retries with an existent DOM node as the subject of an assertion e.g.:
cy.get('.element-exists').should('not.exist')
our logic only pretty formatted err.actual on test failure, not retry, so the browser hangs trying to serialize an entire DOM node.
Bkucera
on 9 Sep 2020
The code for this is done in cypress-io/cypress#8527, but has yet to be released.
We'll update this issue and reference the changelog when it's released.
![cypress-bot[bot] picture](https://avatars3.githubusercontent.com/u/8908513?v=4&s=40)
cypress-bot[bot]
on 14 Sep 2020
Released in 5.2.0.
This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v5.2.0, please open a new issue.
cypress-bot[bot]
on 15 Sep 2020
Related issues
rbung
路
3Comments
carloscheddar
路
3Comments
egucciar
路
3Comments
tahayk
路
3Comments
brian-mann
路
3Comments
Most helpful comment
The code for this is done in cypress-io/cypress#8527, but has yet to be released.
We'll update this issue and reference the changelog when it's released.