Cypress: cy.wait() non-deterministically times out when waiting for mocked responses: No response ever occurred.

[lawsumisu]

Current behavior:

Sometimes, cy.wait() times out when waiting for some nth response of a mocked out endpoint, even though the Cypress UI shows that the request was stubbed out.

Desired behavior:

The documentation for cy.wait() says:

Each time we use cy.wait() for an alias, Cypress waits for the next nth matching request.

It seems that this behavior should be deterministic and the nth call to cy.wait() should find the nth response of a mocked endpoint if n responses have been successfully stubbed out.

Steps to reproduce:

1. Mock out an API response, and simulate n calls using that endpoint
2. Queue n cy.wait() calls (either by manually calling cy.wait() n times, or cy.wait([alias, alias, ..., alias])
3. This test should always pass, but sometimes the test will fail due a timeout while waiting for one of the responses

Versions

3.0.2, Mac OSX El Capitan, Chrome 67

[jennifer-shehane]

There is a difference between manually calling cy.wait() n times and using cy.wait([alias, alias, ..., alias])

Each cy.wait() command has a default requestTimeout of 5000 ms and a default responseTimeout of 30000. This means:

Scenario 1

cy.wait('@alias1') // waits 30 seconds to respond
cy.wait('@alias1') // waits 30 seconds to respond 2nd time
cy.wait('@alias1') // waits 30 seconds to respond 3rd time
cy.wait('@alias1') // waits 30 seconds to respond 4th time

Scenario 2

cy.wait(['@alias1', '@alias1', '@alias1', '@alias1']) // waits 30 seconds for request to respond 4 times

I find the behavior in Scenario 2 to be incredibly confusing - as I myself had many response timeouts seemingly for no reason until I discovered this was the case. I've since changed nearly all of my cy.wait() to be chained, like in Scenario 1, rendering the Scenario 2 usage pretty much useless.

Proposal

In order to fix this behavior, I would ideally want the timeouts to apply to each defined alias within the array individually.

---

@lawsumisu Please let me know if this is not the case and there is also a non-deterministic timeout when serially defining the cy.wait().

[lawsumisu]

@jennifer-shehane There is still a non-deterministic timeout when listing out cy.wait() as in Scenario 1.

[jennifer-shehane]

@lawsumisu Can you paste a screenshot of the network time from within the Network tab to see how long it is taking with the timeout?

We'll need a reproducible example to really track down any bug with this though.

[jlmart88]

@jennifer-shehane I wrote up a quick app to reproduce this issue:

https://github.com/jlmart88/cypress-wait-stress-test

After a lot of toying around, I have gotten this test to fail about half of the time on Electron 59 (headless), using cypress run. The error will always be something like:

CypressError: Timed out retrying: cy.wait() timed out waiting 30000ms for the 5th response to the route: 'testEndpoint'. No response ever occurred.

The failure will sometimes (but rarely) happen when using cypress run --browser chrome, and will almost never happen when using cypress open.

The key parts for reproducing I believe to be having something blocking the main thread in the application, while the API calls are resolving, and performing cy.visit in a .then block.

[jennifer-shehane]

Hey @jlmart88, I would likely rewrite some of this code to run more synchronously like below. Also, arbitrary cy.wait() definitions are generally not a best practice. It's best to define the DOM element's state you expect or wait for the request that you expect to happen within the 1000ms.

Let me know if this helps your nondeterministic behavior.

describe('stress test', () => {
  beforeEach('mock the api', () => {
    cy.server();
    cy.route('/api/endpoint/**', 'fixture:endpoint').as('testEndpoint');
    cy.fixture('endpoint')
    cy.visit('/');
  });

  it(`can queue ${NUM_REQUESTS} wait commands, visit, then queue ${NUM_REQUESTS} more wait commands`, () => {
    cy.wrap(NUM_REQUESTS).each(() => {
      cy.wait('@testEndpoint');
    }
     cy.visit('/');

    cy.wrap(NUM_REQUESTS).each(() => {
      cy.wait('@testEndpoint');
    }
  });
});

[michaelsharpe]

I am currently experiencing a similar issue. Random tests across our test suite fail on a cy.wait, but it only happens in our CI environment, not when running things locally. I can run the same test suite three times, and of the three two fail and one passes; of the two failing, they each fail in different test suites, but both from waiting on cy.wait commands that never returned.

My current theory is that some responses are so fast that cypress doesn't have time to hook in. The main difference from our CI environment to our local development is that the CI environment is entirely inside docker. API server, DB, static server that serves the client, and cypress. The docker network connections may be so fast on some responses that it is missed by Cypress.

That is my going theory, at least. I don't have time to do a minimal reproduction, but I will try to make some time in the next few days. I think setting it all up inside docker-compose should make for a reproducible demonstration of the bug.

[AdilHoumadi]

We have the same issue here, our tests fail randomly on CI, we are thinking to deploy a GUI container with chrome browser to avoid this issue. cy.wait() is not stable in headless mode.

[jennifer-shehane]

Unfortunately we'll have to close this issue if no reproducible example is provided. Can anyone provide a way to reproduce this?

[kristiehowboutdat]

If it helps - we're experiencing this issue using graphql polling every 3s. I cannot reproduce the issue locally using either cypress open or cypress run, however, it is about 60% reproducible on our CI (docker) using cypress run.

When the issue does show up, it is always consistent after switching tabs (changing the URL), which causes the graphql request to change (a new query for the new page).

The cy.wait(@graphql) command works on the original tab and then fails when switching to the new tab.

Minimal Code

 beforeEach(() => {
    cy.server();
    cy.route('POST', GRAPHQL_REGEXP).as('graphql');
  });

  it('should render the details page', function() {
    // This works
    cy.visit(orgURL).wait('@graphql');

    // Some specific assertions about the page that are working...

    // Now check on the other tab
    // NOTE: Use force: true because the text is hidden on the small screen
    cy.queryByText('Manage Devices').click({ force: true });

     // This is the alias that is not working - the test fails here even though the request is clearly made
    cy.wait('@graphql');    

    // Some assertions about the page...
  });

Screenshot

[butch-livly]

What doesn't seem to be addressed here is the use case people have for needing to wait on N requests of the same type, and whether there are better ways to accomplish the same goal rather than calling wait multiple times. Calling wait multiple times might imply that only the Nth wait is the one we care about. Perhaps if there was a command to flush the queue of 'unwaited' requests on an alias before we take the action that triggers the Nth network request, then we can be assured that the next wait command following that action will actually wait on the request we care about.

In the app I'm working on, we have a single endpoint for logging analytics events, I want to write a spec that verifies that we're sending the right analytics when certain actions occur. So let's say I write the spec and it captures the workflow and all the analytics as it works today. It works fine until we get a new requirement adding an additional analytics event. The test will now fail because an additional network request to this endpoint was added, and my test will have to be modified to either do a double-wait or expanded to test this new event.

This scenario makes my tests brittle, and I'd like to have a mechanism (like being able to flush network requests) to make my tests run more predictably, using the same reasoning that my existing UI tests should not fail because I added a new, unrelated, button to the screen.

[mccataldo]

@butch-livly I'm having similar issues. Here's what I implemented today. So far it's working. In my case, count is dynamic.

cy.get('@packageCount', { log: false })
  .then(count => {
    Cypress._.times(count, () => {
      cy.wait('@getPackageEquipment')
    })
  })

Give it a try and let us know if it makes a difference.