Cypress: Confusing Promise / Cypress.Promise behavior depending on browser and run mode

[EirikBirkeland]

Current behavior:

I was using return new Promise all over my codebase using Chrome without issues, though I now realize that this is in conflict with the approach prescribed on the following page: https://docs.cypress.io/api/utilities/promise.html#Usage

So the interesting thing is that I was having no problems as long as I was using Chrome 65, 66 or 67 with cypress open.

However, when using Electron 59 through cypress open or when using any browser through cypress run, the new Promise would apparently be ignored, causing my tests to return early (without throwing any errors).

Another side-effect is that await/async works perfectly in Chrome, but not in Chromium.

Desired behavior:

• Can we possibly forbid "new Promise" altogether, warning that Cypress.Promise should be used?
• Alternatively, can we add issues like this to a list of caveats?

That said, it's actually the case that I did notice the Cypress.Promise article previously, but it was not clear to me at the time that I have to use it. It just seemed as if Cypress.Promise wrapped Bluebird and that was the end of it.

Steps to reproduce:

Please run this integration file in Chrome 67 and Electron 59 through cypress open:

describe('foo', function() {
    it('should have bar', () => {

        outerFn().then(cy.log)

        function outerFn () {
          return new Promise((resolve, reject) => {
            cy.log('outer promise');
            innerFn().then(res => resolve(res));
          });
        }

        function innerFn () {
          return new Promise((resolve, reject) => {
            cy.log('inner promise');
            resolve('string from inner promise')
          })
        }
    })
})

Using Chrome 67 or earlier version will result in the following output:

Using Electron 59 will result in the following output:

Observe that the final step to print a string is never reached. This is all fixed by replacing Promise with `Cypress.Promise``.

Versions

• Chrome 67 & Electron 59
• Cypress 3.0.1, but same result observed for 2.X.X
• Debian-based Linux (Linux Mint)

[brian-mann]

The underlying problem isn't with the promise library or the browser - it's because you're not returning outerFn to the test.

The reason you're seeing differences is purely because you're creating a race condition. Different browsers versions have different implementations for the native Promises. So it's only a chance that you're seeing different results - if you ran it enough times they would all likely be different and be the same. Swapping out to Cypress.Promise is simply using Bluebird which uses its own scheduler for enqueueing promise resolution, which is different than the native implementation.

If you returned outerFn then no matter what browser and no matter what Promise implementation you used - it would all be the same.

The other problem here is that you're trying to mix up Cypress commands and Promises - which can be seriously problematic. Cypress commands aren't true promises - they are a mix of streams with promise-like characteristics. They can interop just fine, but you don't approach them the same way as you approach typical Promises.

In a Cypress test you essentially never want to return an outer promise. You only ever should use Promises once you're inside of a Cypress chain via a .then. No other implementation is necessary because Cypress itself handles the enqueueing of its internal commands - you don't control it, which is the only way we can make them deterministic.

In fact - once you return outerFn you will see Cypress provide you a warning message that says you're mixing up Promises and Cypress commands. You should never call a Cypress command from within a promise (its 100% always unnecessary). You can do it the other way around, which is useful time to time.

[brian-mann]

Read this section and the next two: https://docs.cypress.io/guides/core-concepts/introduction-to-cypress.html#Commands-Run-Serially

Also these two error messages: https://docs.cypress.io/guides/references/error-messages.html#Cypress-detected-that-you-returned-a-promise-from-a-command-while-also-invoking-one-or-more-cy-commands-in-that-promise

[EirikBirkeland]

Thank you, your explanation and that documentation is really awesome!