Cypress: Assertion failure in async test case does not fail test overall

[MadLittleMods]

Related issues

• https://brucelefebvre.com/blog/2019/08/26/cypress-gotchas-with-promises/
• https://github.com/cypress-io/cypress/issues/1417 (this is more about using async/await with the Cypress API, cy.xxx)
• https://github.com/cypress-io/cypress/issues/3724
• https://github.com/cypress-io/cypress/issues/3883
• https://github.com/cypress-io/cypress/issues/3497

Current behavior:

The assertion fails(in red) but the overall test still passes,

Desired behavior:

Asserts with cy work in async test case

Steps to reproduce: (app code and test code)

Use the following testing code with async and notice the test succeeds.

https://github.com/MadLittleMods/cypress-test-tiny/pull/1/files

const generateFixtures = function() {
  return new Promise(resolve => {
    setTimeout(() => {
      resolve("bar");
    }, 1000);
  });
};

describe("Some page", function() {
  it("shows something", async function() {
    const fixtures = await generateFixtures();

    cy.visit(`http://google.com/?foo=${fixtures}`);

    cy.contains("somethingthatodoesNOTexist");
  });
});

Versions

• Cypress 3.4.0
• Node v10.15.1
• Windows 10

[jennifer-shehane]

@MadLittleMods The example above is missing information to run successfully, notably the apiBaseUrl and jQuery dep.

Please provide a completely reproducible example that we can run that shows this bug and we will reopen the issue.

[MadLittleMods]

@jennifer-shehane Here is a barebones test case that demonstrates the problem, https://github.com/MadLittleMods/cypress-test-tiny/pull/1/files

[jennifer-shehane]

Thanks for providing a reproducible example!

[drFabio]

Not sure if related but this test is also failling with native promises mixed with cypress promises with this code:

```javascript
it('It fails on promise but it passes', () => {
new Cypress.Promise((resolve, reject) => {
Promise.reject(new Error('Error from native promise')).catch(err => {
reject(err);
expect(true).to.be.false;
});
});
});

````
It produces:

it also passes while failing on:

```javascript
it('It fails on promise but it passes', () => {
new Cypress.Promise((resolve, reject) => {
Promise.reject(new Error('Error from native promise')).catch(err => {
reject(err);
});
}).catch(err => {
expect(true).to.be.false;
});
});
````

[trucklos]

Temperorary workaround for me is to wrap the promise in a Cypress command. Note that errors in catch will just time out the test that calls this command. In my case, in my commands js:

```javascript
Cypress.Commands.add('resetSomeData', function() {
return new Cypress.Promise((resolve, reject) => {
graphqlFetchObject
.request("mutation { doSomething }")
.catch(function(err) {
let didDeleteData = false;
expect(didDeleteData).to.be.true;
reject(err);
})
.then(function(resp) {
resolve(resp);
});
}).then((resp) => {
expect(resp.didItHappen).to.be.true;
});
})

[agugut]

Temperorary workaround for me is to wrap the promise in a Cypress command. Note that errors in catch will just time out the test that calls this command. In my case, in my commands js:

Cypress.Commands.add('resetSomeData', function() {
    return new Cypress.Promise((resolve, reject) => {
      graphqlFetchObject
        .request("mutation { doSomething }")
        .catch(function(err) {
          let didDeleteData = false;
          expect(didDeleteData).to.be.true;
          reject(err);
        })
        .then(function(resp) {
          resolve(resp);
        });
    }).then((resp) => {
        expect(resp.didItHappen).to.be.true;
    });
})

Hi,
I've the same problem
Can you help me to apply the worarround?

[ryan-snyder]

@jennifer-shehane I would like to try and fix this if no one else is currently looking at this. Would this be in the driver package? Sorry, still a little new at what package pertains to what

[ryan-snyder]

So I've started to take a look at this. Was able to get the test not passing but I'm not able to trigger a test failure. With the async function, mocha triggers the test finish before it's actually finished. Was able to get around that by checking if all the commands are done, but If I do that, mocha never triggers the test finish event at all. Sorry the issue I linked was async describe block. Doesn't seem to be an issue with an async it block. Still seems to be something with mocha unless I'm missing where cypress triggers the test end event

[Bkucera]

So this issue is due a bug where Cypress doesn't support async/await tests bodies or tests that return promises. This wouldn't be too hard to fix, but we need to prioritize it. We'll need to rangle all the issues that are affected by this

[ryan-snyder]

@Bkucera Would you happen to know if it's just a cypress issue? Or is it related to mocha as well? I was trying to work on a fix for it but I couldn't manage to get the mocha test:end event to fire at all. Just curious, I'll probably leave it to you guys.

[Bkucera]

@ryan-snyder its a cypress issue. Mocha supports this, but we do some nasty hacks on top of mocha and likely didn't wire something up properly, such as awaiting a promise if it's returned.

[ryan-snyder]

Gotcha, that makes sense

[Bkucera]

I'll likely spike into fixing this, possibly it will be simple. If not I'll work on getting it prioritized.

[ryan-snyder]

@Bkucera Sounds good. I'll probably do a little of poking around on my own just because I want to get more familar with the codebase. If you need me to do anything, ping me

[ryan-snyder]

Interesting to note that if you do

cy.contains('something')
const fixtures = await generateFixtures()
// rest of code as usual

you get the error Cypress detected that you returned a promise from a command, while also detecting that you invoked one or more cypress commands from inside that promise

Not sure if that's expected? I wouldn't think so because the promise doesn't contain any cypress commands? But I could be wrong

[brian-mann]

@Bkucera and I have been over this a bunch of times before and it's something on our end that we need to fix. It has to do with the interop between promises being returned to the test, and Cypress also knowing that commands were enqueued.

We can generally always figure out how to do the right thing, but it will involve doing things like adding a .then() and additionally monkey patching Promise.prototype.then to know when the promise chain is settled, and to know whether or not cypress commands were enqueued within each .then() block.

We need to do things like intelligently throw if you've enqueued cypress commands inside of a native promise .then() but not returned cy, or we need to automatically figure this out and prevent the .then() from resolving until the cypress command chain has resolved inside of there. I'm in favor of the latter.

[ryan-snyder]

The latter sounds a lot like https://github.com/cypress-io/cypress/pull/5204 that I did for .within(). I don't know if we can somehow extend that solution to .then as well? Or would we need to implement it separately for .then()

[ryan-snyder]

To be honest though, we could probably do a much better solution across all nested commands.

[agugut]

Hello are there any workaround to this case??

[Machiaweliczny]

What is the point of having testing tool that doesn't even assert properly? This should have P1

Can someone guide me to related code?

[nabilamin]

Is there a workaround / solution for this issue yet? Experiencing the same problem.

[ryan-snyder]

A workaround would be to just not use async test cases.

[nabilamin]

@ryan-snyder Was able to solve my problem with a cy.wrap() ...
looks something like this

cy.wrap(
   AWS.listAttachedRolePolicies('Admin').then((list) => {
         expect(5).to.be.null;
   })
);

[amiramix]

Could issue #1417 be related ?

[swathisham]

Is there any update on this issue?

[swathisham]

@jennifer-shehane @brian-mann I am getting the same issue.. assertion fails but test shows as PASSED.

[agugut]

Hello I have the same problem with a test that compare 2 array content with deep equal.
Any news about the topic??

[ppunjjs]

Any update on this issue? My test assertion is failed but test status is still green.

[varunatester]

Any updates on this?

[Thundix]

Any update on this ?

[bahmutov]

I believe most of these failures are due to the assertion failing AFTER the test has successfully finished, read https://www.cypress.io/blog/2020/01/16/when-can-the-test-stop/

[meissadia]

I believe most of these failures are due to the assertion failing AFTER the test has successfully finished, read https://www.cypress.io/blog/2020/01/16/when-can-the-test-stop/

Very helpful blog post @bahmutov! This approach solved my issue:

it('waits for window confirm to happen using variable', () => {
  cy.visit('index.html')
  let called

  cy.on('window:confirm', (message) => {
    expect(message).to.equal('Are you sure?')
    called = true
  })

  cy.get('#click').click()
  // test automatically waits for the variable "called"
  cy.wrap(null).should(() => {
    expect(called).to.be.true
  })
})

[bahmutov]

Nice!

Sent from my iPhone

On Feb 21, 2020, at 19:01, Meis [e-mail] wrote:


I believe most of these failures are due to the assertion failing AFTER the test has successfully finished, read https://www.cypress.io/blog/2020/01/16/when-can-the-test-stop/

Very helpful blog post @bahmutov! This approach solved my issue:

it('waits for window confirm to happen using variable', () => {
cy.visit('index.html')
let called

cy.on('window:confirm', (message) => {
expect(message).to.equal('Are you sure?')
called = true
})

cy.get('#click').click()
// test automatically waits for the variable "called"
cy.wrap(null).should(() => {
expect(called).to.be.true
})
})
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[pshynin]

It looks like there is an open task since January 03 #6088, any chance we can prioritise this ?

As a temporary fix, return entire chain in the test resolved the problem for me (test marked failed finally). So, with async tests we need to invoke the callback when the test is complete.

Since cypress does not handle this behavior and we need to use a standard mocha way, is it possible to remove warning from the cypress logger :

Cypress Warning: Cypress detected that you returned a promise in a test, but also invoked one or more cy commands inside of that promise.
While this works in practice, it's often indicative of an anti-pattern. You almost never need to return both a promise and also invoke cy commands.

It looks like my issue partially related to recent Mocha upgrade: https://docs.cypress.io/guides/references/migration-guide.html#Mocha-upgrade

[jennifer-shehane]

Please refrain from commenting asking for updates 🙏

This issue is still in the 'ready for work' stage, which means no work has been done on this issue as of today, so we do not have an estimate on when this will be delivered.

If there are any updates the status label on the issue will change, a linked PR will be opened, or a comment will be made by someone on the Cypress team. Otherwise, there are no updates.

[BiosBoy]

The same story here.

@jennifer-shehane btw why we should not ping you guys? This is a major issue inside Cypress, caused a lot of people. So that's the only way to say you "Hey, we're still here and need help". As a result, make some impact on reordering among your roadmap prioritizations. Based on how much folks ask for. :(

Otherwise, it stuck in your backlog for a while...

Thanks.

[agugut]

Hello
I solved used cy.wrap instead assert
Also transform some assertion with promises in cypress tasks

[Sathish787]

Guys, I faced same issue, BDD assertions fails but test passes.
Use await before async function or line of code:

For example: I used jquery inside BDD assertions
[reason want to use jquery to avoid log messages in test runner. In this way, it takes only 20 secs to execute test whereas If I do cypress way assertion, it was taking more than 50 secs coz need to validate css properties of each cell value in ag-grid table],

assert fails not matching [1,2,1] but test was passing. No success with try and catch. I tried with await like below and finally test fails.

it.only("Sample Test", async () => {
... line of codes ...
await cy.get(".ag-header-container .ag-header-row").then((element) => {
          {few assertions}
          .....
            cy.wrap(element).find(".ag-cell-label-container").each(elem => {        
                expect([
                    Cypress.$(elem).find(".ag-sort-ascending-icon").length,
                    Cypress.$(elem).find(".ag-sort-descending-icon").length,
                    Cypress.$(elem).find(".ag-sort-none-icon").length
                ]).to.deep.equal([1,2,1])                
            })
        })
})

[thril]

I didn't think that the cy functions returned promises that may be awaited. Perhaps the await is working, but that code is misleading? I'm curious if adding a cy.wait(0) right before the cy.get has any impact, even if it's not the "correct" solution.

[jennifer-shehane]

This is still behaving incorrectly in 5.0.0

Repro

const getBar = () => {
  return new Promise(resolve => {
  setTimeout(() => {
    resolve('bar')
  }, 1000)
});
};

it('test', async () => {
  cy.contains(await getBar())
})

[Sathish787]

After spending lot of time in Cypress, I came out with solutions which helped me in writing good testcases and hope same would be for others too.

1. Don't use async in testcase like I specified in my previous post. It affects lot of areas like in assertion (doesn't assert properly sometimes) and also if you check mochawesome report it doesn't show your line of codes rather shows some other promise return function which I felt like not convincing.

Always stick to :

it("sample test", () {
    })         

(not)

it("sample test", async() {
    })   

1. Try to avoid using async functions in cypress. I replaced all my functions asynchronous to synchronous functions and it's works perfectly and same result all time. Only place I used async function where I need to do recursive to get all elements text until given column value in dynamic loading table.

Async & Sync function in customFunctions.js file

export class customFunctions {

Asynchronous function: (iter is for terminating infinite loop if text does not exist)
    async scrollhztnlAndGetAllElements(exp_value, webElement, elemlist = [], iter = 0) {
        iter++
        if (iter == 100)
            return cy.wrap([], { log: false })
        let vlist = await cy.get(webElement, { timeout: 10000, log: false }).text({ log: false }).promisify()
        elemlist = elemlist.concat(vlist)
        if (elemlist.includes(elemlist.find((value) => value.includes(exp_value)))) {
            return cy.wrap(Cypress._.uniq(elemlist), { log: false })
        } else {
            cy.get(webElement, { log: false }).last({ log: false }).click({ waitForAnimations: false, log: false })
            return customFunct.scrollhztnlAndGetAllElements(exp_value, webElement, elemlist, iter)
        }
    }
 Synchronous function:
  replaceCharInArray(exp_array, emp_arr = []) {
        exp_array.forEach(evalue => {
            let value = evalue.replace("(mm)", "").replace("(000s)", "").trim()
            emp_arr.push(value)
        })
        return emp_arr
    }

}
export const customFunct = new customFunctions() 

And in testcase, I import class file and called both async function (scrollhztnlAndGetAllElements) and sync functions (replaceCharInArray) as below shown. Synchronous function doesn't need to be wrapped and only async function need to be wrapped as in below sample code.

In testcase file:
import { customFunct } from '../../support/customFunction'

it("Column Header Test", () => { 
     cy.wrap(customFunct.scrollhztnlAndGetAllElements(Cypress._.last(expectedHeaders), locator.tableColumnHeader))
              .then(actualHeaders => {
                     expect(customFunct.replaceCharInArray(actualHeaders)).to.deep.equal(expectedHeaders)  
                     expect(Cypress._.join(actualHeaders)).to.not.include("undefined")
    })
})

(OR) If it is single assertion, i would have done like this for values from async function:

it("Column Header Test", () => { 
     cy.wrap(customFunct.scrollhztnlAndGetAllElements(Cypress._.last(expectedHeaders), locator.tableColumnHeader))     
          .should("to.deep.equal", expectedHeaders)                       
 })

If you want to get single element value, then use Jquery inside thenable or each:

cy.get(locator.tablerowlist).each((elem, indx) => {
       let nameColumnValue = Cypress.$(elem[indx]).text()
      ....
     ....
      rest lines of codes
})

1. If you want to do multiple assertion for list of elements then you can use similar to below function so that execution would be faster than usingcy.get(locator).each(elem=> {})

sortCheckElements(elements, len = 0) {
        for (var i = 0; i < Cypress.$(elements).length; i++) {
            if (Cypress.$(elements[i]).has(locator.sortAscIcon).length > 0 &&
                Cypress.$(elements[i]).has(locator.sortDescIcon).length > 0 &&
                Cypress.$(elements[i]).has(locator.sortNoneIcon).length > 0) {
                len = len + 1
            }
        }
        if (len === Cypress.$(elements).length)
            return true
        else
            return false
    }

hgHeaderCssPropertyValidation(elements, len = 0) {
        for (var i = 0; i < Cypress.$(elements).length; i++) {
            if (Cypress.$(elements[i]).css("font-size") === "11.5px" &&
                Cypress.$(elements[i]).css("font-weight") === "700" &&
                Cypress.$(elements[i]).css("color") === "rgb(255, 255, 255)" &&
                Cypress.$(elements[i]).css("text-overflow") === "clip" &&
                Cypress.$(elements[i]).css("white-space") === "normal" &&
                Cypress.$(elements[i]).css("text-align") === "center") {
                len = len + 1
            }
        }
        if (len === Cypress.$(elements).length)
            return true
        else
            return false
    }

In testcase, use:

cy.get(locator).then(elem => { expect(customFunct.sortCheckElements(elem)).to.be.true })

cy.get(locator).then(elem => { expect(customFunct.hgHeaderCssPropertyValidation(elem)).to.be.true })

1. If you want to assert list of values. then install cypress-commands library (here I use to('array')) from Cypress plugin page.

cy.get(locator.tableColumnHeader).text().to("array")
      .should("to.deep.equal", ["Time", "Legacy (mm)", "Standard Origin (mm)"])

1. Likewise, instead of using cy.fixture and thenable. I used import fixture file and assigned to a variable and then I use variable in all testcases.

In testcase file:
const drillData = require('../../fixtures/drillData.json')

it("some test" () => {
   cy.server()
   cy.route("POST", APIEndPoint, drillData).as("dataDrill")
})

1. You could place expected values into json file in fixture and could use in testcase:

In UserInfo json file:
{
"expectedCusInfo": [{"Region", "Time Period", "XYZ", "ABC"],
"expectedMessage": "User has been added successfully"
}

In testcase file:

const userData= require('../../fixtures/UserInfo.json')

it("some test" () => {
  cy.get(locator).text().to("array").should("be.deep.equal", userData.expectedCusInfo)
  cy.get(locator).should("have.text", userData.expectedMessage)
})

1. Also, instead of using UI elements to make cypress to wait use api call. Example, for page to load, instead of waiting for progress bar to disappear. I followed to wait for api call to finish, here api is not mocked by data still it hits application server.

it("some test", () => {
  cy.server()
  cy.route("GET", ApiEndPoint).as("customerData")
  cy.visit("/customerpage")
  cy.wait("@customerData")
})

Hope, this would be helpful. Object chaining better than using async in testcase to get reliable results all time.