Cypress: Assertion inside test fails, but test doesn't fail

[Atticus29]

Current behavior:

An assertion after a cy.wait() fails, but test passes.

Desired behavior:

Test fails.

Steps to reproduce: (app code and test code)

This test should work as a standalone.

describe('Standalone temporary test for github issue', function() {
  it('navigates to known bottle nosed dolphin encounter page and re-runs matching, waits ten minutes, and then checks the output of the iaResults from the taskId of the known bottle nosed dolphin encounter github issue version', function(){
    cy.request({
      method: 'POST',
      url: 'https://www.flukebook.org/LoginUser',
      form: true,
      body: {
        username:'tempuser',
        password:'tempuser'
      }
    })
    .then((resp)=>{
      expect(resp.status).to.eq(200);
      cy.log(resp.requestHeaders.cookie);
    });
    cy.visit('https://www.flukebook.org/welcome.jsp');
    cy.url().should('match',/welcome/);
       cy.visit('https://www.flukebook.org/encounters/encounter.jsp?number=5f555acb-1a4a-4753-8313-75464a9b1894');
       cy.get('div[class=image-enhancer-menu]').click();
       cy.get('div[class=menu-item]').contains('start another matching job').click({force: true});
       cy.get('p[id=activeTaskId]', {force: true}).invoke('text').then((theText)=>{
         let bottleNoseTaskId = theText;
         cy.log(bottleNoseTaskId);
         cy.visit('https://www.flukebook.org/iaResults.jsp?taskId=' + bottleNoseTaskId);
         cy.contains('waiting for results').should('exist');
         cy.get('span[class="annot-info-num"]', {force: true}).should('not.exist');
         cy.get('span[class="img-info-type"]', {force: true}).should('not.exist');
         cy.wait(480000);
         cy.visit('https://www.flukebook.org/iaResults.jsp?taskId=' + bottleNoseTaskId);
         cy.get('span[class="annot-info-num"]', {force: true}).should('exist');
         cy.get('span[class="img-info-type"]', {force: true}).should('exist');
         //TODO test passing despite assertion failing!! WTH
       });
   });
});

Versions

Cypress package version: 3.1.3
Cypress binary version: 3.1.3
Google Chrome Version 72.0.3626.96 (Official Build) (64-bit)
MacOS Mojave Version 10.14.3 (18D109)

[jennifer-shehane]

Hey @Atticus29, when running this, my test actually passed - and when failing, properly failed. So, I've got a few more questions.

• Is this failing during cypress open or cypress run?
• You're selecting Chrome 72 to run it in?

I also see that during the test run, a new tab is opened within the browser. I can guess that this may be causing some issues since the tab with the currently running test would no longer be in focus.

As a side note: Is this page supposed to eventually show results after 'waiting for results'? If so, I would probably rewrite this test to not have a static wait time, but write something like:

cy.contains('.waiting', 'waiting for results', {timeout: 480000}).should('not.be.visible')

[Atticus29]

Hi, @jennifer-shehane !
I was using cypress open. Should I be using cypress run?

Right now, I'm just testing this stuff from my local machine (eventually, this will be kicked off in circleCI). So, Re: Chrome 72 - that's the current version of my browser (should I be using something different? Is Cypress secretly using something else?)

I've been reading on the cypress site that static waits are not best practice; your suggestion I'm sure will come in handy! Thank you for that!

[Atticus29]

@jennifer-shehane if you shorten the cy.wait() to cy.wait(60000), do you then get a failed assertion but a passing test? For that matter, is there any cy.wait() duration that passes the test but fails the assertion?

[jennifer-shehane]

Hmm, no @Atticus29, my test always accurately passes or fails even with different wait times. 😢

cypress open and Chrome 72 should be fine, I just wanted to make sure I was as close as possible to replicating the issue as you have it.

[Atticus29]

Hi, Jennifer et al.!
I managed to reproduce my issues as a video. While on a public bus (I was
alone on the bus at list initially; I'm not a monster). Perhaps it will
clarify the issue on my end? I apologize about the sound quality; a
lavalier mic on a bus is not ideal, apparently.
https://youtu.be/LsCm1_5UTB4

Many thanks again,
Mark

On Mon, Feb 18, 2019 at 7:02 PM Jennifer Shehane [e-mail]
wrote:

Hmm, no @Atticus29 https://github.com/Atticus29, my test always
accurately passes or fails even with different wait times. 😢

cypress open and Chrome 72 should be fine, I just wanted to make sure I
was as close as possible to replicating the issue as you have it.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/cypress-io/cypress/issues/3497#issuecomment-464961315,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACpZmNmTEhmTqCfv83w94jYMyG0unIlIks5vO2lSgaJpZM4a-aaS
.

[jennifer-shehane]

Ok, this reminds me of this similar issue, that we did not get any more info back on: https://github.com/cypress-io/cypress/issues/2720

We've discussed the possibility of why this may happen. From what I remember, Cypress may initially find the element of cy.get(), then the element disappears (is removed from the DOM) so that the second part of the assertion fails and retroactively updates that line to be red, but the test still passes. This is just a theory though - that there's some sort of timing situation with how the assertions are run basically.

One other thing that stands out to me is the fact that you are calling cy.visit() with the same url that you are currently navigated to. There was an issue previous to Cypress 3.1.5 that would cause some strange behavior when doing this. https://github.com/cypress-io/cypress/issues/1311 Can you try updating to version 3.1.5 and see if this is resolved?

[Atticus29]

Hi, @jennifer-shehane ! I can confirm that the update to version 3.1.5 worked (cypress version says 3.1.5) and the assertion still fails and the test still passes. Thanks for your continued help with this.

[Atticus29]

FYI I also get the same unexpected behavior even if I remove that cy.visit() line.

[Atticus29]

What other info. can I provide? Anything actionable on my part?

[MarekJavurek]

@jennifer-shehane FYI I have similar issue:

Seems like #2413 looks pretty similar... Thanks for reply.

[MarekJavurek]

@jennifer-shehane

EDIT:
this was my problem!! I Have to comment whole function (in commands.js)
Cypress.on("fail", (err, runnable) => { // debugger; });

fixed:
/* Cypress.on("fail", (err, runnable) => { debugger; }); */

For me it seemd like I have just subscribed to event "fail", So I have no idea I have to propagate error...

[jennifer-shehane]

@Atticus29 Are you listening to the Cypress.on("fail"... event anywhere? In your support / plugin files even?

[Atticus29]

@jennifer-shehane yes I am! Interesting!

[jennifer-shehane]

If you listen to Cypress.on("fail"... and do not later throw the error, then the test will display as passing even though it failed, since you did not propogate the error onwards. See this documentation on how to have the test fail.