Cypress: Cypress is running flaky tests in CircleCI Test Environment

[sully1313]

• Operating System: OSX
• Cypress Version: 1.4.1

Is this a Feature or Bug?

not sure

Current behavior:

We are currently using Cypress to test both our customer facing website and our Admin-Panel. These tests are run each time someone pushes their branch up to staging by CIRCLECI. Currently, the tests are producing inconsistent results(passing 1 time then failing the next) in this environment even though they are all passing locally.

Desired behavior:

We would like to be able to run Cypress tests in the CIrcleCI env consistently.

How to reproduce:

We are using the following configuration for our CIRCLECI cypress test setup

  test_cardash-web:
    <<: *cardash_circleci
    steps:
      - attach_workspace:
          at: ~/

      - *setup_docker
      - *create_ssh_key
      - *set_permissions_ssh_key
      - *npm_link_cardash-cli
      - *load_db

      - run:
          name: Load cardash-api
          command: |
            # Sets the env variables in our current login shell
            `cardash setup api`

            # Open the tunnel for this step
            cardash tunnel up

            export DB_PORT="3306"
            export DB_PASS="root"
            export VCDB_PORT="3306"
            export VCDB_PASS="root"
            export NODE_ENV=development
            env | sort
            # Instantiate cardash-api
            PORT=3000 node scripts/dev.js &> /tmp/cardash-api-output.log
          working_directory: packages/cardash-api
          background: true

      - run:
          name: Boot cardash-web
          # Redirect both stdout and stderr to webserver-output.log with "&>"
          command: |
            `node ~/repo/packages/cardash-cli/cardash-setup.js api`
            node scripts/generateSitemap.js
            NODE_ENV=local ./node_modules/.bin/webpack-dev-server --env.dev --hot &> /tmp/cardash-web-output.log
          background: true
          working_directory: packages/cardash-web

      - run:
          name: Run Cypress tests
          command: |
            Wait 30s for cardash-api to finish loading
            sleep 30
            ./node_modules/.bin/cypress run --headed
             exit $?
          working_directory: packages/cardash-web

      - store_artifacts:
          path: packages/cardash-web/cypress/videos

      - store_artifacts:
          path: packages/cardash-web/cypress/screenshots

      - store_artifacts:
          path: /tmp/cardash-api-output.log

      - store_artifacts:
          path: /tmp/cardash-web-output.log

We are also using the following script command to run the tests locally:

"test": "node_modules/.bin/cypress run --headed"

Please let me know if you need any additional info.

[brian-mann]

Your circle config likely has nothing to do with this - the differences you're experiencing are likely the differences between Electron and Chrome. This is explained in our docs here.

https://docs.cypress.io/guides/guides/launching-browsers.html#

Likely all you need to do is run Electron locally to experience those some hiccups. We are currently upgrading Electron right now from 53 to 59, which brings it much closer to Chrome in feature parity which will likely reduce flakiness experienced from running the different browsers.

Alternatively you could install Chrome in CI and run it there.

[sully1313]

Hi @brian-mann, Thanks for the prompt response and the exciting update! We are currently running the headed electron browser locally and in the CI environment as you suggested in my last issue and we are still receiving flaky inconsistent results.

We will be taking your suggestion to install Chrome in CI. Hopefully, this will solve the inconsistency issue. If not I will reopen this issue. Have a great weekend.

[brian-mann]

Released in 2.0.0.

[chrisvfritz]

Just reporting that I'm still seeing an issue with flaky tests on Circle CI (one test in particular), using Cypress 2.1.0. I have more details here. Happy to provide any additional information as well. Thanks for your great work! 🙂

[brian-mann]

@chrisvfritz I looked at your test code - and the issue isn't really with Cypress - it's the way you're writing your tests.

You're writing your tests with several well known and documented anti-patterns, and this is leading up to the failure.

Specifically you are building up state between your tests, which is documented here: https://docs.cypress.io/guides/references/best-practices.html#Having-tests-rely-on-the-state-of-previous-tests

I really recommend you read all the best practices, and you also watch the latest talk we gave which specifically shows you best practices when writing tests.

Now - as for the flaky test, the reason this is happening is because you are not guarding Cypress well enough. You haven't given it enough information to know to wait until a condition has happened.

If you look closely at the Command Log you will see what's going on. When you click "Log Out" you're not waiting on anything - you're just immediately searching for <a>. What's happening is that Vue has not yet transitioned to the Login page because it renders async. Cypress is so fast, that it begins processing the next command before this transition.

When it does that, it immediately finds 3 anchor links. Why? Because there are 3 anchor links in the DOM and all you've told Cypress is to find <a>. After the cy.get resolves, it now has the subject set to 3 <a> DOM elements. Now you've told Cypress to search inside of those <a> to find Log in. Guess what? None of those 3 anchor links have the text content "Log In" in them. That's why it's failing.

The problem is that you've broken up your steps too much. There's no reason to do this. You've sent Cypress down a tunnel that is always going to resolve - but resolve to the wrong elements.

We should actually write a best practice on this - but you should chain as few commands together as possible. Give Cypress enough information to know what you want instead of chunking it into many little operations as possible. Those little operations aren't providing Cypress with enough context, and instead its going down the wrong direction.

Changing it to any of the following would work:

```js
it('logs the user out when clicking on the "Log out" link', () => {
// fine and most succinct
cy.contains('Log out').click()
cy.contains('Log in')

  // also fine to give contains the selector
  // in case other elements contain the same text
  cy.contains('a' 'Log out').click()
  cy.contains('a', 'Log in')

 // you could also guard cypress from querying
 // by ensuring that the URL has changed prior
 // to moving on...
 cy.contains('Log out').click()
 cy.url().should('contain', '/login') // wait until the page has transitioned...
 cy.contains('Log in')
})

``

[brian-mann]

In this case - the click was fine - it was the second query for <a> which returned three elements.

These three elements were for the PREVIOUS page, and none of them contained "Log in". As the command retried (since it was unsuccessful), the page transitioned, and the three <a> became detached from the DOM, but cy.contains was already bound to them. At that point its impossible for it to recover.

[brian-mann]

I could tell from the video by looking at the Command Log. I could have likely figured it out from the Screenshot too.

[brian-mann]

BTW this goes back to our discussion about synchronizing the command execution with the underlying Vue queue. After performing the cy.click() the command queue would pause until Vue flushes its async reactions, and then move on later.

It would prevent needing to be as explicit or "guard" Cypress from moving on too soon.

[bahmutov]

Yeah on vue specific guard - but the app could do even more stuff behind the scenes! So adding guards that ensure what you think happens really happens in framework agnostic way is the best. Also full even log would show the wrong order of events and be very helpful

Sent from my iPhone

On Mar 18, 2018, at 16:29, Brian Mann [e-mail] wrote:

I could tell from the video by looking at the Command Log. I could have likely figured it out from the Screenshot too.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[chrisvfritz]

Thank you very much for the tips! I did read all the docs some time ago, but it seems some things didn't fully sink in at the time. 😅 I'll be diving back in soon for a refresher now that I've worked more with Cypress.

Something that may be helpful for Vue in particular could be to hook into Vue.nextTick, which accepts a callback that will resolve as soon as the current async render cycle is complete.

[chrisvfritz]

One follow-up note. I think I picked up the pattern of:

cy.get(selector).contains(text)

from the contains doc, where that's listed as "Correct Usage". If I'm understanding correctly, it sounds that's actually _incorrect_ usage and the correct usage should be:

cy.contains(selector, text)

Is that right?

[chrisvfritz]

And following up on Vue.nextTick, here's our source implementation in case it's useful. 🙂

[brian-mann]

It's actually not incorrect use - it just has slightly different semantics, and each is useful in its own particular way. You may want to start with a parent for another purpose and query into its children. But if you're directly targeting an element based on its text content without needing to query off of an existing element, you just use cy.contains directly.

The selector argument to contains is what enables it to target "higher up elements" on the page than the deepest one containing the text content. This is adding behavior that is not possible to do otherwise. It's all documented but it may need to be made clearer.

All in all, each use case is slightly different and there is a proper time to use each.

[chrisvfritz]

Thanks for the clarification! 🙂

[chhuang]

@brian-mann what do you mean by "guard cypress"? In your example,

cy.contains('Log out').click()
cy.url().should('contain', '/login') // wait until the page has transitioned...

I'm trying to do something similar, the only difference is that my click (the first line) triggers a client side routing instead of a browser redirect. But when I check the URL (the second line), sometimes the URL hasn't changed yet (the time it takes the page to transition is uncertain, it also does a few XHR calls). This is happening mostly on CI, but sometimes also on my local.

What I want is when someone does an action (e.g. clicks on a link), they land on the correct page. Is there a better way to do this check?

[brian-mann]

@chhuang guards are assertions and company, documented here: https://docs.cypress.io/guides/core-concepts/introduction-to-cypress.html#Assertions

Per what you're trying to do all that sounds right - you'd add an assertion, but ideally you'd instead wait directly on the XHR's to complete as opposed to the side effect of the page loading and the URL switching. Those waits are automatically attuned to network requests and will wait longer for things to resolve.

Alternatively you could increase the timeout to the cy.url assertions. I believe all this is documented in various guides.

https://docs.cypress.io/guides/guides/network-requests.html#

[jeffbabuyo]

@brian-mann How come the test is working both GUI and CLI but when it comes to circleCI that's were the test is failing? even the debugging cypress is working fine..