Cypress: Simple tests pass in opened UI but fail on command line (application error)

[ndonnellan]

Current behavior:

Simple tests against web-app work, but when I run them in the headless mode, they fail and the error given is an application error that seems unrelated (or an indication that the web-page didn't finish loading.
Uncaught SyntaxError: Unexpected token (

If I replace the 'uncaught:exception' event in cypress like the documentation suggests,

Cypress.on('uncaught:exception', (err, runnable) => {
  return false
})

I get a totally different error complaining about my usage of get

CypressError: Timed out retrying: Expected to find element: 'input[name=email]', but never found it.
I suspect this is because the page never finished loading, so it's probably not important.

Desired behavior:

My tests should run in both interactive mode and headless mode.

How to reproduce:

I'm not sure how to reproduce except with my app (React/Redux with Apollo). I have confirmed that I can run the example cypress tests fine (in a clean state) both from UI and from command-line.

Test code:

const DEFAULT_URL = 'http://localhost:3000'

describe('Basic e2e tests', function() {
  it('.should() - assert that <title> is correct', function() {
    cy.visit(`${DEFAULT_URL}/login`)
    cy.title().should('include', 'React App') // <----- this works
  })

  context('Logging in', function() {
    beforeEach(function() {
      cy.visit(`${DEFAULT_URL}/login`)
    })

     it('Should be able to login', function() {
      cy.get('input[name=email]') // <------ failure here

I tried adding a wait, but it seems like it never loads. The screenshot definitely makes it look like it never loads.

Additional Info (images, stack traces, etc)

Full stack trace of the code where I let the error propagate:

CypressError: Timed out retrying: Expected to find element: 'input[name=email]', but never found it.
      at Object.cypressErr (http://localhost:3000/__cypress/runner/cypress_runner.js:65978:11)
      at Object.throwErr (http://localhost:3000/__cypress/runner/cypress_runner.js:65943:18)
      at Object.throwErrByPath (http://localhost:3000/__cypress/runner/cypress_runner.js:65970:17)
      at retry (http://localhost:3000/__cypress/runner/cypress_runner.js:60079:16)
      at http://localhost:3000/__cypress/runner/cypress_runner.js:52597:18
      at tryCatcher (http://localhost:3000/__cypress/runner/cypress_runner.js:6268:23)
      at Promise._settlePromiseFromHandler (http://localhost:3000/__cypress/runner/cypress_runner.js:4290:31)
      at Promise._settlePromise (http://localhost:3000/__cypress/runner/cypress_runner.js:4347:18)
      at Promise._settlePromise0 (http://localhost:3000/__cypress/runner/cypress_runner.js:4392:10)
      at Promise._settlePromises (http://localhost:3000/__cypress/runner/cypress_runner.js:4467:18)
      at Async._drainQueue (http://localhost:3000/__cypress/runner/cypress_runner.js:1200:16)
      at Async._drainQueues (http://localhost:3000/__cypress/runner/cypress_runner.js:1210:10)
      at Async.drainQueues (http://localhost:3000/__cypress/runner/cypress_runner.js:1084:14)

• Operating System: macOS Sierra 10.12.6 (16G1212)
• Cypress Version:1.4.2
• Browser Version:Chrome?

[brian-mann]

I am almost 100% positive this is because when running headlessly we use Electron (Chromium 53) - and when you launch the browser it is a much newer version of Chrome.

The difference is that the older browser likely does not have newer JS features which is why it's failing. Turning off uncaught exceptions won't actually make the page render - it just hides catching that error. If you were to open your dev tools when running from Electron you would see the error.

This document explains all of this: https://docs.cypress.io/guides/guides/launching-browsers.html

However we are about to release 2.0.0 very soon - like today or tomorrow which bumps Chromium 53 to 59 - which is much newer and these types of differences will be covered up.

I'm closing the issue assuming this is the problem.

[brian-mann]

You can also switch the browser to Electron in the GUI to inspect the error, or make it run in headed mode with --headed when running from the CLI.

[brian-mann]

Or you could also just run in Chrome from the command line with --browser chrome argument.

https://docs.cypress.io/guides/guides/command-line.html#

[mkatrenik]

@brian-mann I have same problem - piece if code in headed electron it works, in headless electron it fails. But there is 1 strange thing - it fails on selector which returns 1 match, this is visible in inspector, but in UI it says NO matches.

[brian-mann]

@mkatrenik there's no way based on what you've provided to be of any assistance. In headless you get a full recorded video + screenshots. You'd need to post all of those, plus your test code.

If I were to guess it's likely a problem of elements being re-rendered while Cypress code is running. It's possible that you can lead Cypress down a path of elements, deeper and deeper, and then a parent element is rerendered which detaches all of the children. In those cases it's impossible to proceed and it will fail.

Best practice is to always perform actions as high up in the chain as possible. In your case you have a get and then find and then click. It's best to do get and then click to avoid this as much as possible.

Another option is to add more assertions and "guards" around Cypress so it doesn't proceed until the state of your application has settled first.

[brian-mann]

There is little difference between running headed and headlessly in Electron - except that headless is optimized for long runs and certain things are disabled which means it will technically run faster than headed. This is where it could deviate. However, it could just be that you've written flaky test code and that if you were to repeat the same test say 100 times, you'll see the same failures in the UI too. That's another best practice - when you have inconsistent results, repeat the test (in a loop) so you can see what it looks like the moment that it fails.

[mkatrenik]

So for others who may have same issue - I tried to run it in docker/browsers image headlessly and there it works. Looks like some difference between osx and linux electron... Anyway thanks for suggestions.

[brian-mann]

Different environments will stress the browser in slightly different ways - there are different network configurations, timings, etc.

From our experience, this is most likely an issue with the way you've written your test and will likely crop up even in the environments where its primarily passing.

[ndonnellan]

I didn't respond to the closing, but transpiling my code to a lower ES version made my issue go away so I don't think your bug (@mkatrenik ) is the same.

[ryan-mulrooney]

I'm having a similar issue... When I run headlessly I incur essentially the same error. If i run via the CLI but use -b chrome the test fails again, but freezes when typing in an input (i have 4 inputs in an iframe, and it always fails when typing in one of them, a different one each time).

However, when I run the test in the test runner, it always passes. I guess there's some fundamental difference between running the test via the CLI compared to the test runner - is it a combination of the memory allocation between that, and the local version of my app that's under test @brian-mann ?

[z0d14c]

I have a test that passes in electron/chrome (headless or no) on Mac OS but fails on headless ubuntu :(

[raelgc]

In my experience, most of these cases (when tests fails only in headless mode) it's due the faster speed.

Example: in my app, after the login, we have a request to get more info about the user, and only then store some data in local storage and cookies.

While testing in Chrome, or even with Electron, it works. Headless, it was randomly failing (because the request was not completed, so user info was not saved, redirecting to login again).

To fix, I just include a check for some element that should be displayed only after the user request is finished, like one of item of the logged user menu, like cy.contains('Home').

[predictiveXu]

I just found that running headless never waited when the page was loading. The issue for me is that running tests in chrome or headed only works for the first 5 tests. After that, the chrome or electron became very slow, and sometimes chrome would be broken and show the error message 'Aw, Snap'. RIP

[amiral-jion]

I'm having the same issue, the tests passed correctly in the with open and it fails sometimes (in an indeterministic way) with run

[Vindexus]

I am having the same issue as well.

[paxcodes]

I had this issue. It was failing on cypress run as well as on cypress open if my browser is Electron. The test consistently passes on an open UI using Chrome/Canary even if I loop the test 2000 times.

I followed @brian-mann's advice and rewrote my tests so that it "waits" until the element I want to assert is visible. My test now consistently passes across all environments.

    cy.get(element).then(element => {
        const actual = element.text();
        const expected = "sometext";
        expect(actual).to.equal(expected);
      });

to

cy.get(element)
      .should("be.visible")
      .then(element => {
        const actual = element.text();
        const expected = "sometext";
        expect(actual).to.equal(expected);
      });

[vedmant]

I have the same issue with latest Cypress, is there an official workaround for this issue?

[Q-back]

In my case I cy.wait('@requestAlias') was the key. It seems like I tried to make assertion before underlying request finished, which didn't happen in gui mode (because of longer rendering??)
https://docs.cypress.io/guides/guides/network-requests.html#Waiting

[kirill578]

I had [].flatMap() in my code that worked in chrome but failed with cypress. An easy fix, can be replaced with map().reduce()

[imartinflores]

Hi, I am having same issue where my test works on cy.open but fails on cy.run , I am using cy.request to POST a call to my api, giving timeout when using cy.run, why could it be ?

The API call is POST to login.microsoftonline.com/mycompany.onmicrosoft.com

Thanks.

[alexravenna]

I just had the same discrepancy between headless Chrome 84 and GUI Chrome 84, which really stumped me for a bit.

I was adding data to the internal data structure of my Vue SPA through objects attached to window as part of a test fixture (a test data API if you will) so that I didn't have to have Cypress "click together" each new configuration to be tested. Adding this data should have changed the state of my UI to a state that caused certain elements to become visible (i.e. a table should be filled), so that I could further interact with them through my test API. And this is what happened when running

vue-cli-service test:e2e --browser chrome --mode cypress

However, the following command

vue-cli-service test:e2e --browser chrome --headless --mode cypress

couldn't find the elements I was telling it to look for. I was expecting them to "just already be there" since that's the behavior I was used to from testing my app in real life and through the Cypress GUI, but the screenshots and videos proved my assumptions wrong.

Since I wasn't trying to explicitly test the existence of the elements added implicitly through my test API (but rather, what came aftewards) I, well...didn't 😆. And I'm guessing that's what Cypress running in headless Chrome didn't like?

So basically I needed to assert the existence of the new element(s) in the table before I could further use them. This changed my code from

cy.window().its('testApi').invoke('addData', data);
cy.window().its('testApi').invoke('selectRow', rowName);

// The actual test assertions I wanted to make

to

cy.window().its('testApi').invoke('addData', data);
cy.get(`${tableId} > table > tbody`).contains(rowName)
cy.window().its('testApi').invoke('selectRow', rowName);

// The actual test assertions I wanted to make

I still don't entirely understand why this made a difference between --headless and not, but maybe this will help someone else that stumbles across this problem.

I believe this is similar to what paxcodes is referring to in their comment above.

[jennifer-shehane]

I would suggest following on recommendations on troubleshooting why tests are fail in CI but pass locally: https://on.cypress.io/using-cypress-faq#Why-do-my-Cypress-tests-pass-locally-but-not-in-CI