Cypress: Ability to opt out of waiting for 'load' event before resolving cy.visit() - onload event takes too long to fire

[c-kirkeby]

Current behavior:

Cypress runs extremely slowly. Run time for 2 tiny tests is ~60-65 compared to similar Protractor test suite which takes ~20-25 seconds to complete the same tests. My timing for Cypress is based on the timer in the runner and my timing for Protractor is based on the timestamp in the console from when it starts and finishes.

Desired behavior:

Significant improvement to run time.

How to reproduce:

Using test code below.

Test code:

Cypress code:

(I would set timeout as 15000 just like my Protractor code, but it times out when I do that)

describe('Attempt to log in', () => {
  beforeEach(() => {
    cy.visit('/')
  })

  it('without a username or password', () => {
    cy.get('button', {timeout: 30000})
      .contains('SIGN IN')
      .should('be.visible')
      .click()
    cy.get('#error')
      .should('have.text', 'Please enter your username')
  })

  it('without a password', () => {
    cy.get('#username', {timeout: 30000})
      .should('be.visible')
      .type('example_username1')
    cy.get('button')
      .contains('SIGN IN')
      .click()
    cy.get('#error')
      .should('have.text', 'Please enter your password')
  })
})

Comparable Protractor code:

describe('Attempt to log in', () => {
    const EC = protractor.ExpectedConditions;

    beforeAll(() => {
        browser.get('/');
        browser.wait(EC.presenceOf($('#username')), 15000);
    });

    it('without a username or password', () => {
        element(by.buttonText('SIGN IN').click();
        expect($('#error').getText()).toBe('Please enter your username');
    });

    it('without a password', () => {
        $('#username').sendKeys('example_username1');
        element(by.buttonText('SIGN IN').click();
        expect($('#error').getText()).toBe('Please enter your password');
    });

    afterEach(() => {
        browser.executeScript('window.sessionStorage.clear();');
        browser.executeScript('window.localStorage.clear();');
        browser.refresh();
    });
});

Cypress is really great and powerful, but it's a shame that it takes so long to execute these tests.

• Operating System: Windows 7
• Cypress Version: Beta 1.0.2
• Browser Version: Version 61.0.3163.100 (Official Build) (64-bit)

[bahmutov]

Do you have ability to try same test on windows 10 for example?

Sent from my iPhone

On Oct 22, 2017, at 18:52, JheeBz [e-mail] wrote:

Operating System: Windows 7
Cypress Version: Beta 1.0.2
Browser Version: Version 61.0.3163.100 (Official Build) (64-bit)
Is this a Feature or Bug?

Bug

Current behavior:

Cypress runs extremely slowly. Run time for 2 tiny tests is ~60-65 compared to similar Protractor test suite which takes ~20-25 seconds to complete the same tests.

Desired behavior:

Significant improvement to run time.

How to reproduce:

Using test code below.

Test code:

Cypress code:

(I would set timeout as 15000 just like my Protractor code, but it times out when I do that)

describe('Attempt to log in', () => {
before(() => {
cy.visit('/')
})

it('without a username or password', () => {
cy.get('button', {timeout: 30000})
.contains('SIGN IN')
.should('be.visible')
.click()
cy.get('#error')
.should('have.text', 'Please enter your username')
})

it('without a password', () => {
cy.get('#username', {timeout: 30000})
.should('be.visible')
.type('example_username1')
cy.get('button')
.contains('SIGN IN')
.click()
cy.get('#error')
.should('have.text', 'Please enter your password')
})

afterEach(() => {
cy.reload(true)
})
})
Comparable Protractor code:

describe('Attempt to log in', () => {
const EC = protractor.ExpectedConditions;

beforeAll(() => {
    browser.get('/');
    browser.wait(EC.presenceOf($('#username')), 15000);
});

it('without a username or password', () => {
    element(by.buttonText('SIGN IN').click();
    expect($('#error').getText()).toBe('Please enter your username');
});

it('without a password', () => {
    $('#username').sendKeys('example_username1');
    element(by.buttonText('SIGN IN').click();
    expect($('#error').getText()).toBe('Please enter your password');
});

afterEach(() => {
    browser.executeScript('window.sessionStorage.clear();');
    browser.executeScript('window.localStorage.clear();');
    browser.refresh();
});

});
Cypress is really great and powerful, but it's a shame that it takes so long to execute these tests.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[c-kirkeby]

I tried to run the same test on Windows 10 (using my laptop) and it took about 37-40 seconds on average. That's still almost double the run time of Protractor. Cypress seems to take a long time to load the page and perform assertions.

[MarcLoupias]

At work i run e2e tests under windows 7 with cypress 1.0.2 and chrome 61.0.3163.100 (Build officiel) (32 bits) and it is super fast, no problem at all.

[jennifer-shehane]

I am curious why you don't just move the cy.visit('/') to a beforeEach and remove the cy.reload() from the afterEach. This should achieve the same thing.

[brian-mann]

Okay want to clarify a few things...

Cypress has a completely different architecture than Selenium based automation tools. Whereas Selenium works by executing remote commands over the network - Cypress runs in the same run loop as the application. Because of this, very few commands are executed over the network (only the ones that require a higher privilege) and therefore in almost every circumstance Cypress will run faster. I am simplifying this quite a bit, but we have had a considerable number of users rewrite their protractor tests and they rerun much faster in Cypress.

With that said - Cypress has a much different behavior and set of opinions than Selenium. For instance, our cy.visit does not resolve until the remote page fires its load event. It's technically possible for the DOM to be ready and available for work before this event such as DOMContentLoaded and there is an open issue https://github.com/cypress-io/cypress/issues/440 asking to be able to change cy.visit to resolve on that event.

Most of the time the DOMContentLoaded and the load event will resolve incredibly close to each other. Using load offers a much higher guarantee - it's impossible for scripts not to be ready at that point.

Your case seems pretty unique in that your application is taking (even in protractor) an average of 20-25s to run two tests. That is incredibly slow. Ideally your application loads in under a second or two. I'll also assume in this case you have a huge number of blocking scripts / css / images which are pushing the load event off by a huge amount. In this case its possible DOMContentLoaded is firing seconds or dozens of seconds ahead of the load event. By switching to that, it could shave off a huge amount of time. The downside is that if your scripts are bound after the DOMContentLoaded event, you'll have to account for that in your test code. If you interact with your application before the javascript is ready, it may then not react accordingly.

Because Cypress runs commands serially you don't need to add a timeout to the cy.get. Why? Because by then the cy.visit has resolved only after the page is entirely loaded. In that situation, the element should already be there. I haven't looked at protractor in several years so I'm not sure their implementation details of their visit. If they aren't waiting for the load event, that's why its running faster, but as I explained above, there is a specific reason we wait for it. We could simply choose not to wait for that event at all. But then you'd be forced into pushing timeouts into lower commands downstream.

Lastly I'm also assuming these tests are also taking longer because of that afterEach. cy.reload, cy.visit, cy.go all wait for the subsequent page to fire its load event. Therefore you're getting hit with the initial visit, the first reload and the second reload. If your page takes 20s to fire the load event that's where 60s comes from.

There's almost never a reason to ever use after or afterEach hooks in Cypress. They are almost always indicative of an anti pattern. By using them you lose the ability to work with your application at the state it was last in when the test ran. In your example, the page will always be refreshed to ground zero whenever a test ends. What you're trying to achieve could be more simply achieved as @jennifer-shehane pointed out by simply visiting your application beforeEach test. State should be reset prior to a test, not after the test.

My post makes a lot of assumptions based on the way your application works, so if I'm wrong there is of course room for more debate / suggestions. Another possible area is that because Cypress is a network proxy, it means all traffic gets routed through it locally. There are areas for performance improvements to this layer, but even with that, we have seen much better performance than any other Selenium based tool.

You'd have to provide more details related to your specific application for us to make any changes or investigate further.

[c-kirkeby]

@jennifer-shehane there didn't appear to be another way to clear the cache between tests but further reading seems to suggest that the cache is automatically cleared between each test. I've removed the afterEach hook and changed my before hook to a beforeEach, but it hasn't made a difference on the timing at all. I will note though that a similar change to my Protractor test has reduced the timing to 13 seconds.

@brian-mann thanks for the information. As above the way I wrote my tests to check out Cypress was in an attempt to mirror a couple of existing tests. One limitation of Protractor is the lack of commands available within their API, so I've had to improvise a lot.

Unfortunately I have to add a timeout to the cy.get command, otherwise the command gets fired as soon as the loading animation happens (I don't know much about the implementation of our app as I'm just a software tester) and fails to locate the element. I also have to add the cy.should assertion or Cypress will halt the loading of the application for as long as my timeout.

[brian-mann]

Do you have a public URL that we can test against? It sounds like your application loads immediately and then lazily loads in the JS required to actually use the application. In that case, yes you would need to add the timeout to the first command so it blocks until that happens.

Alternatively you could wait on an XHR to finish indicating to you that its done loading - or possibly check some other value like something local storage, etc.

[alidlorenzo]

I'm facing similar issues on Windows 10

When I run a test file browser takes 12+ seconds to boot up, and simple tests (just checking html element existence) take 12 sec each as well. I'm guessing being on Windows could be a factor but this was the only issue I found on the matter.

Here's a video of the experience: https://drive.google.com/file/d/1R9ADU7IMmLp-VmTYjbX18F8zmi4xsUEG/view?usp=sharing (ignore the flicker)

Here's an example test:

describe('login', () => {
  beforeEach(() => {
    cy.visit('/login')
  })

  it('has correct html', function() {
    cy.contains('Login to your account').should('exist')
    cy.contains('New here? Request early access.').should('have.attr', 'href', '/register')
  })
})

[arfs]

I have the same experience as well. Booting up the headless browser takes 15-20 seconds longer on Windows 10 than the same exact set up in OSX.

Again, this is just timing the startup process and not the running of the actual specs. Any ideas for improving the startup in Windows?

[jennifer-shehane]

@alidcastano @arfs That's pretty rough and definitely not what should be happening.

We'll really need a reproducible example to test against - which I know is difficult if you can't share a public url - but if it's at all possible, we'd love to look at it.

[louptony]

We had an issue where Cypress was really slow (45 ~ 50 sec each test). We found out the problem was due to Google Analytics. We change our server setting for the execution of our component test to deactivate it and now each test take arround 2 ~ 3 sec.

[jennifer-shehane]

Unfortunately we'll have to close this issue if no reproducible example is provided. Can anyone provide a way to reproduce this?

[rrdhani7]

I face same issue. Cypress is extremely slow when call cy.visit() because it waits all url is loaded.

[ahaurat]

Same here unfortunately. I _love_ the idea of Cypress but have been struggling with it for a couple of days now and am going to have to give up on it if I can't get the problem solved in short order. I almost wonder if a whitelistHosts might be possible/better? The sites I'm working on make ad calls etc. to _tons_ of hosts and enumerating them in blacklistHosts has proven to be impractical...

[ahaurat]

@jennifer-shehane With regard to a way to reproduce, as a generic example let's say a dev is working on an online media site such as bhg.com

A cy.visit() to any article page is so slow that it becomes really difficult. Example: https://bhg.com/news/all-the-current-food-recalls-you-need-to-know-about/

Another possible way to address the issue beside the whitelistHosts concept might be to provide a way to short circuit cy.visit() after a certain amount of time. In other words, don't wait for the page load event to fire. Just go to the url and _try_ interacting with the page after a certain number of seconds? e.g. cy.visit('/foo', {continueAfter: 5000})

Totally understand that might not be feasible for any number of reasons; just thinking out loud 😄

[jennifer-shehane]

Some of these comments are related to other slow running issues, but I'd like to focus this issue on the topic of Cypress waiting for the load event to fire during cy.visit(), since that was extensively addressed in https://github.com/cypress-io/cypress/issues/788#issuecomment-338747293 and also there is a reproducible example of that being an issue provided here https://github.com/cypress-io/cypress/issues/788#issuecomment-482173298

If your issue is still happening and not related to the time that the cy.visit() command appears in the Command Log to the time it resolves (when the pages load event fires) I suggest opening a new issue.

There has been some discussion of, instead of waiting for the load event to fire, to wait for network idle. This would not address the bhg.com issue however since the network is constantly fetching resources for 85 seconds until the load event fires.

The only way to get around this issue would be to perhaps listen to the DOMContentLoaded event, as @brian-mann suggested, or to have the ability to opt in and disable listening for the load event as on on cy.visit(), which would mean it is up to the user how they wish to proceed after calling cy.visit().

[guinnberg]

I'm recreating the same test in protractor and cypress in my app too and getting a similar result.

I think the reasons for this happening have been already explained, but basically, my app is composed of very old pieces and very new ones using Angular 7. Testing around angular 7 it's quite fast but when testing the old pages which can have all sort of bad practices about blocking the load event...

Hope we get the option to opt out soon

[KevinSimple]

OH God, i sold to my boss using Cypress for e2e testing, and i encountered this issue, i cant believe this is an issue reported over an yr and not been fixed or getting an work around, should i start pack up and looking for new interviews ?

[gcoombe]

There are cases where DomContentLoaded and Load events can be fired at significantly different times, one case can be when the page needs to load very large images. ~Also https://github.com/cypress-io/cypress/issues/8679 further causes issues for us having to wait on the load event. Due to this bug the load event often never fires as images are constantly loading.~

Correct me if I'm wrong but the main difference between DomContentLoaded + Load is stylesheets and image loading. From https://github.com/cypress-io/cypress/issues/788#issuecomment-338747293 @brian-mann commented that using load event offers higher guarantees, why is that? At DOMContentLoaded all scripts are parsed and loaded (https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event) and I don't see what cypress commands generally rely on image being loaded. IMO using DomContentLoaded as the event that cy.visit waits on makes more sense to me in scenarios where static assets like images etc slow things down would be great if that was an option.