Cypress: `cy.visit` in 3.3.1 is 2x-4x as slow as 3.2.0

[fr0]

Current behavior:

Running my tests in 3.2.0 takes 1 hour 54 minutes.
Running my tests in 3.3.1 takes 4 hours 41 minutes.

(This is running on headless electron using cypress.run with the module API; I haven't measured it with cypress open).

Both cases are running on the same WIndows 10 VM, and not using parallelization.

Desired behavior:

I'd like my tests to take less than 2 hours.

Versions

Cypress 3.3.1

[fr0]

I tried downgrading from 3.3.1 to 3.2.0 and my tests once again ran faster (under 2 hours).

[flotwig]

Hey @fr0, are you using the built-in Electron browser? Do you have a proxy set up?

[fr0]

Thanks for the quick response. I am using the built-in Electron browser, but I don't have a proxy set up.

[flotwig]

This could be a related issue to #4298 - does your app rely on any kind of non-200 status code, any redirects or anything like that?

[fr0]

It doesn't, but that's a good thought. It does make heavy use of WebSockets. Also, my tests rely on some setup pre-work that calls that WebSocket API from the beforeEach handler (and I use cy.wrap + cypress pipe for this).

I'm not sure if any of that is related, though.

Is there any additional logging I can turn on that would help to debug this issue? I'd love to send you guys the code so you can reproduce it, but unfortunately it's all proprietary (including the back-end).

[fr0]

I guess technically the WebSocket upgrade is a 101, so the answer to your question may actually be "yes".

[flotwig]

You can get full debug logs by following these steps and share it to shed some light on the issue: https://docs.cypress.io/guides/guides/debugging.html#Print-DEBUG-logs

That will help to debug the issue. Make sure to scrub any sensitive data before sharing them.

[fr0]

This is just with a single spec file, but it's more than enough to show off the difference (59 seconds in 3.2 vs 114sec in 3.3.1).

debug-cypress-3.2.0.log
debug-cypress-3.3.1.log

[exan-atsui]

We've updated to 3.3.1 from 3.1.5 and we're also seeing this behavior. Things are much slower when running our tests in run.

[brian-mann]

We're triangulating a few different issues in the 3.3.x releases and several may all be related to the same underlying problem. Likely will have a patch in the next few days and then we can go from there.

[nthum]

We are seeing same behavior when upgraded from 3.2.0

[ChuckOp]

This is a major problem for us.

[flotwig]

@nthum @ChuckOp @exan-atsui Hey, thanks for letting us know. Can you provide some additional details to help us debug? What operating system are you using? Are you on Cypress 3.3.1 or 3.3.0? Are you behind a proxy? What browser are you using?

[nthum]

@flotwig We are running 3.3.1 on Windows 10 & using headless electron. We don't have proxy

[fr0]

I am fairly certain that this is not OS-specific, as I've seen it both on Windows 10 and Mac OS.

[flotwig]

I wrote a small test case to benchmark cy.visit:

describe '', ->
  it 'benchmark cy.visit', ->
    Cypress._.times 100, ->
      cy.visit('/index2.html')

    cy.wrap ->

In Cypress 3.2.0, this executes in 17.25 seconds.

In Cypress 3.3.1, this executes in 61.58 (!) seconds.

So it does look like the issue is with cy.visit(). I recorded a short video of the benchmark which shows that every few cy.visit()s, it seems to hang for a couple of seconds before continuing: https://drive.google.com/file/d/15a26L9dC9EQHFJKd-hQfvxkxMnFgmPmN/view?usp=sharing

Looks like #4349 is a duplicate of this issue, then. I'll close that issue and log future progress on tracking down the cy.visit slowness here.

[bahmutov]

good idea to create a performance test like this, and the video shows a
weird "fast - slow" visit pattern, like you have noticed. Some time ago I
did an experiment where I timed the load of a page multiple times to see if
Cypress can give us performance metrics - the very first load was really
slow, but then it would quickly drop, but I have not noticed increased
times. I wonder if there is some browser cache getting full then cleared,
causing new full fetch. Or could it be connected to garbage collection
pause? We can also record and dump timings from Electron vs Chrome to see
if the same behavior is noticeable

Gleb

On Tue, Jun 4, 2019 at 2:40 PM Zach Bloomquist [e-mail]
wrote:

I wrote a small test case to benchmark cy.visit:

describe '', ->
it 'benchmark cy.visit', ->
Cypress._.times 100, ->
cy.visit('/index2.html')

cy.wrap ->

In Cypress 3.2.0, this executes in 17.25 seconds.

In Cypress 3.3.1, this executes in 61.58 (!) seconds.

So it does look like the issue is with cy.visit(). I recorded a short
video of the benchmark which shows that every few cy.visit()s, it seems
to hang for a couple of seconds before continuing:
https://drive.google.com/file/d/15a26L9dC9EQHFJKd-hQfvxkxMnFgmPmN/view?usp=sharing

Looks like #4349 https://github.com/cypress-io/cypress/issues/4349 is a
duplicate of this issue, then. I'll close that issue and log future
progress on tracking down the cy.visit slowness here.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/cypress-io/cypress/issues/4313?email_source=notifications&email_token=AAQ4BJQR2CKXYL735U4FGEDPY2ZKDA5CNFSM4HPRVP32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODW5PUOQ#issuecomment-498793018,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAQ4BJU6DS7J5B3A7M4XT2DPY2ZKDANCNFSM4HPRVP3Q
.

--
Dr. Gleb Bahmutov, PhD

Schedule video chat / phone call / meeting with me via
https://calendly.com/bahmutov
gleb.[email protected] @bahmutov https://twitter.com/@bahmutov
https://glebbahmutov.com/ https://glebbahmutov.com/blog
https://github.com/bahmutov

[danceric0919]

Hi
is there a time line to fix this issue?
we are suffering from this for couple days.

[flotwig]

@danceric0919 We have a fix in place, it will be out with the next patch release (3.3.2).

[cypress-bot[bot]]

The code for this is done in cypress-io/cypress#4385, but has yet to be released.
We'll update this issue and reference the changelog when it's released.

[fr0]

Would it be possible to create a beta release in npm so I can verify that the fix addresses my issue?

[flotwig]

@fr0 You can use the latest beta build for Windows:

In Git bash:

CYPRESS_INSTALL_BINARY=https://cdn.cypress.io/beta/binary/3.3.2/win32-x64/appveyor-develop-e81fbae9e2345d2e4d23f224309dd35f58d3ff59-25353717/cypress.zip npm i https://cdn.cypress.io/beta/npm/3.3.2/appveyor-develop-e81fbae9e2345d2e4d23f224309dd35f58d3ff59-25353717/cypress.tgz

In other shells, you'll need cross-env to set the environment variable:

npm i -g cross-env
cross-env CYPRESS_INSTALL_BINARY=https://cdn.cypress.io/beta/binary/3.3.2/win32-x64/appveyor-develop-e81fbae9e2345d2e4d23f224309dd35f58d3ff59-25353717/cypress.zip npm i https://cdn.cypress.io/beta/npm/3.3.2/appveyor-develop-e81fbae9e2345d2e4d23f224309dd35f58d3ff59-25353717/cypress.tgz

[fr0]

I tried that, and at first it looked promising:

⚠ Warning: Forcing a binary version different than the default.

  The CLI expected to install version: 3.3.2

  Instead we will install version: https://cdn.cypress.io/beta/binary/3.3.2/win32-x64/appveyor-develop-e81fbae9e2345d2e4d23f224309dd35f58d3ff59-25353717/cypress.zip

  These versions may not work properly together.

Installing Cypress (version: https://cdn.cypress.io/beta/binary/3.3.2/win32-x64/appveyor-develop-e81fbae9e2345d2e4d23f224309dd35f58d3ff59-25353717/cypress.zip)

 ✔  Downloaded Cypress
 ✔  Unzipped Cypress
 ✔  Finished Installation /Users/crfrolik/Library/Caches/Cypress/3.3.2

You can now open Cypress by running: node_modules/.bin/cypress open

But then when I actually tried to run...

yarn run v1.16.0
$ /Users/crfrolik/Documents/atlas-dev-tools-proto/node_modules/.bin/cypress open
No version of Cypress is installed in: /Users/crfrolik/Library/Caches/Cypress/3.3.2/Cypress.app

Please reinstall Cypress by running: cypress install

----------

Cypress executable not found at: /Users/crfrolik/Library/Caches/Cypress/3.3.2/Cypress.app/Contents/MacOS/Cypress

----------

Platform: darwin (17.7.0)
Cypress Version: 3.3.2
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[flotwig]

Oh! I thought you were running on Windows, sorry. Here's the correct command for macOS:

CYPRESS_INSTALL_BINARY=https://cdn.cypress.io/beta/binary/3.3.2/darwin-x64/circle-develop-e81fbae9e2345d2e4d23f224309dd35f58d3ff59-123895/cypress.zip npm i https://cdn.cypress.io/beta/npm/3.3.2/circle-develop-e81fbae9e2345d2e4d23f224309dd35f58d3ff59-123894/cypress.tgz

You might have to delete /Users/crfrolik/Library/Caches/Cypress/3.3.2 to get rid of the Windows package.

I know it's kinda a mess, we're working on making this more accessible to end-users.

[fr0]

npm lets you tag a release as "pre-release", so users won't get it unless they specifically opt in. It makes this sort of thing very easy. :)

[bahmutov]

yeah @fr0 , we just don't want to pollute NPM with 10k builds of Cypress per month (like TypeScript), and that's why we take advantage of static folders to install pre-release builds. But we know some tricks to make it simpler, just did not get to implement them yet

[fr0]

You definitely don't want to publish every single CI build as a pre-release to npm, true.

Back on-topic: At first glance, the nightly build looks like it did not fix my issue. I'll try again on a different environment to be sure.

[brian-mann]

@fr0 i think the PR that's going to fix your issue hasn't been merged into develop yet. This is likely the one https://github.com/cypress-io/cypress/pull/4358/files

When you see that get merged in then you can see if it fixes your problem.

[fr0]

@fr0 i think the PR that's going to fix your issue hasn't been merged into develop yet. This is likely the one https://github.com/cypress-io/cypress/pull/4358/files

When you see that get merged in then you can see if it fixes your problem.

Thanks, I'll take a look. Is there a way to get the cdn beta link for a given develop commit?

[flotwig]

Right now you have to pull it out of CI:

1. Go to the "Mac build" CI stage:
   
2. Click on "mac" to get to the workflow:
   
3. The package CDN url is in Mac NPM package, the binary URL is in Mac binary:
   
4. In Mac NPM package, the package URL is in the "upload NPM package" step:
   
5. In Mac binary, the binary URL is in the "upload unique binary" step:
   
6. Then it's just CYPRESS_INSTALL_BINARY=<binary .zip URL> npm install <package .tgz URL>

The process is similar for Linux (also on Circle CI) and for Windows (except you need to go to Appveyor for the Windows builds)

[fr0]

I tried the latest build (from c7b340) and the issue is not resolved.

EDIT: Looks like I did get back some performance, but it's definitely still slower than 3.2.0.

[flotwig]

@fr0 Could you share spec code that reproduces the slowness that changed between 3.2.0 and 3.3.2? We had some test cases where we improved test performance (like the 100x cy.visit test I posted earlier in this thread, which now runs as fast as it did in 3.2.0), but it's possible that we missed an edge case that you're still running in to.

[fr0]

Sure, I'll work on trying to get a reproducible example.

[fr0]

I was able to reproduce the problem with the most trivial example I can think of that uses Angular + Cypress.

https://github.com/fr0/cypress-slowdown

Here's the spec file. https://github.com/fr0/cypress-slowdown/blob/master/cypress/integration/sample.spec.ts

(It repeats 50 times just to get a bigger sample set.)

Note that the problem isn't in cy.visit. In testing with my app before I created this trivial example, I found that pretty much every cypress command was slower in 3.3.2 than in 3.2.0.

(Note: I created a cy.task that measures timing using process.hrtime(). If this is misguided, or if there is a better way, I'd love to be clued in.)

Thanks!

[flotwig]

@fr0 Thank you for the example repo, I see exactly what you mean now. After some debugging, I think you're having the same issue as #4411, check out this comment to see what I mean. If so we can move future discussion of this non-cy.visit() slowdown over there.

[fr0]

Excellent, thanks for the update. Glad you were able to reproduce it with my repo.

FYI, I (any many others) are not able to roll back to 3.2.0 because it isn't compatible with Angular 8 (due to https://github.com/cypress-io/cypress/issues/2069 + https://github.com/TypeStrong/ts-loader/issues/929), so that's not an option right now.

I'll continue discussion in the issue you linked. Thanks again!

[cypress-bot[bot]]

Released in 3.3.2.