Cypress: Tests Run Slowly

[OmriAharon]

Current behavior:

Following up on #2081, now that Canary shows the stubbing as 3-8ms, our tests still run just as slow.
Screenshots below are from the test repo I created, in our actual project each stub can take up to 2-3s.

You can see that in the Network panel, the stub is reported as fast: (3ms)

But if I click on the log entry in the runner, I see this print in the console: (notice the 79 duration)

In performance tab I see something a little different: (61 ms)

Desired behavior:

Tests run much faster.

Steps to reproduce:

I have this repo https://github.com/OmriAharon/CypressSlowStub, you can follow the readme.

Versions

Cypress 3.0.2
Google Chrome Canary 70/Chrome 66
Mac OS High Sierra

[brian-mann]

It's to be expected that these durations are all different - because they are all measuring different things.

The least accurate is going to be the snapshotted one, because that's the duration using Date.now() which is inaccurate at the millisecond level. That's also accounting for more callbacks than say the network or the performance indicator.

I don't really believe that your slow 2-3s stubs are indicative of what this example repo is - those are likely due to another cause, possibly related to running multiple tests together in cypress open (interactive mode) which is known to cause those kinds of slowdowns. What's happening has been documented on here in various places.

The solution is not running multiple tests at the same time and switching to use cypress run for long runs of multiple tests. We disable all of the debugging features in cypress run, whereas they are enabled in cypress open and will absolutely reduce performance.

Here's an issue for reference which explains this: https://github.com/cypress-io/cypress/issues/1586

Also documented here: https://github.com/cypress-io/cypress/issues/681

[OmriAharon]

Thanks @brian-mann. Unfortunately, this is not referring to cypress open. The cypress run is also slow, otherwise I wouldn't have bothered (that much). We have about 110 tests in 17 files taking an overall of about 10-12~ minutes, both locally & in CI using cypress run command. Many "it" tests take 20-30 seconds each. Granted, we have several asserts there, but I don't think _that_ many that should account for these times.
I don't think most users encounter these log times for the same number of tests. Or maybe that's fine and I'm not aware?

I know the test repo is probably not indicative, but that's the best I could try to reproduce without pushing our entire repo, which I can't do.

P.S: I showed @bahmutov about a month ago what's happening, he agreed something was off. Would be happy to take this "offline" if you care to continue there, there's already an email thread with @RandallKent.

[brian-mann]

Correct, this is not something that is normal. We'll need to take it offline to dig into it.

As a sidenote have you tried using parallelization (released in 3.1.0)? Your 17 files would get chunked up and distributed across N number of machines and it'll likely go from 10-12 minutes to 3-4 (or less).

[OmriAharon]

We just utilized it last night, indeed went down to 4.5 minutes. I must say, a feature awesomely done!

It's 4.5 minutes since we have one test that takes 4.5 minutes so that's our minimum. Would have been much faster otherwise.

[kflavin]

@OmriAharon I'm curious if you were you able to resolve your issue? I may be experiencing something similar.

[yumao5]

Any suggestion about it yet? thanks

[jennifer-shehane]

We made several performance improvements as part of 3.3.2 release. Please update to the latest version of Cypress - if there are still performance issues after updating, open a new issue including a reproducible example.

[nym]

Would like to see some kind of guide that talks about how to get test times down.

[nzn-idt]

@nym check out my suggestion here: https://github.com/cypress-io/cypress/issues/3829#issuecomment-626785960