Playwright: [REGRESSION]: Very slow on v1.1

[jinjor]

Context:

• GOOD Playwright Version: v1.0.2
• BAD Playwright Version: v1.1.0
• Operating System: Mac
• Extra: Both local (mac) and CI (ubuntu) environment, using chromium for tests

Code Snippet

Sorry, no snippet to reproduce this. I have a huge proprietary code and have not find the suspicious code in it. I will share it whenever I find it.

Describe the bug

I'm trying to upgrade playwright v0.11.1 to v1.1.1, but found a huge performance loss between v1.0.2 and v1.1.0. Our test becomes about 4x slower.

On local (mac):

• v0.11.1: npm run page-test 430.59s user 64.33s system 133% cpu 6:11.71 total
• v1.0.2: npm run page-test 396.36s user 53.51s system 106% cpu 7:00.52 total
• v1.1.0: npm run page-test 2600.17s user 224.96s system 199% cpu 23:38.11 total
• v1.1.1: npm run page-test 2621.26s user 223.12s system 200% cpu 23:40.09 total

On CircleCI (ubuntu):

• v0.11.1: 7m30s
• v1.0.2: 7m45s
• v1.1.1: 28m42s

It seems the new version uses CPU much more than before. Has something internally changed? Is there anyone who has run into similar situation?

For more context:

• Tests are written in mocha framework (not in parallel), they are written in one large file.
• Playwright is launched only once, context, page object is reused through all tests.
• Proxy is not used.

[pavelfeldman]

Thank you for the report. If we fail repoducing it locally, can you help us bisecting it? We will give you a range of Playwright packages and/or Chromium revisions and ask you to run your tests against this range to find the culprit.

[pavelfeldman]

The first would be ruling out the upstream (Chromium change). You would point 1.0.2 to Chromium that 1.1.1 uses. It should work and if it is slow, we would know the regression is in Chromium. If it is fast, we would know that the regression is in Playwright. Depending on the result, we would ask you to bisect the range of given Chromium revisions or a range of Playwright revisions.

[pavelfeldman]

@jinjor: could you join our slack and ping me in the #general channel?

[pavelfeldman]

@jinjor I'll start typing instructions here, but in case you show up on Slack we can continue there.

• in your 1.0.2 setup that works fine, edit file node_modules/playwright/browsers.json and replace chromium's revision 764964 with 775089:

    {
      "name": "chromium",
      "revision": "775089"
    }

• run node ./node_modules/playwright/install.js

This should install newer Chromium like this:

Downloading chromium v775089 - 125.5 Mb [====================] 100% 0.0s 
chromium v775089 downloaded to /Users/pfeldman/Library/Caches/ms-playwright/chromium-775089

Now you have 1.0.2 that runs against new Chromium. Run your tests and tell us if that is slow. If it is, we'll give you Chromium revisions to put there.

[jinjor]

Thank you for the instruction 😃

@jinjor: could you join our slack and ping me in the #general channel?

Okay, I'll join it when I get ready.

[pavelfeldman]

Also, even though you can't share a repro snippet, if you could tell us what your tests do, it might help. For example, if they take a lot of fullPage screenshots, etc.

[pavelfeldman]

Another question is whether you are using launch or launchPersistent

[jinjor]

Okay.

Also, even though you can't share a repro snippet, if you could tell us what your tests do, it might help. For example, if they take a lot of fullPage screenshots, etc.

• The test is like an E2E test, but the server is just a mock running in the local machine (so the cost of server side logic is very cheap).
• It visits a page, and then click buttons, input text, submit, etc. The app is a SPA, so there are a lot of JSON requests.
• "console", "dialog", "pageerror", "response" hook is used for logging and assertion.
• The first test case takes some screenshots (~100), but it does not take so long.

Another question is whether you are using launch or launchPersistent

launch

[jinjor]

• 1.0.2 - 775089 => fast
  
  
  
  • 410.55s user 56.31s system 105% cpu 7:23.87 total
  
  
• 1.1.0 - 764964 => slow (some tests are skipped because of timeout)
  
  
  
  • 1757.85s user 140.82s system 188% cpu 16:46.77 total
  
  

1.0.2 with the newer chromium(775089) was fast, so the slowdown should not be caused by chromium.

[pavelfeldman]

Awesome, thank you! Let us prepare a bisect script for you. You probably don't need to run all the tests all the time. With the 4x difference, you could find a test or two that are now much slower and only take tens of seconds to run. That way bisect process will take few minutes altogether.

[pavelfeldman]

@aslushnikov ^^

[jinjor]

I found a workaround. If Page object is created for each test case, the result gets much better.
Here are some results (all on v1.1.1).

Call context.newPage() and page.close() in every test case:

npm run page-test  536.37s user 84.96s system 105% cpu 9:50.80 total

Reuse Page object through all test cases:

npm run page-test  2591.17s user 223.55s system 200% cpu 23:23.48 total

[pavelfeldman]

Oh, you should create entire contexts, not even the pages for each test! @jinjor

[jinjor]

Ah nice, thanks. This way should be better for testing too!

So the half of the reason was the usage, but the problem still remains when a long-living page works (maybe something is leaking?). I keep this issue opened for now, but feel free to arrange how you like (change title, open another issue, etc.).

[aslushnikov]

@jinjor we discussed this offline, and one of the thing that might be provoking this is a some long-living waitForSelector command with timeout: 0. Do you have any of these?

[jinjor]

No, I did not find any.

[jinjor]

FYI: This is not related to v1.0.2 -> v1.1.1 change, but I found that firefox is also slow recently. This workaround does not work on firefox.

• v0.11.1 => npm run page-test 333.83s user 85.72s system 117% cpu 5:58.57 total (updated)
• v0.12.1 => npm run page-test 353.42s user 88.95s system 105% cpu 7:00.13 total (added)
• v0.13.0 => npm run page-test 358.94s user 95.63s system 112% cpu 6:45.62 total (added)
• v0.14.0 => npm run page-test 363.62s user 90.69s system 114% cpu 6:35.76 total (added)
• v0.15.0 => npm run page-test 359.83s user 92.25s system 117% cpu 6:25.49 total (added)
• v0.16.0 => npm run page-test 356.50s user 88.58s system 116% cpu 6:20.70 total (added)
• v0.17.0 => npm run page-test 360.42s user 91.21s system 116% cpu 6:28.32 total (updated)
• v0.18.0 => npm run page-test 365.57s user 92.08s system 119% cpu 6:23.63 total (added)
• v1.0.2 => npm run page-test 364.12s user 92.60s system 118% cpu 6:25.00 total (updated)
• v1.1.1 => npm run page-test 1742.20s user 413.58s system 166% cpu 21:36.60 total (updated)
• v1.2.0 => npm run page-test 1470.80s user 402.72s system 174% cpu 17:55.82 total (added)

Sorry, I haven't tested all the versions between v0.11.1 and v0.17.0 yet (because of breaking changes). I'll update this list when I have time. done

[aslushnikov]

@jinjor do I read this right and v0.11.1 was the last quick version for you?

[jinjor]

Yes, I’m on the way of upgrading from v0.11.1.

[aslushnikov]

@jinjor any updates on this?

[jinjor]

I just updated the list of Firefox's cases.

Maybe the previous result was incorrect. Now the regression point is between 1.0.2 and 1.1.1 which is the same as the case of Chromium.

[aslushnikov]

Hey @jinjor,

What's the status of this? Maybe there's another chance for me to see the regression locally? Since you mention that the regression happens with Chromium as well, it might be that we do something unfortunate on the node.js side.

[jinjor]

We are still using 1.0.2, have little time to dig into this issue. Sorry.
(I don't know if it is easy to make small test case or not, because it only occurs after a long run.)

[aslushnikov]

We are still using 1.0.2, have little time to dig into this issue. Sorry.
(I don't know if it is easy to make small test case or not, because it only occurs after a long run.)

@jinjor sure, understandable. Let me close this for now since it's not very actionable on our end without a repro.

If/when you have time to get back to this, I'll appreciate if you re-file with details.

[jinjor]

Okay, thank you for taking your time!