Playwright: [Feature] Run Playwright tests on real mobile device browsers?

[Cosmin-Gramada]

Is there any way to connect to a device, start a browser (e.g. chromium instance on Android) and run the tests?

If not, what is missing at this point to be able to use it on both Android and iOS?

[pavelfeldman]

There is no way to attach to Chromium on Android or MobileSafari on iOS. Playwright aims at running on your desktop and in your CI/CD, targeting device farms is outside its scope.

We will emulate device metrics, geolocation, orientation, etc on Chromium on Android and MobileSafari though. Could you share more about your use case?

[Cosmin-Gramada]

We took into account the possibility of moving away from Selenium WebDriver based tools and switching to DevTools API based tools.

A really big portion of our tests are running on mobile web browsers, where we tests our own player that makes use of browser related technologies (e.g. what types of medias is able to render, what's the appropriate format, etc) and sends specific tracking events.

Our concern is that there will be differences between a real device and an emulated one, as we're able to emulate any device resolution, but only the viewport size is guaranteed to be accurate.

[pavelfeldman]

Our concern is that there will be differences between a real device and an emulated one

Yes, there will be

as we're able to emulate any device resolution, but only the viewport size is guaranteed to be accurate

More than the viewport size, but yes, if you need mobile-specific media to be accurate, you need to attach to the device.

This is technically feasible, but is outside of the scope for v1.

[tanaypratap]

This could be useful for us too. We are a fully fledged PWA email app. Testing on actual devices would catch bugs. We have also faced this issue with non-existent media queries on some phones which @Cosmin-Gramada mentioned. Especially older phones.

[Filipoliko]

Since Playwright is trying to be more testing-friendly, it would be really great to have real mobile device support. More and more users are coming to our websites from mobile devices, it is nearly 50:50 for us. Most of our bugs on mobile devices are not discoverable via simple mobile emulation. It is really important for us to be able to test on real mobile devices, or at least on emulated Android/iOS systems.

[arjunattam]

@Filipoliko - thank you for sharing. Can you elaborate on which bugs are not discoverable via emulation? Examples will really help!

[Filipoliko]

@arjun27 - They are usually some very specific (and kinda weird) use cases, typically on iOS devices. One current bug on our website, that is discoverable only on real iOS device (safari, or chrome) can be currently found on this page. (The fix will be probably released during next week, hopefully you can check it before that time).

If you click on Domácí (see the picture below), then with chrome mobile emulation, you will be redirected to a section page Domácí (correct behaviour). But if you are running chrome, or safari on real iOS device, then the click will open the mobile menu. This is fixed by setting pointer-events: none on the hidden mobile menu element. Android/Windows/macOS works fine, but iOS requires this css rule for some reason.

Probably most typical use cases (as far as I can think of) are related to some invisible elements overlapping other elements and making it impossible to click on the overlapped element. This behaviour is usually iOS specific, other systems interpret these css rules in a bit different way.

[arjunattam]

Thanks @Filipoliko. I just tried running this case with iPhone emulation on Playwright WebKit and I'm able to repro this issue. Chrome mobile emulation is not reproducing it, as you've rightly pointed it out. Also, you're right about an invisible overlapping element (which is why I had to use force: true to issue the click). Sharing code below in case you'd like to try it out.

const { webkit, devices } = require('playwright');

(async() => {
  const browser = await webkit.launch({headless: false});
  const context = await browser.newContext({...devices['iPhone 11 Pro']});
  const page = await context.newPage();
  await page.goto('https://www.seznamzpravy.cz/clanek/v-korupcni-kauze-stoka-obvinila-policie-dalsi-27-lidi-111820');
  await page.click('[data-dot="ogm-breadcrumb-navigation"] >> text=Domácí', {force: true});
})();

[Kradirhamik]

Hi,

I would like to also give support for this idea. Our use cases requires our Angular site to get input from other apps (e.g. BankID for authentication). So the real flow can only be properly tested on a real device.

[alejandroiglesias]

Hi,
I'm curious to know what the workflow is on this project for an issue like this to move from feature request to development by the team? Does it need to reach a certain amount of upvotes? Does it need any funding? Does it take a certain amount of time to pass, like waiting for the next major release planning? It would be good for users to know and be able to set expectations on the development of feature requests. Thanks!

[pavelfeldman]

We have already seen enough interest in it, so we will invest into this area. There are several questions about the use cases and the API, so we would appreciate your feedback, the more details the better. I'll post some initial questions below as comments, please upvote them in case they apply!

[pavelfeldman]

Please 👍 if you'd like to

Use Playwright with the Chrome Browser (WebView integration is not essential for you).

[pavelfeldman]

Please 👍 if you'd like to

Use Playwright against your WebView and can bring WebView into a testable state over adb.

[pavelfeldman]

Please 👍 if you'd like to

Use Playwright against AVD (Emulator) as opposed to the real device.

[pavelfeldman]

Please 👍 if you'd like to

Use Playwright against the physical device as opposed to AVD (Emulator).

[pavelfeldman]

Please 👍 if

Your testing setup requires Android w/ Play Store

[pavelfeldman]

Please 👍 if

Your testing setup does not require Android w/ Play Store

[pavelfeldman]

Please 👍 if

You intend to run tests in the cloud.

[pavelfeldman]

See https://github.com/microsoft/playwright/blob/master/docs/mobile.md for experimental support.