Wpt: Check what preferences need to be set to run tests reliably

[gsnedders]

See also #20058.

Per https://searchfox.org/mozilla-central/source/testing/profiles/profiles.json, Firefox applies a bunch of different pref files: base, common, unittest-required, unittest-features, and web-platform.

We should go through and check if there's anything else in there that should be documented and in infrastructure/assumptions with equivalents in all browsers.

[gsnedders]

@jgraham this certainly used to set layout.css.dpi but it doesn't seem to nowadays? (cf #5498)

[stephenmcgruer]

Tentatively marking roadmap since this seems to have an owner and activity happening. Feel free to downgrade to backlog if we aren't intending to work on it.

[gsnedders]

Related: #20057.

[guest271314]

Related https://github.com/web-platform-tests/wpt.fyi/issues/1648.

What is the WPT policy or practice relevant to preferences and tests?

If the feature is not enabled by default, rather requires being enabled, is the policy or practice simply to print the result as the object or function not being defined?

[jgraham]

For experimental runs we are using whatever set of flags the vendor wants. For stable and beta runs, it's only about correctness.

I agree that better understanding what's actually required here would be good.

[guest271314]

@jgraham The policy and practices should be clear.

Was not aware that the preferences for experimental runs are what the vendor wants, implying those are not, or at least should not be changed by WPT, unless otherwise clearly stated that is what is occurring.

Arrived at this issue by way of independent and WPT OffscreenCanvas tests for Firefox, which prints undefined at WPT tests, which is not the case when gfx.offscreen.enabled is set to true. Further, even when that preference is set an image transferred to a <canvas> using transferFromImageBitmap() will not be displayed at Nightly 72 (experimental, which uses WebRender) if gfx.webrender.force-disabled is not set to true (https://bugzilla.mozilla.org/show_bug.cgi?id=1598237#c15).

Given that OffscreenCanvas implementation at Firefox is WIP, there should be some codified and clear means of setting preferences to run tests for features that are not fully implemented to give feedback from the wild to implementers, and the preferences set for the browser used should be displayed at wpt . fyi, without having to navigate away from wpt . fyi to locate the preferences set by the vendor.

[guest271314]

@jgraham Re

Given that OffscreenCanvas implementation at Firefox is WIP

https://bugzilla.mozilla.org/show_bug.cgi?id=1390089

[gsnedders]

Things from the m-c prefs that I think we want equivalents of:

user_pref("dom.max_script_run_time", 0); // no slow script dialogs

// Ensure autoplay is enabled for all platforms.
// [gsnedders: though don't the tests in principle handle autoplay requiring user-initiated click now? AFAIK Chrome sets no such thing.]
user_pref("media.autoplay.default", 0); // 0=Allowed, 1=Blocked, 2=Prompt
user_pref("media.autoplay.enabled.user-gestures-needed", true);
user_pref("media.autoplay.ask-permission", false);
user_pref("media.autoplay.block-webaudio", false);
user_pref("media.allowed-to-play.enabled", true);
// Ensure media can always play without delay
user_pref("media.block-autoplay-until-in-foreground", false);

// Don't allow background tabs to be zombified, otherwise for tests that
// open additional tabs, the test harness tab itself might get unloaded.
user_pref("browser.tabs.disableBackgroundZombification", true);

// Disable antialiasing for the Ahem font.
user_pref("gfx.font_rendering.ahem_antialias_none", true);

I'm not sure there's much else?

[guest271314]

@gsnedders

I'm not sure there's much else?

At least

• gfx.offscreencanvas.enabled
• gfx.webrender.force-disabled

and

• javascript.options.writable_streams
• media.hardware-video-decoding.enabled

[stephenmcgruer]

In https://github.com/web-platform-tests/wpt/issues/23136, @koddsson discovered that the network.preload preference is being enabled for Firefox stable in the CI runs, whereas it is disabled for a 'normal' user of Firefox stable I believe.

Unfortunately at the same time I have to downgrade this to backlog, as @gsnedders is no longer working on it and it's not clear to me that anyone else is willing to own this :(

[stephenmcgruer]

@guest271314 - this issue is not about communicating what flags are being used, it is about determining which flags should be used. Please keep any comments on showing which flags are being used to https://github.com/web-platform-tests/wpt.fyi/issues/1648, thanks.

[guest271314]

Posted in incorrect issue. Two side of the same coin.

[domenic]

Maybe a good next step is to post a Mozilla Bugzilla bug asking them to turn off the preload preference when running WPT? Or, is that something we can control in this repository?

[koddsson]

Maybe a good next step is to post a Mozilla Bugzilla bug asking them to turn off the preload preference when running WPT? Or, is that something we can control in this repository?

I have no idea if I did it correctly but I took a stab at exactly that here.

[guest271314]

Or, is that something we can control in this repository?

Setting and verifying preferences for tests set should be under control of WPT. Is there any reason why this cannot be done? Here, perform the tests by hand. WPT downloads an entire browser, therefore the browser running the test is under complete control of WPT.

[koddsson]

My patch got rejected since Firefox is going to ship the preload functionality soon. I feel a bit weird about it since WPT will then continue to show the tests passing on Firefox even though preload doesn't work at all on Firefox stable with the default settings, and therefore isn't a correct representation of the browsers capabilities.

[domenic]

To me this demonstrates that there are conflicting priorities between wpt.fyi and browsers' internal WPT runs, which makes letting browsers configure the options they use for wpt.fyi a bad choice. I.e., it's understandable that Firefox wants more coverage of features, but doing so shouldn't cause wpt.fyi to become inaccurate.

So I'd suggest the wpt.fyi infra folks figure out a way to separate Firefox's config that they use for running tests internally, from the one that is used on wpt.fyi.

[jgraham]

There is a separation. In this case a pref got set in the global config rather than in one of the configs that's only applied to nightly or in gecko CI. It's difficult to ensure that never happens.

Allowing browsers to set prefs is necessary; there are a bunch of prefs that have to be set for gecko to reliably run tests. We also end up setting some prefs for Chrome.

[stephenmcgruer]

Allowing browsers to set prefs is necessary; there are a bunch of prefs that have to be set for gecko to reliably run tests.

Agreed, but as far as I know Firefox is the only browser where we download a general set of preferences from a server, rather than explicitly setting flags with quite specific purposes. It's true that Chrome could change the meaning of one of the flags we set to do something different and we end up with bad wpt.fyi results, but it seems less likely than when we're downloading a large set of profiles.

In https://github.com/web-platform-tests/wpt/issues/20284#issuecomment-560057039 gsnedders suggested (I think, excuse me if I'm misinterpreting accidentally) pulling out a set of prefs to set. That would be more akin to the flag-setting we do for Chrome imo.

For experimental releases, incidentally, I think it's reasonable to be more liberal; we set --enable-experimental-web-platform-features for Chrome and that's just a massive grab-bag of whatever people like to set 😆 . It definitely doesn't mean 'this feature is coming', albeit we do try and launch most things...

All this said, I think the larger concern to me is how undiscoverable this is, for all the browsers. I had to dig in source code to find we set (say) --short-reporting-delay for Chrome, and I have no idea (aside from reading the OP here) how to find out what flags Firefox runs with. I would suggest a two-pronged approach of:

1. Documenting how flags/prefs are set for each browser, with some code pointers, and
2. Plumbing some sort of 'configuration info' through to wpt.fyi, if we can find a useful way to display it. I'm concerned this will be too much info however (potentially dozens of prefs), so we'd have to think how to make it reasonably consumable.