Manifest: Create test suite to check Web App Manifest feature support

[tomayac]

The core issue is that most features come to play only after the “add to home screen” step, i.e., require manual interaction with a browser first, and then manual review with the operating system (e.g., verify which icon was used on the home screen) and other apps (e.g., verify if the scope is respected). A first stab at the most basic level (initially focused on iOS but useful on other platforms as well) is https://ios-a2hs-demo.glitch.me/, but many questions remain.

Checklist

Tested:

• [x] display
• [x] icons
• [x] name
• [x] orientation
• [x] short_name
• [x] theme_color
• [x] shortcuts (https://github.com/web-platform-tests/wpt/pull/24452)
• [x] shortcuts / icons CORS & CSP (https://github.com/web-platform-tests/wpt/pull/27620)
• [x] icons / CORS & CSP (https://github.com/web-platform-tests/wpt/pull/27620)
• [ ] scope
• [x] start_url
• [ ] dir
• [ ] lang

HTML: https://github.com/web-platform-tests/wpt/issues/24389

Not testing for now:

• description (moved to Manifest App Info, #900)
• categories (moved to Manifest App Info, #900)
• screenshots (moved to Manifest App Info, #900)
• iarc_rating_id (moved to Manifest App Info, #900)
• related_applications (currently single-engine, #950)
• prefer_related_applications (currently single-engine, #950)
• icons/monochrome purpose (currently single-engine, #905)

Results sheet:
https://docs.google.com/spreadsheets/d/1_Ts5r4WksveBivvQ_KjdBrSDHDmNpNWqiER019A1onM/edit?usp=sharing

[tomayac]

Also see this Twitter thread for further background.

[marcoscaceres]

All interested folks: should we have a call about how we should do the testing? I have ideas, but want to make sure we are all on the same page.

@honry, you'd made a start on some of this already (on the API side, at least). However, we want to start just by testing and validating the manifest properties as manual tests.

[Honry]

@marcoscaceres, I'm really interested in your ideas.

Test automation is an ultimate goal, and we still have a long way to go. I agree to start creating manual tests to fill the gap of no tests for Web App Manifest.

[marcoscaceres]

@Honry agree... but we will have to do manual tests for now :( At least, we can evaluate where we are at with regards to conformance in each use agent.

@Honry, what time zone are you in?

[marcoscaceres]

fixed typo above.

[Honry]

@marcoscaceres, UTC+08:00

[marcoscaceres]

Ok, cool. I'm UTC+10. Let's wait a day to see if anyone wants to help with testing. If not, you and I can meet and discuss a plan of action. Should be pretty straight forward tho.

[tomayac]

I’d be happy to help, but am attending a conference (The Web Conf, Lyon, France) all week. If you plan on ultimately having automated tests, I’m not that sure it’s “pretty straight forward”, but manual tests would get the job done for now. Albeit would be a multistep test, when you focus on "scope" etc.

[tomayac]

Maybe for now just listing test approaches for the various manifest properties would be enough. Happy to get this going, but quite limited timewise this week.

[marcoscaceres]

@tomayac, maybe you could help us with review? I can make add you to the OWNERs over on WPT. That way, we can distribute the workload a little bit more.

I'd like to look at a bit more at the research @firtman has done in this area. I recall he found some pretty significant interop issues, which we need to make sure we expose.

[tomayac]

SGTM!

[mgiuca]

I'm interested. Of course, this fits into the bucket where it's very hard to even define manual tests, because of the user interface being so user-agent-defined. I'm also travelling this week (UTC-7). Next week is better for VC but I'll also be free today from 9am to 11am Melbourne time.

[marcoscaceres]

@mgiuca, ok, let's wait till next week as we are in the same timezone. That also makes it easy to sync with @Honry.

[Honry]

@marcoscaceres, seems it's a bit difficult to find an appropriate time slot to involve all interested folks in a call. How about just paste you ideas here, then we can discuss and move this forward?

[Honry]

@marcoscaceres, soft ping. :)

[patrickkettner]

👋 gonna start writing some of these tests in the near future, hopefully @Honry could check it out if I get it in soon?

[marcoscaceres]

@Honry, if you could help review or create tests with @patrickkettner it would be awesome. The plan is basically just to make little tests for each manifest property:

So, a HTML file:

<!doctype html>
<link rel="manifest" href="manifest-short_name.json"> 

And then manifest-short_name.json:

{
  "short_name": "PASS"
}

And "add to homescreen"... and having the short_name show up would be considered a pass.

@patrickkettner is going to provide us with a link to where we can put the results... might just be an excel sheet, which we can then generate the tests results from as JSON.

There are some things we can't really test, like the ARC rating stuff... but we might point to the Microsoft Store as evidence that it's being used.

[Honry]

@patrickkettner, looking forward to your tests, and I'd be glad to help review and create more tests.

[aarongustafson]

@patrickkettner I started writing tests if you’re still willing/able to help: https://github.com/web-platform-tests/wpt/pull/23816

[marcoscaceres]

We have a spreadsheet with tests and results so far:
https://docs.google.com/spreadsheets/d/1_Ts5r4WksveBivvQ_KjdBrSDHDmNpNWqiER019A1onM/edit?usp=sharing

[marcoscaceres]

Manual tests PR https://github.com/web-platform-tests/wpt/pull/24146

[aarongustafson]

@annevk has requested additional tests https://github.com/whatwg/html/pull/5581#pullrequestreview-439224722

[marcoscaceres]

Test plan for whatwg/html#5581 is at https://github.com/web-platform-tests/wpt/issues/24389

[marcoscaceres]

@christianliebel I deleted your checklist and moved it to the very top. Thanks for creating it!

[aarongustafson]

@christianliebel It’s been a while since we last touched this. Can we set up a sync and sort out what & how we want to test the remaining items? I am carving out a few hours a week to try to get back on top of a bunch of this stuff.

[christianliebel]

@aarongustafson Sure, let's follow up on Twitter.

[aarongustafson]

From our call today:

• @christianliebel to look at CORS & CSP for icons & shortcut icons.
• scope is ill-defined & hard to test. Could we test to see if DevTools correctly reads the value and call that good?
• @thejohnjansen & I are taking an action item for myself on how to test dir and lang. If we don’t have a good direction from that, we could we test to see if DevTools, again, correctly reads the value.

I will open the App Info-related WPT items on that spec, move the related_applications-related items to their own bug (not tied to CR) and add a callout on the monochrome purpose thread that a WPT will be required if that gets more implementations.