Wpt: Refactor documentation into something more meaningful

[gsnedders]

At the moment, many of us find the documentation very hard to navigate and find relevant information. We should refactor the documentation and the paths through it so that it is more beneficial for both new contributors and existing ones alike.

Ignoring the README and index pages, we currently have:

• An introduction to GitHub, with instructions on how to clone, commit, and make your first PR.
• High level introduction to directory layout, test types, support files, and the lint tool.
• Tutorial and API reference for idlharness.js.
• Tutorial and API reference for testharness.js.
• Sort of explanation of reftests and some documentation.
• When/how to write manual tests.
• Some templates to use as basis for tests for varying types.
• CSS file name rules/guidelines for all tests.
• How to write good tests for user stylesheets.
• Metadata in CSS tests.
• Lint tool documentation, mostly listing the possible errors and how to fix.
• How to write a good test, with some description about when to use the different test types, along with some good techniques to use for tests.
• Some very brief documentation on how to run tests.
• The expected workflow for submitting tests.
• A duplicated copy of most of the GitHub introduction (from the same source file)
• Test review policy in almost no detail, with some introduction about our use of GitHub labels.
• Test review checklist covering many things not stated as required elsewhere.
• How to report a bug to a browser vendor.

That's vaguely in the order I found the pages. I wouldn't be surprised if I missed some, too. I think it's clear that we have lots of related information scattered all over the place, and having clearer paths through the documentation would be beneficial.

I think we have three rough categories of people looking at documentation:

• Browser developers wanting to make sure their tests can be upstreamed easily (i.e., what tests would pass review) or needing to check how to do specific things in a w-p-t compatible way (e.g., looking up testharness.js asserts, idlharness.js API, reftest-wait, reftest-print, fuzzy reftests, etc.).
• Web developers wanting to add a test or ten for some non-interoperable feature (they are far more likely to need a longer tutorial, as they're less likely to be looking APIs/etc. up for reference).
• People submitting tests because they care about some particular part of the platform (these likely have a cross of requirements, initially being walked through and then later reference). (This category includes people such as AT vendors, some of the orgs involved in things like EME, etc.)

We should make sure at a high-level each of those people can find what they need quickly to get started.

That said, I think it is notable that we don't expect people to be looking at documentation on how to run tests often, though some are likely to need their hands held through git and GitHub.

What I want opinions on: any of the above, what you (or people you know) frequently look for in the docs, how to organise the docs for both those who are new and those frequently looking up stuff.

cc/ @jgraham @Ms2ger @sideshowbarker @fantasai @frivoal

[gsnedders]

cc/ @RByers

[annevk]

These are the things I find I most commonly need (and indeed require some searching and a bunch of clicks each time):

• The command for updating submodules
• Documentation for JavaScript tests:
  
  
  
  • Boilerplate required
  
  
  • Basic setup for test/async_test/single-page
  
  
  • Available asserts
  
  
• Documentation for server-side tests

[annevk]

Another thing that does not help is that http://testthewebforward.org/ (not easy to remember since everyone always talks about WPT or web platform tests) is a marketing page but also hosts a bunch of project information. That should have links for writing tests (JavaScript, ref, manual, server-side) right on the front page as that's the main thing most visitors care about (I assume).

[tobie]

Another thing that does not help is that http://testthewebforward.org/ (not easy to remember since everyone always talks about WPT or web platform tests) is a marketing page but also hosts a bunch of project information.

Agreed. You'd want to keep the blog and events pages up for historical reasons, but there's absolutely no reason to keep them in the main navigation bar.

Most of the content of the home page needs to be replaced to be less markety and more wpt-specific.

[tantek]

Thanks for opening this issue @gsnedders. I think this is super critical. Commenting to follow and contribute when I have specifics.

[RByers]

Thanks @gsnedders, I'm glad you're doing this and thinking about it in terms of the difference audiences makes perfect sense to me!

From the perspective of the blink engineer the things I think they most need to know (separate from our own blink-specific engineering docs) are:

• What the different types of tests are and how to run them (especially across browsers)
• Style guidelines
• Review guidelines - in particular what sort of things to look for when reviewing WPT changes upstream that engineers wouldn't already be thinking about when reviewing a blink patch with blink-specific tests (since, like Mozilla, we intend to combine the code review into a single step).

/cc @foolip who probably has other ideas.

I've generally ignored testthewebforwards docs (referring mainly just to the README) as I thought they were mostly web-developer focused and somewhat out of date. But looking over your list, there's probably some stuff I should have read ;-).

[foolip]

FWIW, I have been to http://testthewebforward.org/ once or twice in search for documentation, but gave up. I think it'd be preferable merge as much of its useful documentation as possible into web-platform-tests, so that people don't need to go elsewhere.

In terms of a style guideline, there's some documentation being written in https://codereview.chromium.org/2492733003/ where I commented, so it captures many things that I think are valuable:

• A minimal test example (although I'm not sure about requiring <meta charset=utf-8>)
• Don't forget t.step_func
• What is a good description for tests and individual assertions. (IMHO, concise and with no reference to the pass condition.)
• Which features can be used. For example, right now depending on promises would be OK, but using for (const x of [1]) doesn't work in Firefox 49.

[gsnedders]

What @foolip mentioned is now here having landed in tree. There's also a Blink PSA on How to write reliable layout tests, which has some useful stuff in it.

Otherwise, there's a WebKit wiki page on Writing good test cases and a Mozilla wiki page on Avoiding intermittent test failures.

[gsnedders]

https://github.com/w3c/web-platform-tests/issues/3757 also contains some stuff about Edge testing policies which is likely relevant here.

[gsnedders]

So my basic suggestion is some rough hierarchal structure something along the lines of:

• Introduction
  
  
  
  • Basic overview of the directory structure of the testsuite
  
  
  • Owners, GitHub labels, other administrata (public-test-infra, etc.)
  
  
  • Setup of testsuite locally (mostly what's in the README.md)
  
  
• Writing tests
  
  
  
  • How to choose what type of test to write
  
  
  • Common test guidelines
  
  
  • HTTP headers
  
  
  • A page each for each test type
  
  
  • Server-side scripting
  
  
  • lint
  
  
  • Subtree specific guidelines:
    
    
    
    
    
    • userstyle tests (from the existing page)
    
    
    
    • CSS metadata
    
    
    
  
  
• Running tests
  
  
  
  • How to run a bunch of tests locally
  
  
  • Some rough pointers for people implementing automated runners to try and avoid that being some cabal
  
  
• Reviewing tests
  
  
  
  • This should obviously correlate with what we have in writing tests (i.e., we shouldn't have review requirements we don't document there!), but we probably want to present some of it differently
  
  
  • I believe some are fond of having some review checklist of things to check for?
  
  
• Appendices
  
  
  
  • Introduction to git
  
  
  • README.md
  
  
  • A page of test templates, given some people seem to like them
  
  

I don't envision all of these being separate pages; some of them will be sections, some will be pages. We should have pages for all the top-level things which are more than just a bunch of links, though.

[annevk]

I think someone earlier made a good point that all that information should either be put directly in or linked from the README. That way the repository becomes self-contained and addresses all questions.

[gsnedders]

Pretty much all the docs are already in the docs/ subdirectory of the repo; I think the plan is to move to publishing all the content using GitHub pages directly from the master docs/.

[frivoal]

One thing I'm not 100% sure: should all the git/github info go into Appendices/Introduction to git, or should some of it be spread around. For sure, the very basics of git should get their own section, but I'm more thinking about stuff like:

• When making changes based on a review, should I add more commits, or amend / rebase, and how is that done?
• When I want to locally checkout tests in a PR that hasn't been merged yet to play around with them, how do I pull in a branch from a separate repo? How do I get rid of it later on?
• When I want to make some minor edits to a PR while merging it, should I merge then fix, or fix then merge, and how do I do that?

This kind of question comes up relatively frequently, so we should answer them somewhere. I'm also OK with being tasked to write this up if this needs a volunteer. Just wondering where that goes, and thinking it should probably be spread throughout, possibly in <aside>s titled “git tips” or something like that.

[gsnedders]

@frivoal My basic thought was that actual introductory material for people who haven't used git goes into an appendix, information on how we want you to use git goes in the relevant places (most writing tests).

[gsnedders]

https://gsnedders.github.io/wpt-docs/ is around if anyone wants to have a look over it. I don't claim that it's done, but it is perhaps getting towards the point where we can consider replacing the live docs with it.

[tobie]

Looks great.

A couple suggestions if I may:

1. Don't make two links to the intro on the first page. Instead treat it like GitHub treats README on mobile: display the first paragraph at the top and the have a [continue reading overview] link.
2. Add a sitemap in the footer.

[frivoal]

@gsnedders Nice job. I've been through it, and submitted a bunch of issues and Pull Requests, but I agree this is already in a better shape than the live docs.

[annevk]

@gsnedders I think it's still an awful lot of clicks to get to actual documentation. One thing that could help is to flatten testharness.js. Link directly to the library, idl, and wptserve documentation. However, even then it seems that writing tests should be expanded directly on the front page. The other topics require far less frequent consultation and can be addressed by a sidebar or some such.

[gsnedders]

@tobie That's incidental from how I'd refactored the docs a few days prior. But yeah, obviously silly.

@frivoal Right, the big goal here is to be strictly better and get the right high-level structure so we can then iterate on that.

@annevk There's the difficult balancing act between information overload for people new and writing their first test and ease-of-reference who know what they're looking for. testharness.js is hard at the moment because of https://github.com/jekyll/jekyll/issues/5728 and the fact that its documentation largely lives in another repo, and I think it's obviously bad as a result here. I'm also deeply unhappy with the wptserve documentation, because it's largely written as if you're going to start using wptserve for another project, which just isn't the case here.

[gsnedders]

(I also have a whole load of fixes to make having gone through the whole thing, on paper, with a red pen, on transatlantic flight.)

[domenic]

The things I most often need to look up but cannot find in the current documentation are server features, especially:

• What different domains are available
• Substitutions in general
• What behaviors are triggered by different file extensions (e.g. .sub.html, .any.js, .worker.js)

[gsnedders]

There's now a menu in the top right with links to every page. Perhaps this is a bit too many, but there's frequently people wanting one specific thing. Hopefully the titles are good enough!

[gsnedders]

FWIW, I'm going to move towards trying to get this all merged and live soon—should then probably move everything else into specific issues to fix later.

[frivoal]

Can you add something like .wpt-site-nav { padding-right: 1rem; } to limit the interference between the top-right menu's content and overlay scrollbars?

[gsnedders]

That makes it look very asymmetrical. And making it sufficiently large on all sides makes it overflow the header. Can we have a media query for this already? :)

[frivoal]

No, but we have this: https://drafts.csswg.org/css-overflow-4/#scollbar-gutter-property
It is not yet implemented though.

[gsnedders]

4833 contains the refactoring and what's been done so far of rewriting stuff; I'll split out the remaining issues mentioned in here into new issues.

[foolip]

Yay! In https://codereview.chromium.org/2696323003/ I'm linking to some of these new docs.