Wpt: Cause of Taskcluster failures are too hard to figure out

[foolip]

In https://github.com/web-platform-tests/wpt/pull/15401 and many other issues, people have asked about why Taskcluster is failing. It is currently too much work to understand failures.

For ease of reference, here's how I figure it out:

• follow the "Details" link to a page, e.g. https://tools.taskcluster.net/groups/DrNoOZW5TQmw4pK53xNBIg
• click on a failing task, e.g. https://tools.taskcluster.net/groups/DrNoOZW5TQmw4pK53xNBIg/tasks/Xv8vrSDLQ5WH4XI8kvF0jg/details
• click "Run Logs" and live.log, e.g. https://tools.taskcluster.net/groups/DrNoOZW5TQmw4pK53xNBIg/tasks/Xv8vrSDLQ5WH4XI8kvF0jg/runs/0/logs/public%2Flogs%2Flive.log
• click "View raw log", e.g. https://taskcluster-artifacts.net/Xv8vrSDLQ5WH4XI8kvF0jg/0/public/logs/live_backing.log
• scroll to the bottom of the logs
• if the logs are too long (https://github.com/web-platform-tests/wpt/issues/15171) download them and run tail -n1000 live_backing.log
• the problem will usually be apparent at the end. if it's a markdown table from ./wpt run --verify, copy it into a comment and preview or paste to make sense of it

Some known failure causes are listed in https://github.com/web-platform-tests/wpt/issues/14210.

Understanding a Taskcluster failure is too much work.

@jgraham will the Checks API integration allow us to improve this at some point? Ideally we'd be able to populate the details of the checks page with an artifact containing markdown.

[jgraham]

@owlishDeveloper might be better able to answer this, but I'd certainly like some way to produce a markdown artifact that ends up being the content of the checks API.

We could also consider adding the wpt repo to treeherder, which has features for extracting errors from a log, and would also allow us to use the classification facilities to track different causes of failure. But we'd still need a way to link to the push on treeherder.

[foolip]

Ping @owlishDeveloper, is a way to supply arbitrary markdown for the Taskcluster integration using GitHub Checks planned?

[stephenmcgruer]

Nobody currently owns driving this, so dropping to backlog. Note that we do have a Q2 OKR to try to migrate to the Taskcluster Checks API, but that alone will only reduce this flow by at most one click. We need the ability mentioned in https://github.com/web-platform-tests/wpt/issues/15412#issuecomment-538882616 to push text to the GitHub Checks description.

[stephenmcgruer]

The WPT Taskcluster Checks migration is (mostly) complete, so this should now be possible to address. https://docs.taskcluster.net/docs/reference/integrations/github/checks#custom-text-output-in-checks lays out the approach for producing an 'output' object for GitHub Checks:

... it looks for an artifact on the task named public/github/customCheckRunText.md and uses the artifact content as the check run text ...

For example:

payload:
  ...
  artifacts:
    public/github/customCheckRunText.md:
      type: file
      path: checkrun.md

One can also override task.extra.github.customCheckRun.textArtifactName to change the directory that is checked.

[stephenmcgruer]

First experiment on lint: https://github.com/web-platform-tests/wpt/pull/24556/checks?check_run_id=858217231

We'll probably want to do some legwork to dump something a bit more readable (this is just log to file) but not bad.

[stephenmcgruer]

https://github.com/web-platform-tests/wpt/pull/24741 just added this output for stability checks (woo). It looks like:

[foolip]

That is sweet, thanks @stephenmcgruer! I've used this view a few times this week already.

[stephenmcgruer]

https://github.com/web-platform-tests/wpt/pull/24556 added one for lint too:

The various unittest setups could probably also do with some sort of output like this, but sadly they are much harder to integrate with (afaik), so I won't be tackling them currently. This issue can remain to track those.

[foolip]

@stephenmcgruer I would consider the original problem I reported here resolved. Closing this issue in celebration would be appropriate I think, and maybe open new issues for the case that remains? That'll probably be less important than what you've already addressed.

[stephenmcgruer]

@stephenmcgruer I would consider the original problem I reported here resolved. Closing this issue in celebration would be appropriate I think, and maybe open new issues for the case that remains? That'll probably be less important than what you've already addressed.

Seems reasonable. https://github.com/web-platform-tests/wpt/issues/25663 can be its successor