Webrtc-pc: Is the testing policy being applied / working?

[foolip]

Taking a look at https://foolip.github.io/day-to-day/ today, I see that the WebRTC spec is much more active than the test suite. Looking at just the most recent PR, https://github.com/w3c/webrtc-pc/pull/2399, I see that lots of instances of "degradationPreference" remain in WPT.

In any individual instance it's easy to overlook whether test updates are needed. Would tooling to help check this be useful? It might take the form of an extra check on PRs which says "action required" if no link to a test update can be found. (Just an idea.)

[foolip]

@dontcallmedom WDYT?

@plehegar FYI

[alvestrand]

We do have a "test needed" label for issues, which we frequently remember to apply. Unfortunately we don't have anyone making the concerted effort needed to get the tests written, so there's a backlog.

[foolip]

It sounds like there's not generally the expectation that the test changes be made at the same time as the spec changes, and the decoupling results in a growing (?) backlog. Would the group be up for requiring a wpt PR together with spec PR before merging, to avoid this problem? The total amount of work should be the same, or less if the process of updating tests reveals new things about the spec, as often happens.

I think the tradeoff is that applying that policy will lead to some PRs being open for longer, but this seems to be working OK in other groups I've seen do this.

[henbos]

We have reviewed the test coverage of the entire document (except for chapter five, covered here: #2470). We have also deleted some featured since this issue was filed. We make mistake sometimes but we do ask for test coverage for every PR we review.

We don't have a strict policy of "test before merge".

[youennf]

We don't have a strict policy of "test before merge".

But editors ask whether tests can be written for each PR and if so, push for WPT tests to be written.

[henbos]

The degradationPreference tests have been moved to a separate spec, but so has the tests:
https://source.chromium.org/chromium/chromium/src/+/master:third_party/blink/web_tests/external/wpt/mst-content-hint/RTCRtpSendParameters-degradationPreference.html?q=degradationPreference%20f:wpt

[henbos]

Closing as "we think we're not perfect but we're doing better", please feel free to re-open if more action is needed.