Runtime: SerialPort timeout tests are slow and flaky

[JeremyKuhne]

They don't play well with CI builds and take some time.

We need to:

1. Pick some more generous thresholds for validation
2. Try to make tests faster where possible
3. Move the slowest tests to the outer loop

dotnet/runtime#20091

[JeremyKuhne]

We really should avoid playing with the thread priority if at all possible. We should also look to reduce any unnecessary redundancy.

[willdean]

As I said on the PR, it's not just the timeout tests which are slow and flaky - there are lots and lots of slow and flaky tests in there, it's just that these are the ones which have caused trouble in the small subset which is currently running on the CI.

How would I go about marking all the 'needs a serial port' tests as non-CI? I think that would be better for now anyway.

[karelz]

@JeremyKuhne is it test bug or test enhancement?

[JeremyKuhne]

@karelz Er, test bug I think.

How would I go about marking all the 'needs a serial port' tests as non-CI?

@willdean We can mark things as outer loop, but that doesn't address the problem. I don't know if there is a good way to do this. @weshaggard, @tarekgh: do we have any patterns established for running tests locally only?

[weshaggard]

do we have any patterns established for running tests locally only?

I would suggest a conditional fact, which you can add any logic you want in it to detect whether or not you have a serial port or not.

[JeremyKuhne]

@weshaggard We already have that- the issue is that the CI machines have a serial port present and the tests don't work reliably in the CI environment.

[willdean]

We could just decide that the 'no loopback, just a single serial port' subset isn't very useful anyway, and effectively upgrade all the requirements to require either single-port loopback or null modem - that would mean that they'd only get run if there was a deliberate attempt to equip a machine for serial port testing.

There's so much missing from the tests when you don't have at least a loopback plug that we're probably kidding ourselves that it's a useful automated defence against regression anyway - the false sense of security is perhaps worse than the less frequent testing.

[weshaggard]

I see. The other potential way is to mark them as IgnoreForCI, see https://github.com/dotnet/corefx/blob/158800db40864779aadbe2b2e6916dbc495060b6/src/System.Runtime.Extensions/tests/System/EnvironmentTests.cs#L182.

[JeremyKuhne]

@weshaggard What does that do for outerloop? I assume these would struggle on that run as well...

[danmosemsft]

It would be nice if there were at least SOME that were reliable/fast enough we could run in the CI even if all they do is little more than new up a serial port.

[weshaggard]

See https://github.com/dotnet/corefx/blob/master/netci.groovy#L225 we pass IgnoreForCI in outerloop runs as well.

[JeremyKuhne]

Thanks @weshaggard

[willdean]

@danmosemsft There are currently about 120 out of 760 that can run with no serial port at all.

With a serial port but no attached hardware, you get about 320 out of 760 running. (This is what's trying to run on the CI at the moment)

[am11]

The node.js community has configured Jenkins based test machine for testing their serial-port implementation: nodejs/hardware#13. Would be nice to have a similar system employed to test .NET Core implementation on a real hardware.

[willdean]

@am11 Thanks for that - very interesting. In comparison with what we're trying to do they have very relaxed requirements for hardware but the principle is encouraging.

Somewhere in Redmond there is (I like to think so, anyway) a machine running the SerialPort test suite on behalf of .NET 4.6 - what we really need is a clone of that...