Runtime: SerialPort Test migration tracking

[willdean]

@SWY1985 @msalsbery @daflame76 I've opened this issue for the four of us to keep track of what's being done on migrating the SerialPort tests over to CoreFX style.

@SWY1985 It sounds like you're going to go first on this (today?) - perhaps you could let us know in here what bits you're doing/done so that we don't trip over each other.

[JeremyKuhne]

Let me know if you have any questions.

[JeremyKuhne]

Also, if you could add what actual hardware setups you've got and have run against that would be useful. I'm getting a couple USB serial port adapters myself.

[willdean]

@JeremyKuhne You may already know this, but if you're buying USB-Serial adaptors, you want to make sure they're either FTDI or Prolific-based, and you need to try and buy them from somewhere where they're likely to be genuine (in the UK you can just buy stuff from FTDI themselves, but I'm not sure on the international options. People like Digikey are probably pretty safe.) The world is awash with 'fake' devices which try to free-ride on the multi-decade driver investments by the 'big-name' manufacturers, and the named mfgs are in an arms race where they keep trying to break the fake devices via driver updates.

There's a world of wasted time out there for people trying to run library tests against broken hardware.

[shaggygi]

I agree with @willdean. I usually get from DigiKey or Mouser. Here are a couple of links.

• https://shop.clickandbuild.com/cnb/shop/ftdichip?productID=342&op=catalogue-product_info-null&prodCategoryID=293
• http://www.digikey.com/products/en?keywords=CHIPI-X10

[JeremyKuhne]

dotnet/corefx#15796 has a couple blocking fixes. CI is down at the moment- when that gets settled I'll get it in. Changes are relatively minor.

[SWY1985]

I have made a test setup at home with a couple of RS485 devices, but there were some personal issues so I couldn't get around to testing them. I hope I will be able to set up my test program within the next couple of days.

[willdean]

@SWY1985 Unless I hear you shout, I'll make a start today on straightforward migration of the legacy test files, in the way that @JeremyKuhne has already done for Close.cs This will probably touch a lot of files but not alter the structure of the project in any significant way.

[willdean]

I just pushed about 20 or so migrated tests to dotnet/corefx#15867 - anybody thinking modding test code in the next day or so should check on progress first, because these are very large changes which would be hard to merge.

[willdean]

OK - I've completed a first pass through everything 'normal' (i.e. can be run simply in xUnit) in Legacy/SerialPorts. There is still Legacy/SerialStream to do, and there's lots of clean up to do.

It's clear that the tests push very hard for exact compatibility with serial.sys, and there are a bunch of failures with FTDI devices (and on the CI, not sure what the serial port actually is on there) which don't occur when running with a real (16550A/serial.sys) port.

Here are my local results, on a Win10 x64 machine:

1. No serial port at all
System.IO.Ports.Tests Total: 763, Errors: 0, Failed: 0, Skipped: 641, Time: 0.447s

2. Real (16550/serial.sys) port COM1, no loopback:
System.IO.Ports.Tests Total: 763, Errors: 0, Failed: 3, Skipped: 435, Time: 128.238s

These seem to have no support on CoreFX - they pass on 4.6.1:

Encoding_Property.Encoding_Custom [FAIL]
Encoding_Property.Encoding_Japanese_JIS [FAIL]
Encoding_Property.Encoding_ISCIIAssemese [FAIL]

3. Real (16550/serial.sys) port COM1, with data-loopback plug:
System.IO.Ports.Tests Total: 763, Errors: 0, Failed: 5, Skipped: 267, Time: 390.287s

(Same three encoding failures as above)

This fails on 4.6.1 too - I don't think I've broken the test in the migration:
ReadChar.Read_Surrogate [FAIL]

This lost the proper unicode test data in the migration into GitHub - someone (@JeremyKuhne?) needs to go back to the original source:
SerialPortRegressions.UTF8Encoding [FAIL]

4. FTDI Port (COM4), no loopback/null modem:
System.IO.Ports.Tests Total: 763, Errors: 0, Failed: 23, Skipped: 435, Time: 127.188s

This is just cababilities - the FTDI H/w doesn't support 1.5 stop bits
DataBits_Property.DataBits_8_StopBitsOnePointFive [FAIL]

There are probably a mixture of subtle driver divergences from serial.sys (some might be driver 'bugs', some might not be)

    DiscardOutBuffer.OutBufferFilled_Discard_Multiple [FAIL]
    DiscardOutBuffer.OutBufferFilled_Discard_Once [FAIL]
    DiscardOutBuffer.OutBufferFilled_Discard_Cycle [FAIL]

    WriteLine_Generic.BytesToWriteSuccessive [FAIL]
    WriteLine_Generic.SuccessiveReadTimeout [FAIL]
    WriteLine_Generic.BytesToWrite [FAIL]
    WriteTimeout_Property.WriteTimeout_Default_Write_str [FAIL]
    WriteTimeout_Property.WriteTimeout_Default_WriteLine [FAIL]
    WriteTimeout_Property.WriteTimeout_Infinite_Write_char_int_int [FAIL]
    WriteTimeout_Property.WriteTimeout_Default_Write_char_int_int [FAIL]
    WriteTimeout_Property.WriteTimeout_Default_Write_byte_int_int [FAIL]
    WriteTimeout_Property.WriteTimeout_Infinite_Write_str [FAIL]
    WriteTimeout_Property.WriteTimeout_Infinite_WriteLine [FAIL]
    WriteTimeout_Property.WriteTimeout_Infinite_Write_byte_int_int [FAIL]
    Write_byte_int_int_generic.SuccessiveReadTimeout [FAIL]
    Write_char_int_int_generic.SuccessiveReadTimeout [FAIL]
    Write_str_Generic.BytesToWrite [FAIL]
    Write_str_Generic.SuccessiveReadTimeout [FAIL]
    Write_str_Generic.BytesToWriteSuccessive [FAIL]

5. FTDI Port (COM4), data-loopback plug fitted :
System.IO.Ports.Tests Total: 763, Errors: 0, Failed: 35, Skipped: 267, Time: 349.379s

Not going to list-out all the failures - various of them might be down to subtle timing differences, which could be considered bugs in the test suite rather than in the port/drivers. A USB-Serial adaptor is NEVER going to have exactly the same timing characteristics as a real port, as there's packetisation in the stack which just doesn't happen with a real port.

6. FTDI Port (COM4/5), null modem cable fitted :

This is a complete wreck right now - it enables every test, and currently hangs somewhere. I'm going to sort out some hardware with two 'real' ports and a null-modem to let me get it right before we worry about FTDI compat.

[willdean]

After we've merged this when I've blocked-up the tests that fail on the CI, anyone who wants a gentle play with the source could go through things in Legacy/SerialPort and gradually migrate some of the if(xxx) { Fail() } pattern into proper xUnit test assertions. I've done it where I could be bothered and where the predicate was very simple, but in the whirlwind of migrating this many tests I have not risked getting greater-than/less-than calculations backwards or messing-up the rather elaborate "check this exception is thrown perhaps sometimes or maybe don't check it at all" methods which are all over the place.

Or just do spelling mistakes and typos, of which there are an almost infinite number.

[willdean]

In case anyone's watching this but not seeing the other stuff go through:

• We've migrated all the original tests (~1000 of them) over to work with the CoreFx test framework.
• The tests all pass using the Core version of SerialPort where you have well-behaved hardware
• There are various hardware configurations which don't (and won't ever) pass all the tests, but I might be able to improve this a bit further.
• There is still much cosmetic clean-up to be done in the test suite.
• All the tests which rely on hardware (~900 of the set) are disabled on the CI system because they're insufficiently reliable there - it's currently unresolved how we might run the full suite 'officially'.

[shaggygi]

@willdean I can't express how much I (and others) appreciate your work in this space. I'm watching your progress as this is a MUCH needed feature. Keep up the good work 😄

[JeremyKuhne]

Closing this as the test are ported and there are individual issues to track other test problems.