runtime 🚀 - Test failure: System.Net.NetworkInformation.Tests.NetworkInterfaceBasicTest/BasicTest_AccessInstanceProperties_NoExceptions_Linux

@ericeil I'm seeing this in some Ubuntu 16.10 runs as well. I'm pretty sure that the only way this could happen is if the actual statistics file from the system (/sys/class/net//speed) actually contains "-1" instead of a real speed value. I think we should just return null from the parsing routine in that case; it already returns null for other invalid values / IO errors.

mellinoe on 16 Sep 2016

Failing constantly again. Reopening.

System.Net.NetworkInformation.Tests.NetworkInterfaceBasicTest.BasicTest_AccessInstanceProperties_NoExceptions_Linux

netcoreapp5.0-Linux-Release-x64-Mono_release-Ubuntu.1804.Amd64.Open
netcoreapp5.0-Linux-Release-x64-Mono_release-RedHat.7.Amd64.Open
netcoreapp5.0-Linux-Release-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open
netcoreapp5.0-Linux-Release-x64-CoreCLR_release-Ubuntu.1804.Amd64.Open
netcoreapp5.0-Linux-Release-x64-CoreCLR_release-RedHat.7.Amd64.Open

Assert.InRange() Failure\r\nRange: (-1 - 9223372036854775807)\r\nActual: -1539607552
 at System.Net.NetworkInformation.Tests.NetworkInterfaceBasicTest.BasicTest_AccessInstanceProperties_NoExceptions_Linux() in /_/src/libraries/System.Net.NetworkInformation/tests/FunctionalTests/NetworkInterfaceBasicTest.cs:line 75
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) in /_/src/mono/netcore/System.Private.CoreLib/src/System/Reflection/RuntimeMethodInfo.cs:line 339

ViktorHofer on 12 Feb 2020

It fails on Assert.InRange(nic.Speed, -1, long.MaxValue); . I think it might be the conversion to int here: https://github.com/dotnet/runtime/blob/master/src/libraries/Native/Unix/System.Native/pal_interfaceaddresses.c#L376, because the function itself returns uint: https://elixir.bootlin.com/linux/v5.5.3/source/include/uapi/linux/ethtool.h#L123.
@wfurt Unless you immediately know what the problem is and the fix would literally take you just few minutes, I'll look into it. Seems like fun investigation 😄

ManickaP on 12 Feb 2020

Hit on #2098

CoffeeFlux on 12 Feb 2020

Another tracking issue for this: https://github.com/dotnet/runtime/issues/32179

akoeplinger on 12 Feb 2020

Not sure why my GitHub search didn't turn up this issue. Closing #32179 and using this as the primary. Will bring the test failure table over here.

jaredpar on 12 Feb 2020

System.Net.NetworkInformation.Functional.Tests Work Item

Console Log Summary

===========================================================================================================
~/work/AFA10939/w/A8C90920/e ~/work/AFA10939/w/A8C90920/e
  Discovering: System.Net.NetworkInformation.Functional.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.NetworkInformation.Functional.Tests (found 75 of 92 test cases)
  Starting:    System.Net.NetworkInformation.Functional.Tests (parallel test collections = on, max threads = 2)
    System.Net.NetworkInformation.Tests.NetworkInterfaceBasicTest.BasicTest_AccessInstanceProperties_NoExceptions_Linux [FAIL]
      Assert.InRange() Failure
      Range:  (-1 - 9223372036854775807)
      Actual: -1539607552
      Stack Trace:
        /_/src/libraries/System.Net.NetworkInformation/tests/FunctionalTests/NetworkInterfaceBasicTest.cs(75,0): at System.Net.NetworkInformation.Tests.NetworkInterfaceBasicTest.BasicTest_AccessInstanceProperties_NoExceptions_Linux()
      Output:
        - NetworkInterface -
        Name: lo
        Description: lo
        ID: lo
        Type: Loopback
        Status: Up
        Speed: -1
        SupportsMulticast: False
        GetPhysicalAddress(): 
        - NetworkInterface -
        Name: eth0
        Description: eth0
        ID: eth0
        Type: Ethernet
        Status: Up
        Speed: -1539607552
  Finished:    System.Net.NetworkInformation.Functional.Tests
=== TEST EXECUTION SUMMARY ===

Builds

|Build|Pull Request | Test Failure Count|
| --- | --- | --- |
|#515813|#1273|6|
|#516798|#2098|13|
|#516907|#31892|24|
|#517052|Rolling|7|
|#517102|#32155|9|
|#517112|#32157|7|
|#517133|#31834|1|
|#517135|#32158|2|
|#517142|#32159|2|
|#517177|#31898|7|
|#517179|Rolling|8|
|#517188|#32143|7|
|#517193|#32090|19|
|#517197|#32155|7|
|#517210|#32160|1|
|#517242|Rolling|5|
|#517252|#2111|9|
|#517290|#32160|6|
|#517322|Rolling|7|
|#517332|#31744|9|
|#517344|#32162|3|
|#517347|#32163|3|
|#517393|#31856|2|
|#517396|Rolling|3|
|#517427|#31957|1|
|#517446|Rolling|12|
|#517466|#32127|12|
|#517470|#32167|1|
|#517476|#32169|1|
|#517489|#771|3|
|#517491|#32170|3|
|#517495|#32039|3|
|#517500|#32032|4|
|#517502|#32130|5|
|#517515|Rolling|4|
|#517567|#31953|8|
|#517617|#32173|8|
|#517643|#32115|9|
|#517645|#32174|7|

Configurations

netcoreapp5.0-Linux-Debug-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open
netcoreapp5.0-Linux-Debug-x64-CoreCLR_release-Centos.7.Amd64.Open
netcoreapp5.0-Linux-Debug-x64-CoreCLR_release-Debian.9.Amd64.Open
netcoreapp5.0-Linux-Debug-x64-CoreCLR_release-RedHat.7.Amd64.Open
netcoreapp5.0-Linux-Debug-x64-CoreCLR_release-SLES.15.Amd64.Open
netcoreapp5.0-Linux-Debug-x64-CoreCLR_release-Ubuntu.1604.Amd64.Open
netcoreapp5.0-Linux-Debug-x64-CoreCLR_release-Ubuntu.1804.Amd64.Open
netcoreapp5.0-Linux-Debug-x64-Mono_release-Centos.7.Amd64.Open
netcoreapp5.0-Linux-Debug-x64-Mono_release-Debian.9.Amd64.Open
netcoreapp5.0-Linux-Debug-x64-Mono_release-RedHat.7.Amd64.Open
netcoreapp5.0-Linux-Debug-x64-Mono_release-SLES.15.Amd64.Open
netcoreapp5.0-Linux-Debug-x64-Mono_release-Ubuntu.1604.Amd64.Open
netcoreapp5.0-Linux-Debug-x64-Mono_release-Ubuntu.1804.Amd64.Open
netcoreapp5.0-Linux-Release-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open
netcoreapp5.0-Linux-Release-x64-CoreCLR_release-Centos.7.Amd64.Open
netcoreapp5.0-Linux-Release-x64-CoreCLR_release-Debian.9.Amd64.Open
netcoreapp5.0-Linux-Release-x64-CoreCLR_release-RedHat.7.Amd64.Open
netcoreapp5.0-Linux-Release-x64-CoreCLR_release-SLES.15.Amd64.Open
netcoreapp5.0-Linux-Release-x64-CoreCLR_release-Ubuntu.1604.Amd64.Open
netcoreapp5.0-Linux-Release-x64-CoreCLR_release-Ubuntu.1804.Amd64.Open
netcoreapp5.0-Linux-Release-x64-Mono_release-Centos.7.Amd64.Open
netcoreapp5.0-Linux-Release-x64-Mono_release-Debian.9.Amd64.Open
netcoreapp5.0-Linux-Release-x64-Mono_release-RedHat.7.Amd64.Open
netcoreapp5.0-Linux-Release-x64-Mono_release-SLES.15.Amd64.Open
netcoreapp5.0-Linux-Release-x64-Mono_release-Ubuntu.1604.Amd64.Open
netcoreapp5.0-Linux-Release-x64-Mono_release-Ubuntu.1804.Amd64.Open
netcoreapp5.0-Windows_NT-Release-x86-CoreCLR_release-Windows.7.Amd64.Open

Helix Logs

jaredpar on 12 Feb 2020

As a part of actually fixing this bug I think we need to do a root cause investigation here given the wide scale impact of this break. It essentially took down all builds for quite a long period of time.

jaredpar on 12 Feb 2020

@ManickaP if you're looking into this may want to take a look at https://github.com/dotnet/runtime/pull/1273. That may be the source of these failures. It's the earliest reference I can find to the break.

jaredpar on 12 Feb 2020

A bsd version of this test System.Net.NetworkInformation.Tests.NetworkInterfaceBasicTest.BasicTest_AccessInstanceProperties_NoExceptions_bsd fails in https://github.com/dotnet/corefx/pull/42850

https://helix.dot.net/api/2019-06-17/jobs/48eecc4c-5977-4aab-99bf-199a1cf97334/workitems/System.Net.NetworkInformation.Functional.Tests/console

This in 3.1 branch

Anipik on 12 Feb 2020

@wfurt Unless you immediately know what the problem is and the fix would literally take you just a few minutes, I'll look into it. Seems like fun investigation

no, I don't. At least for the OSX failures, it feels like something changes with the infrastructure. Since loopback should not really have speed, I would expect this to be -1.

wfurt on 13 Feb 2020

actually, never mind. I look at the wrong part @ManickaP. We should capture the output of ethtool eth0 on failure or you can update RunTest.sh to do that and try to get repro with a private run.

Fox OSX we should get output of networksetup -listallhardwareports ; ifconfig -a

wfurt on 13 Feb 2020

So I added the ethtool eth0 into RunTest.sh and got this:

Settings for eth0:
    Supported ports: [ ]
    Supported link modes:   Not reported
    Supported pause frame use: No
    Supports auto-negotiation: No
    Supported FEC modes: Not reported
    Advertised link modes:  Not reported
    Advertised pause frame use: No
    Advertised auto-negotiation: No
    Advertised FEC modes: Not reported
    Speed: 50000Mb/s
    Duplex: Full
    Port: Other
    PHYAD: 0
    Transceiver: internal
    Auto-negotiation: off
    Link detected: yes

For instance in: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-32223-merge-389ed0c11d0c4fedb6/System.Net.NetworkInformation.Functional.Tests/console.ec529df2.log?sv=2019-02-02&se=2020-02-23T13%3A17%3A47Z&sr=c&sp=rl&sig=keU1SXAgQX57de4XmOeEL%2FT5IO9iN7xD%2BuOE7LwtZu0%3D. More can be found in the failed tests of this WIP PR https://github.com/dotnet/runtime/pull/32223.

The problem isn't in the cast, but in the multiplying by 1 000 000 in here: https://github.com/dotnet/runtime/blob/master/src/libraries/Native/Unix/System.Native/pal_interfaceaddresses.c#L380
If you let unchecked(50000*1000000) overflow in int32, you'll get the incriminating -1539607552.

ManickaP on 13 Feb 2020

👍1

Good find @ManickaP. Probably time to bump it to long to get ready for 100G and beyond.

wfurt on 13 Feb 2020

🎉1

Good find @ManickaP. Probably time to bump it to long to get ready for 100G and beyond.

I'm already running tests in my PR for this fix.

ManickaP on 13 Feb 2020

@ManickaP thanks for tracking this down!

jaredpar on 13 Feb 2020

👍1

Runtime: Test failure: System.Net.NetworkInformation.Tests.NetworkInterfaceBasicTest/BasicTest_AccessInstanceProperties_NoExceptions_Linux

All 16 comments

System.Net.NetworkInformation.Functional.Tests Work Item

Console Log Summary

Builds

Configurations

Helix Logs

Related issues