Runtime: Clean Enlistment Test Failures

[kingces95]

After installing Windows 10 on a Parallels VM (on a MacPro), installing VS 2015, cloning, then running build, and build-tests from a VS 2015 Developer Command Prompt I consistently get the following errors:

Verified fixed by dotnet/corefx#24052

~C:\git\corefxTools\tests.targets(483,5): error : One or more tests failed while running tests from 'System.Globalization.Tests' please check C:\git\corefx\bin/tests/System.Globalization.Tests/netcoreapp-Windows_NT-Debug-x64/testResults.xml for details! [C:\git\corefx\src\System.Globalization\tests\System.Globalization.Tests.csproj]~

Verified fixed by dotnet/corefx#24016

~C:\git\corefxTools\tests.targets(483,5): error : One or more tests failed while running tests from 'System.Console.Tests' please check C:\git\corefx\bin/tests/System.Console.Tests/netcoreapp-Windows_NT-Debug-x64/testResults.xml for details! [C:\git\corefx\src\System.Console\tests\System.Console.Tests.csproj]~

Verified fixed by dotnet/corefx#24147

~C:\git\corefxTools\tests.targets(483,5): error : One or more tests failed while running tests from 'System.Net.NameResolution.Functional.Tests' please check C:\git\corefx\bin/tests/System.Net.NameResolution.Functional.Tests/netcoreapp-Windows_NT-Debug-x64/testResults.xml for details!
[C:\git\corefx\src\System.Net.NameResolution\tests\FunctionalTests\System.Net.NameResolution.Functional.Tests.csproj]~

Verified fixed by dotnet/corefx#24009 and dotnet/corefx#24097

~C:\git\corefxTools\tests.targets(483,5): error : One or more tests failed while running tests from 'System.Drawing.Common.Tests' please check C:\git\corefx\bin/tests/System.Drawing.Common.Tests/netcoreapp-Windows_NT-Debug-x64/testResults
.xml for details! [C:\git\corefx\src\System.Drawing.Common\tests\System.Drawing.Common.Tests.csproj]~

The console log.

[mellinoe]

The console log should include the error messages from the tests that failed. Is these any more information higher up in the log?

[kingces95]

@mellinoe Thanks for taking a look.

I posted a gist of the log here.

Here are some excerpts:

System.Globalization.Tests.NumberFormatInfoNumberDecimalDigits.NumberDecimalDigits_Get(format: NumberFormatInfo { CurrencyDecimalDigits = 2, CurrencyDecimalSeparator = \".\", CurrencyGroupSeparator = \",\", CurrencyGroupSizes = [3], CurrencyNegativePattern = 1, ... }, expectedWindows: 2, expectedIcu: 3) [FAIL]
Assert.Equal() Failure
Expected: 2
Actual: 3

System.Net.NameResolution.Tests.GetHostEntryTest.Dns_GetHostEntryAsync_NoneIPAddress_Fail(address: 255.255.255.255) [FAIL]
Assert.Throws() Failure
Expected: typeof(System.Net.Sockets.SocketException)
Actual: (No exception was thrown)

[mellinoe]

There are a few spurious System.Drawing failures in there -- I will try to submit a PR that fixes them.

@tarekgh Could you look at the Globalization failure and see if we know about it?

@danmosemsft I think he is hitting the issue you described here: https://github.com/dotnet/corefx/pull/21518. Did we file anything about that, or did it get fixed at some point?

Out of curiosity, what version of Windows are you running on?

[tarekgh]

@kingces95 what Windows version you are running on?

[kingces95]

Win 10 Enterprise, fully patched. Running inside parallels with a mac host.

[tarekgh]

@kingces95 if you can give me the exact version? in the command window type "systeminfo" and then give me the line like the following:

OS Version:                10.0.15063 N/A Build 15063

I am just wondering why this is failing on your machine and not failing in ours?

[jkotas]

@tarekgh The console tests are failing on some of my machines too. I think it has something to do with high-dpi and/or having physical monitor attached. If somebody wants to stop by my office, I will be happy to provide repro (I do not think it repros over TS).

[danmosemsft]

@mellinoe I don't see that info (about the console failure) but if it is that, I believe it was a bug in RS2. Not sure about RS1. No, I did not complete the PR dotnet/corefx#21518 - It's just setting console title to a huge string, so IMO it's not worth bothering to fix, the right thing is to adorn this test so it doesn't run on affected Windows versions.

[danmosemsft]

It may even have been particular builds of RS2 - I know it reproed consistently on my home laptop, but never at work. Again, that assumes it was failing for that reason and not some other random console problem.

[tarekgh]

@jkotas the globalization test failure has nothing to do with the machine configuration in general. do you see the globalization test failing on your machine?

[tarekgh]

@danmosemsft is right. Windows had a regression in RS2 which they fixed on RS3. The regression caused the framework to fail when setting the console title with long text. I agree it is not worth adding work around in the framework for this one as the problem occurred on RS2 only and happen when setting the Window title to a long string which is not a common scenario at all.

[jkotas]

Globalization tests are fine for me. The tests I see failing intermittently or on some machines are System.DirectoryServices.*, System.Net.*, System.Drawing and System.Console.

[tarekgh]

@kingces95 one more ask. could you run "intl.cpl" and then click on "Additional Settings" button and tell me the value of "No. of digits after decimal"? I assume you are running with en-US locale. thanks for your help.

[tarekgh]

@jkotas if you have RS2 then "WindowAndCursorProps.Title_Set_Windows(lengthOfTitle: 1024) [FAIL]" expected to fail

[danmosemsft]

perhaps @tarekgh you could slap a check for Windows version on that one so it doesn't come up again? I should have done this myself.

[tarekgh]

@danmosemsft I'll submit a PR for that.

[tarekgh]

dotnet/corefx#24016

[kingces95]

Sure. I'm running OS Version: 10.0.15063 N/A Build 15063.

It should be easy to reproduce my config: I'm running in a parallels VM on a MacBookPro. Start with a MBP, install latest version of Parallels, the follow the prompts. Parallels will automatically download windows and away you go.. patch, install VS, clone, build, test, bam. Reproduces every time..

[kingces95]

[kingces95]

What tool is everyone using to render the testResults.xml files and\or generate a dashboard of the test results?

[mellinoe]

@kingces95 I don't think anybody is doing anything like that.

[kingces95]

Hm. So how are folks investigating test failures? They're just looking at the raw console output and raw xml xunit output?

[tarekgh]

Thanks, @kingces95 for sending the OS info. It looks in your case you customized the globalization settings as it looks different than the exact version I am running on. But it is a good thing to fix the test anyway to handle such cases. I'll try to get the globalization tests fixed soon.

[mellinoe]

Hm. So how are folks investigating test failures? They're just looking at the raw console output and raw xml xunit output?

Basically, yeah. Hopefully when we sort out your initial problems and get to a clean state, it is more manageable. Usually the tests are expected to be 100% clean, unless you have changed something (and then the failure should be obvious). Also, it is possible to run individual test projects, which cuts down on some of the verbosity and makes things more manageable, especially when you are only working on one library at a time.

[kingces95]

@mellinoe Is there any more information you need about the Drawing and NameResolution failures? Or will I need to debug those?

[mellinoe]

@kingces95 Could you print out the failure message from the latest version of the tests in the master branch? I added some logging that should make it clear what the problem is.

[mellinoe]

And unfortunately, I'm not familiar with the NameResolution tests. @davidsh do you know anything more about that particular issue?

     System.Net.NameResolution.Tests.GetHostEntryTest.Dns_GetHostEntryAsync_NoneIPAddress_Fail(address: 255.255.255.255) [FAIL]
        Assert.Throws() Failure
        Expected: typeof(System.Net.Sockets.SocketException)
        Actual:   (No exception was thrown)
        Stack Trace:
           --- End of stack trace from previous location where exception was thrown ---
              at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
              at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
              at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
           C:\git\corefx\src\System.Net.NameResolution\tests\FunctionalTests\GetHostEntryTest.cs(106,0): at System.Net.NameResolution.Tests.GetHostEntryTest.<Dns_GetHostEntryAsync_NoneIPAddress_Fail>d__10.MoveNext()
           --- End of stack trace from previous location where exception was thrown ---
              at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
              at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           --- End of stack trace from previous location where exception was thrown ---
              at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
              at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           --- End of stack trace from previous location where exception was thrown ---
              at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
              at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

[davidsh]

@davidsh do you know anything more about that particular issue?

I don't know anything about this particular failure. What platform is this on? What version of Windows?

[kingces95]

@davidsh Sure. I'm running OS Version: 10.0.15063 N/A Build 15063.

It should be easy to reproduce my environment: I'm running in a parallels VM on a MacBookPro. Start with a MBP, install latest version of Parallels, the follow the prompts. Parallels will automatically download windows and away you go.. patch, install VS, clone, build, test, bam. Reproduces every time..

The XML and Console logs from my test run are above. @mellinoe copied what looks to be the relevant portion of the console log above.

If you're unable to reproduce it, then I could give you TeamViewer creds to a VM running on my laptop.

[davidsh]

cc: @Priya91 @mellinoe
Since this test failure is on Mac, @Priya91 should investigate. Thx.

System.Net.NameResolution.Tests.GetHostEntryTest.Dns_GetHostEntryAsync_NoneIPAddress_Fail(address: 255.255.255.255) [FAIL]
Assert.Throws() Failure
Expected: typeof(System.Net.Sockets.SocketException)
Actual: (No exception was thrown

[kingces95]

@Priya91 It's actually running in Windows 10, in Parallels, on a Mac. But, yeah, if you have a Mac then that'll make it much easier to reconstruct the environment. Just install Parallels, let it download windows, install VS, sync, build, run tests.

Could very well be that the virtual networking behaves in an unexpected fashion.

It'd be great if we could add a mac/parallels run to the CI.

[Priya91]

@kingces95 Since you already have the repro environment, can you report what is the output of the following code on the windows vm with .NET framework and .NET Core,

```c#
using System;
using System.Net;

public class Program
{
public static void Main(string[] args)
{
Console.WriteLine(Dns.GetHostEntryAsync(IPAddress.None).GetAwaiter().GetResult().AddressList[0]);
}
}
```

[kingces95]

@Priya91, they both print out the same thing:

VS2015 / Desktop 4.6.1 -> 255.255.255.255
VS2017 / NET Core 2.0 -> 255.255.255.255

[mellinoe]

@kingces95 Could you let me know what the error message is from the System.Drawing.Common tests is, if you are still hitting issues there? I added some logging that will help me diagnose the issue.

[Priya91]

Thanks @kingces95 ! @davidsh The same issue repros on .NET Framework as well. Can you please take a look?

[davidsh]

@Priya91

The same issue repros on .NET Framework as well. Can you please take a look?

Are you saying the test is failing in CI against .NET Framework Desktop (on a real Windows machine)?

Or are you saying that that the test is failing when run against .NET Framework in a Windows container on top of Parallels on a Mac?

[kingces95]

@mellinoe I updated the gist links above to the System.Drawing.Common XML as well as the console log during the test run. Hope it helps!

[Priya91]

Or are you saying that that the test is failing when run against .NET Framework in a Windows container on top of Parallels on a Mac?

Yes, the same code, when run against .NET Framework should throw SocketException, but it does not on a windows vm on top of parallels on mac. Below is the repro code used, same as the failing test.

```c#
using System;
using System.Net;

public class Program
{
public static void Main(string[] args)
{
Console.WriteLine(Dns.GetHostEntryAsync(IPAddress.None).GetAwaiter().GetResult().AddressList[0]);
}
}
```

[davidsh]

Yes, the same code, when run against .NET Framework should throw SocketException, but it does not on a windows vm on top of parallels on mac

Ok, so the variable here is that the "hardware" is not a real machine but rather the Parallels hypervisor architecture on a Mac.

cc: @karelz You'll need a Mac person to troubleshoot this since this is a Mac specific problem causing different behaviors to leak thru into the Windows container.

[karelz]

@kingces95 It'd be great if we could add a mac/parallels run to the CI.

Naively I would expect Windows under Parallels on Mac to behave in similar way as in any other virtualization technology (with Windows/Linux host). Ideally it should behave the same way as on physical HW (I assume that's how Windows VM on Windows/Linux host behaves).
I wonder why that is not the case here - maybe Parallels/driver bug?

Given that we do not have unlimited Mac HW, adding new CI legs to run Windows in Mac/Parallels is very costly for us. It also adds complexity to the infrastructure - and we have much higher priority gaps/bugs in infrastructure, so this would be very low pri from that perspective.
The impact on CoreFX contributors seems to be fairly contained ("just" 4 tests difference, brought down to 2 now) compared to other problems/differences in our test bed affecting contributors.

Overall, it feels like being reactive to discovered differences seems the right thing to do from cost/benefit point of view at this moment.

If there is a way to detect we are running under Mac/Parallels virtualization, we should be able to disable/modify the tests in such environment. Is it possible to detect?
Otherwise we could just consider this kind of error in particular tests as "by design".

[jkotas]

Parallels is injecting extra DNS mappings. They maybe unusual, but I would not consider them to be a bug.

You can replicate the parallels behavior on regular Windows installation by going to C:\Windows\System32\drivers\etc\hosts and adding the following line there:

       255.255.255.255  whatever

Console.WriteLine(Dns.GetHostEntryAsync(IPAddress.None).GetAwaiter().GetResult().AddressList[0]); will happily print 255.255.255.255 once you do that. So this looks like a test sensitive to machine environment that should better be OuterLoop or something.

[jkotas]

Also, adding the 255.255.255.255 entry is recommended as workaround for all sorts of problems. For example, https://technet.microsoft.com/en-us/library/security/ms16-077.aspx suggests to add 255.255.255.255 wpad as a workaround. So this problem is not specific to Parallels. Any machine with these sort of workarounds applied will have the same problem.

[davidsh]

Also, adding the 255.255.255.255 entry is recommended as workaround for all sorts of problems. For example, https://technet.microsoft.com/en-us/library/security/ms16-077.aspx suggests to add 255.255.255.255 wpad as a workaround. So this problem is not specific to Parallels. Any machine with these sort of workarounds applied will have the same problem.

Interesting. We'll need to study this test and replace "255.255.255.255" with something else that is appropriate for this particular test.

[karelz]

Thanks @jkotas! We should probably split off the Networking test discussion and harden the test more against not-so-common environment settings.

[davidsh]

We should probably split off the Networking test discussion and harden the test more against not-so-common environment settings.

Based on this discussion, it sounds like these particular tests are invalid. The test was originally created because it was assumed that doing a DNS lookup against IPAddress.None (which is the broadcast address, 255.255.255.255) would always be invalid. Then it was tweaked to ignore the test on MacOS.

```c#
public static IEnumerable {
yield return new object[] { IPAddress.None };
}

[PlatformSpecific(~TestPlatforms.OSX)] // macOS will resolve IPAddress.None to broadcasthost and produce a valid listing
[Theory]
[MemberData(nameof(GetNoneAddresses))]
public async Task Dns_GetHostEntryAsync_NoneIPAddress_Fail(IPAddress address)
{
string addressString = address.ToString();

await Assert.ThrowsAnyAsync<SocketException>(() => Dns.GetHostEntryAsync(address));
await Assert.ThrowsAnyAsync<SocketException>(() => Dns.GetHostEntryAsync(addressString));

}

[PlatformSpecific(TestPlatforms.OSX)] // macOS will resolve IPAddress.None to broadcasthost and produce a valid listing
[Theory]
[MemberData(nameof(GetNoneAddresses))]
public async Task Dns_GetHostEntryAsync_NoneIPAddress_Success(IPAddress address)
{
IPHostEntry result = await Dns.GetHostEntryAsync(address);
Assert.NotNull(result);
Assert.NotNull(result.AddressList);
Assert.Equal(1, result.AddressList.Length);
Assert.Equal(address, result.AddressList[0]);
}
```

But in reality, since this IP address could have a DNS mapping on any platform via hard-coding it in the hosts file, this test should be removed. I will submit a PR to take care of it.

[kingces95]

Woot! Everything is fixed!

@mellinoe I'd close this issue out except I'm not sure which change fixed the drawing issue so I'm not sure if it's actually fixed or just a sporadic error and I happened to get lucky this time.

[mellinoe]

@kingces95 Glad to hear it! Drawing was fixed by https://github.com/dotnet/corefx/pull/24009 and https://github.com/dotnet/corefx/pull/24097. The problem in both is that the tests were making assumptions about what kind of printer you had installed, basically, and would fail if something was different than expected.