Fsharp: stackoverflow when running ./build -tests

[smoothdeveloper]

edit: known issue when running test compiled in Debug mode code.

work around:

build.cmd -test -c Release

I'm getting this when running ./build -tests

  V Reverse slicing should not work with at symbol in 1st slice index [2ms]
  V Reverse slicing should not work with at symbol in 2nd slice index [3ms]
  V Reverse slicing should not work with at symbol in both slice index [2ms]
  V Fixed index 4d slicing should not be available in 47 [5s 249ms]
  V Else branch is a function that returns int while if branch is string [5s 297ms]
  V Reverse slicing should work with overloaded infix hat [5s 528ms]
The active test run was aborted. Reason: Test host process crashed : Process is terminated due to StackOverflowException.

Test run in progress.Results File: C:\dev\src\github.com\Microsoft\masterfsharp\artifacts\TestResults\Debug\FSharpSuite.Tests_net472.xml

Test Run Aborted.
Total tests: Unknown
     Passed: 97
 Total time: 1.3206 Minutes
Deferred Messages

  Detailed Build Summary
  ======================


  ============================== Build Hierarchy (IDs represent configurations) =====================================================
  Id                  : Exclusive Time   Total Time   Path (Targets)
  -----------------------------------------------------------------------------------------------------------------------------------
  0                   : 80.547s           80.547s       C:\dev\src\github.com\Microsoft\masterfsharp\tests\fsharp\FSharpSuite.Tests.fsproj (VSTest)

  ============================== Node Utilization (IDs represent configurations) ====================================================
  Timestamp:            1        Duration   Cumulative
  -----------------------------------------------------------------------------------------------------------------------------------
  637146961850490571:   0        80.549s     80.549s # (scale 1:1000)
  -----------------------------------------------------------------------------------------------------------------------------------
  Utilization:          100.0    Average Utilization: 100.0

Build FAILED.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:01:20.58
Command failed to execute with exit code 1: C:\Program Files\dotnet\dotnet.exe test C:\dev\src\github.com\Microsoft\masterfsharp\tests\fsharp\FSharpSuite.Tests.fsproj -c Debug -f net472 -v n --test-adapter-path . --logger "nunit;LogFilePath=C:\dev\src\github.com\Microsoft\masterfsharp\artifacts\TestResults\Debug\FSharpSuite.Tests_net472.xml" /bl:C:\dev\src\github.com\Microsoft\masterfsharp\artifacts\log\Debug\FSharpSuite.Tests_net472.binlog --no-restore --no-build
System.Management.Automation.RuntimeException: Command failed to execute with exit code 1: C:\Program Files\dotnet\dotnet.exe test C:\dev\src\github.com\Microsoft\masterfsharp\tests\fsharp\FSharpSuite.Tests.fsproj -c Debug -f net472 -v n --test-adapter-path . --logger "nunit;LogFilePath=C:\dev\src\github.com\Microsoft\masterfsharp\artifacts\TestResults\Debug\FSharpSuite.Tests_net472.xml" /bl:C:\dev\src\github.com\Microsoft\masterfsharp\artifacts\log\Debug\FSharpSuite.Tests_net472.binlog --no-restore --no-build
at Exec-CommandCore, C:\dev\src\github.com\Microsoft\masterfsharp\eng\build-utils.ps1: line 50
at Exec-Console, C:\dev\src\github.com\Microsoft\masterfsharp\eng\build-utils.ps1: line 119
at TestUsingNUnit, C:\dev\src\github.com\Microsoft\masterfsharp\eng\build.ps1: line 260
at <ScriptBlock>, C:\dev\src\github.com\Microsoft\masterfsharp\eng\build.ps1: line 345
at <ScriptBlock>, <No file>: line 1

I've seen similar error running tests in VS and Rider.

Also, I'm not able to use nunit GUI test runner which gives:

FSharpSuite.Tests.dll (TestFixtureSetUp):
Could not load file or assembly 'FSharpSuite.Tests, Version=10.7.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. Strong name signature could not be verified. The assembly may have been tampered with, or it was delay signed but not fully signed with the correct private key. (Exception from HRESULT: 0x80131045)

I've tried loading artifacts\bin\FSharpSuite.Tests\Debug\net472\FSharpSuite.Tests.dll in it.

Sorry if this is known issue, looking for a work around.

[cartermp]

The NUnit GUI runner issue sounds like a typical .NET Framework binding redirects problem. I'd be shocked if that issue is in this repo since it looks like a very very old program that hasn't been updated for NUnit 3.

As for the test itself, I'm not sure what the issue is here but 5 seconds for running some of these tests is way too long, which would indicate something is going on. @TIHan touched the compiler assertion framework last, but none of those changes looked like they would hit this. We don't see this in CI either (and CI machines are generally not that great). Does this reproduce for you?

[smoothdeveloper]

Thanks for the feedback

I'd be shocked if that issue is in this repo since it looks like a very very old program that hasn't been updated for NUnit 3.

The program is up to date and for nunit 3, it is called testcentric-gui (on nuget), and it generally works way faster than any of the in IDE runners around.

5 seconds for running some of these tests is way too long

It is a 2017 laptop with dual core i7 with 16gb of RAM, and I have two fsharp.sln open, one in vs2019 and the other in Rider.

The tests running a little slow is ok, I'll give it another shot to see if this happens in another place in the test list, and giving my laptop more breathing room while doing so, I'll report back soon.

[smoothdeveloper]

Here is an updated run, I've switched to "high performance" and closed all the IDEs

after running the tests

Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.
NUnit Adapter 3.11.2.0: Test execution started
Running all tests in C:\dev\src\github.com\Microsoft\masterfsharp\artifacts\bin\FSharpSuite.Tests\Debug\net472\FSharpSuite.Tests.dll
   NUnit3TestExecutor converted 600 of 600 NUnit test cases
  V Basic constants compile  [2s 545ms]
  V Defensive copy on .NET struct for inref [3s 441ms]
  V NotStructConstraintFail [3s 581ms]
  V Test1 [3s 702ms]
  V float 32 with underscores [2s 363ms]
  V Extension method scope errors [1s 501ms]
  V NotStructConstraintPass [1s 371ms]
  V Test2 [1s 260ms]
  V float with dot [446ms]
  V No defensive copy on .NET struct [447ms]
  V StructConstraintFail [452ms]
  V Test3 [459ms]
  V Do Cannot Have Visibility Declarations [40ms]
  V float without dot [437ms]
  V StructConstraintPass [396ms]
  V Test4 [391ms]
  V Matching Method With Same Name Is Not Abstract [463ms]
  V floats with dot should be equal to floats without dot [439ms]
  V Literal Value [1s 120ms]
  V Test5 [1s 129ms]
  V Member Has Multiple Possible Dispatch Slots [1s 101ms]
  V int with multiple underscores literal [1s 274ms]
  V Mutation 01 [1s 262ms]
  V Test6 [1s 253ms]
  V Method Is Not Static [1s 210ms]
  V int with single underscore after leading zero literal [1s 183ms]
  V Mutation 02 [965ms]
  V Simple exe should execute if dependency was found and is not statically linked [1s 823ms]
  V No Matching Abstract Method With Same Name [1s 820ms]
  V int with single underscore after leteral with leading zero  [1s 825ms]
  V Mutation 03 [1s 830ms]
  V Simple exe should fail to execute if dependency was not found and is not statically linked [1s 780ms]
  V Tuple In Abstract Method [1s 783ms]
  V int with single underscore after octo leteral  [1s 727ms]
  V Mutation 04 [1s 757ms]
  V Standalone linking [14s 804ms]
  V Wrong Arity [14s 813ms]
  V int with single underscore Hex literal [14s 806ms]
  V Mutation 05 [14s 815ms]
  V Standalone linking - optimized [6s 901ms]
  V TailCall 01 [7s 282ms]
  V int with single underscore literal [7s 376ms]
  V Comma In Rec Ctor [6s 916ms]
  V Static link quotes in multiple modules [2s 315ms]
  V TailCall 02 [2s 302ms]
  V int with underscore hexWords [2s 290ms]
  V Extra Argument In Ctor [2s 265ms]
  V Static link quotes in multiple modules - optimized [1s 987ms]
  V TailCall 03 [2s 100ms]
  V int with underscore nybbles [2s 57ms]
  V Extra Argument In Ctor2 [2s 48ms]
  V Static link simple library [1s 858ms]
  V TailCall 04 [1s 834ms]
  V int with underscores bytes [1s 831ms]
  V Invalid Record [1s 841ms]
  V Custom collection with GetSlice and GetReverseIndex should support reverse index set slicing [1s 477ms]
  V TailCall 05 [1s 432ms]
  V int with underscores hexBytes [1s 445ms]
  V Missing Comma In Ctor [1s 450ms]
  V Custom collection with GetSlice and GetReverseIndex should support reverse index slicing [1s 381ms]
  V Dont Suggest Completely Wrong Stuff [951ms]
  V Long with underscores [997ms]
  V Missing Ctor Value [986ms]
  V Custom collection with Item and GetReverseIndex should support n-rank reverse index indexing [1s 4ms]
  V Dont Suggest Intentionally Unused Variables [1s 4ms]
  V Dont Suggest When Things Are Open [4ms]
  V Long with underscores maxLong [956ms]
  V Valid Comma In Rec Ctor [949ms]
  V Custom collection with Item and GetReverseIndex should support n-rank reverse index mutation [1s 6ms]
  V Else branch context doesn't propagate into for loop [1s 21ms]
  V Fail if else branch is missing [565ms]
  V Assign to immutable error [548ms]
  V Custom collection with Item and GetReverseIndex should support reverse index indexing [473ms]
  V Else branch context doesn't propagate into function application [453ms]
  V Fail if else branch is missing in nesting [465ms]
  V Fixed index 3d set slicing should not be available in 47 [525ms]
  V Custom collection with Item and GetReverseIndex should support reverse index mutation [590ms]
  V Else branch context doesn't propagate into function application even if not last expr [597ms]
  V Fail on type error in condition [598ms]
  V Fixed index 3d slicing should not be available in 47 [593ms]
  V Custom collection with Item and no GetReverseIndex should not support reverse index indexing [122ms]
  V Else branch context doesn't propagate to lines before last line [106ms]
  V Script_ForInDoMutableRegressionTest [1s 67ms]
  V At operator should not be usable in prefix context [48ms]
  V Fixed index 4d set slicing should not be available in 47 [1s 93ms]
  V Custom collection without GetReverseIndex should not support reverse index slicing [1s 263ms]
  V Else branch has wrong type in nested if [1s 276ms]
  V Hat operator should be overloadable in infix context [680ms]
  V Hat operator should not be overloadable in prefix context [5ms]
  V Reverse indexing should not work with at symbol [12ms]
  V Reverse slicing should not work with at symbol in 1st slice index [1ms]
  V Reverse slicing should not work with at symbol in 2nd slice index [1ms]
  V Reverse slicing should not work with at symbol in both slice index [6ms]
  V Fixed index 4d slicing should not be available in 47 [752ms]
  V LargeExprDoesNotStackOverflow [3s 475ms]
  V Else branch is a function that returns int while if branch is string [3s 484ms]
  V Reverse slicing should work with overloaded infix hat [3s 457ms]
  V LargeListExprDoesNotStackOverflow [909ms]
  V Else branch is a longer sequence of expressions that returns int while if branch is string [908ms]
  V FieldNotInRecord [530ms]
The active test run was aborted. Reason: Test host process crashed : Process is terminated due to StackOverflowException.

Test run in progress...Results File: C:\dev\src\github.com\Microsoft\masterfsharp\artifacts\TestResults\Debug\FSharpSuite.Tests_net472.xml

Test Run Aborted.
Total tests: Unknown
     Passed: 101
 Total time: 1.0510 Minutes
Deferred Messages

  Detailed Build Summary
  ======================


  ============================== Build Hierarchy (IDs represent configurations) =====================================================
  Id                  : Exclusive Time   Total Time   Path (Targets)
  -----------------------------------------------------------------------------------------------------------------------------------
  0                   : 64.000s           64.000s       C:\dev\src\github.com\Microsoft\masterfsharp\tests\fsharp\FSharpSuite.Tests.fsproj (VSTest)

  ============================== Node Utilization (IDs represent configurations) ====================================================
  Timestamp:            1        Duration   Cumulative
  -----------------------------------------------------------------------------------------------------------------------------------
  637147109090490654:   0        64.002s     64.002s # (scale 1:1000)
  -----------------------------------------------------------------------------------------------------------------------------------
  Utilization:          100.0    Average Utilization: 100.0

Build FAILED.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:01:04.03
Command failed to execute with exit code 1: C:\Program Files\dotnet\dotnet.exe test C:\dev\src\github.com\Microsoft\masterfsharp\tests\fsharp\FSharpSuite.Tests.fsproj -c Debug -f net472 -v n --test-adapter-path . --logger "nunit;LogFilePath=C:\dev\src\github.com\Microsoft\masterfsharp\artifacts\TestResults\Debug\FSharpSuite.Tests_net472.xml" /bl:C:\dev\src\github.com\Microsoft\masterfsharp\artifacts\log\Debug\FSharpSuite.Tests_net472.binlog --no-restore --no-build
System.Management.Automation.RuntimeException: Command failed to execute with exit code 1: C:\Program Files\dotnet\dotnet.exe test C:\dev\src\github.com\Microsoft\masterfsharp\tests\fsharp\FSharpSuite.Tests.fsproj -c Debug -f net472 -v n --test-adapter-path . --logger "nunit;LogFilePath=C:\dev\src\github.com\Microsoft\masterfsharp\artifacts\TestResults\Debug\FSharpSuite.Tests_net472.xml" /bl:C:\dev\src\github.com\Microsoft\masterfsharp\artifacts\log\Debug\FSharpSuite.Tests_net472.binlog --no-restore --no-build
at Exec-CommandCore, C:\dev\src\github.com\Microsoft\masterfsharp\eng\build-utils.ps1: line 50
at Exec-Console, C:\dev\src\github.com\Microsoft\masterfsharp\eng\build-utils.ps1: line 119
at TestUsingNUnit, C:\dev\src\github.com\Microsoft\masterfsharp\eng\build.ps1: line 260
at <ScriptBlock>, C:\dev\src\github.com\Microsoft\masterfsharp\eng\build.ps1: line 345
at <ScriptBlock>, <No file>: line 1

V Fixed index 4d slicing should not be available in 47 [752ms]

it took shorter for this particular test, but overall the figures look a bit volatile to draw a conclusion.

As it failed a bit later in the test suite, this indicates it is not determistic about a given test.

I'll try to work around by running tests in IDE in smaller bunch and append any info that may be relevant to the issue.

If there is anything I can do to help troubleshooting, or if anyone running the tests hit this, please let me know.

[abelbraaksma]

I think a recent change in testcentric was that now by default they're running parallel, could you check if parallel vs sequential makes a difference?

[smoothdeveloper]

@abelbraaksma thanks.

I'm not able to even run any of them in testcentric-gui, it seems that it has issue with delay signed assemblies as the message indicates, despite as @cartermp was saying, the beginning of the error message looks like a binding redirect.

If there is a way to disable signing on the unit test assemblies locally in the build, I could confirm if this is the issue with the gui runner (which can be useful but is not mandatory for contribution to this repository).

I'm able to run small batches of tests I need in the IDE, and I suspect the issue may be environment performance related: the issue is not blocking me.

It would be good if someone else using the repository can confirm the issue is showing up on equivalent hardware though.

[abelbraaksma]

I'll try to have a go at it, haven't run the tests of F# in a while, but I do remember reporting binding issues before, maybe I see something familiar and it rings a bell (my setup is similar to yours, except for the cpu, which in my case is dual cpu and 12 core each, and not a laptop, so I hope it goes a bit faster ;).

Though I have to say, given your configuration, the average runtime of each test seems rather high

[KevinRansom]

Yes, stack-overflow can happen running the tests with debug. Many work, but some don't. By default the debug build disables use of the .tail command. I suppose this was originally done to improve the debugger experience, but it does mean there is a chance that some looping code will blow the stack.

Kevin

[smoothdeveloper]

@KevinRansom what's the way to run the tests in release build? this should be mentionned in https://github.com/dotnet/fsharp/blob/master/TESTGUIDE.md (seems the instructions are outdated there) or https://github.com/dotnet/fsharp/blob/master/DEVGUIDE.md

[yatli]

@smoothdeveloper hello fellow Elektronaut! -c Release would do.

[smoothdeveloper]

Thanks @yatli!

I actually updated the dev/test guides some time back, and now just updated the comment at the top of the issue with the workaround in case someone else hits the same issue.

[cartermp]

Will close this out as by design for now, it kind of sucks, but at least the proper steps are documented.

[abelbraaksma]

FYI, the test guides have very recently been revised, and this particular issue, and a few others, are addressed, plus more details on using release vs debug builds.