Runtime: Test: System.SpanTests.ReadOnlySpanTests/IndexOverflow failed with "System.OutOfMemoryException"

[dotnet-mc-bot]

Opened on behalf of @Jiayili1

The test System.SpanTests.ReadOnlySpanTests/IndexOverflow has failed.

System.OutOfMemoryException : Exception of type 'System.OutOfMemoryException' was thrown.

    Stack Trace:

       at System.ReadOnlySpan`1..ctor(Void* pointer, Int32 length)
   at System.SpanTests.ReadOnlySpanTests.IndexOverflow() in /root/corefx/src/System.Memory/tests/ReadOnlySpan/Overflow.cs:line 36

Build : Master - 20170301.02 (Core Tests)
Failing configurations:

• suse.421.amd64
  
  
  
  • x64-Debug
  
  
  • x64-Release
  
  

Detail: https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20170301.02/workItem/System.Memory.Tests/analysis/xunit/System.SpanTests.ReadOnlySpanTests~2FIndexOverflow

[ahsonkhan]

@AtsushiKan
I am not sure why constructing a 3GB ReadOnlySpan would fail. Unless the machine that is running this test doesn't have enough memory. I am guessing that is the case here.

In that case, why didn't the try/catch catch the exception?
https://github.com/dotnet/corefx/blob/master/src/System.Memory/tests/ReadOnlySpan/Overflow.cs#L29

```C#
ThreeGiB = 3221225472
Unsafe.SizeOf() = 16 // ?
GuidThreeGiBLimit = 201326592

```C#
ReadOnlySpan<Guid> span = new ReadOnlySpan<Guid>(pMemory, GuidThreeGiBLimit);
...
private const long ThreeGiB = 3L * 1024L * 1024L * 1024L;
private static readonly int GuidThreeGiBLimit = (int)(ThreeGiB / Unsafe.SizeOf<Guid>());

[ghost]

Stack suggests the big allocation succeeded – OOM is happening inside the ReadOnlySpan constructor (not that this makes a whole lot more sense.)

[karelz]

@KrzysztofCwalina says that it is either test bug or bug in runtime (JIT cannot find memory) - which is likely by design.
The test should not run into that.

[ahsonkhan]

cc @jkotas, @russellhadley

[jkotas]

• The tests that allocate gigabytes of memory should be maked [OuterLoop] because of they are not very friendly. For example, I do not want to have tests that allocate gigabytes of memory running on my laptop because of it would make it impossible to do anything else on it.

• The failure above was likely caused by multiple tests that allocate large block of memory running in parallel (there are multiple such tests in Span), the machine was really short on memory because of that, and we failed in a random place on OOM. I can think of following mitigations (from the most preffered to least preffered):

  • Use memory mapped files from TEMP to create the large memory sections. I assume that the memory mapped file does not count against the quota.
  • Move tests that allocate large memory blocks in separate project and make that project run the tests sequentially by using [assembly: CollectionBehavior(CollectionBehavior.CollectionPerAssembly)] XUnit attribute.
  • Some other scheme to coordinate between tests that allocate large amount of memory.

[KristinXie1]

Failed again on build 20170310.01, https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20170310.01/workItem/System.Memory.Tests/analysis/xunit/System.SpanTests.ReadOnlySpanTests~2FIndexOverflow

[KristinXie1]

Failed again on 20170313.01

[KrzysztofCwalina]

I talked to @ahsonkhan about this today. He will take a look at fixing it withing the next 2 weeks or so.

[karelz]

@KrzysztofCwalina @ahsonkhan can you please disable the test right away? We need to decrease noise in CoreFX. Failures like this one impact also per-PR CI runs. Thanks!
All test failures should be treated as P0.5 ...

[ahsonkhan]

@karelz, failing tests disabled in PR dotnet/corefx#17077

[shiftylogic]

• Is there a swap partition enabled? _Note that by default, most cloud linux VMs do not enable swap partitions which means there is no way to page out of physical memory._
• How much physical RAM is on the machine.
• Unless it has changed, Linux does deferred page allocation. So, you don't see an OOM for physical memory limits until you start touching the virtual pages. If you have a low physical memory system with no swap partition (or one that is too small for what you need), what we see above is exactly what would happen.

[shiftylogic]

I'm re-enabling these tests for Windows and OSX and leaving them disabled for Linux (PR dotnet/corefx#17441). This exact issue is likely caused by a low memory scenario combined with Linux's deferred allocations. In low memory scenarios on other platforms, we either get the page file or instance OOM that is caught by the exception handler.

I also added logic so that the test will not run on 32-bit systems since it will certainly fail a 3GB allocation.

We will monitor outer loop builds and disable the test again if the issue returns.

[shiftylogic]

FYI, we (currently) decided against using a memory mapped file to back this allocation. Given that we will be writing data to the entire buffer twice (x2 per run of the tests), writing to a file would be both very slow and also not good for the SSD drives used by the test machines.