Runtime: assert `Failed to SetThreadContext in RedirectThreadAtHandledJITCase - aborting redirect.`
I was trying to write a repro case for #13041 but hit another VM assert:
Assert failure(PID 105560 [0x00019c58], Thread: 99252 [0x183b4]): bRes && "Failed to SetThreadContext in RedirectThreadAtHandledJITCase - aborting redirect."
The test runs many threads with stack-allocated buffer and there is a separate thread that is calling GC.Collect, I do not think it does anything illegal.
Test code
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
using System;
using System.Numerics;
using System.Runtime.CompilerServices;
using System.Diagnostics;
using System.Runtime.Intrinsics;
using System.Security.Cryptography;
using System.Threading;
class Runtime_13041
{
struct s8
{
long f;
}
struct s32
{
s8 f1;
s8 f2;
s8 f3;
s8 f4;
}
struct s128
{
s32 f1;
s32 f2;
s32 f3;
s32 f4;
}
struct s512
{
s128 f1;
s128 f2;
s128 f3;
s128 f4;
}
static int StrangeMethod(int length, object a, object b, object c, object d)
{
s512 s = new s512();
string e = new string("abe");
string f = new string("abf");
string g = new string("abg");
string h = new string("abh");
if (length == 1000)
{
return 1000 + e[0] + f[0] + g[0] + h[0];
}
Span<int> numbers = stackalloc int[length];
int sum = 0;
for (int i = 0; i < length; ++i)
{
numbers[i] = i;
}
for (int i = 0; i < length; ++i)
{
sum += numbers[i];
if (sum == 4000)
{
return 4000;
}
}
return sum;
}
static void CallGC()
{
while (true)
{
System.GC.Collect();
}
}
public static int Main()
{
Thread gcThread = new Thread(() => CallGC());
gcThread.Start();
const int iterCount = 1000;
for (int iter = 0; iter < iterCount; ++iter)
{
int sum = 0;
const int threadCount = 128;
Thread[] threads = new Thread[threadCount];
int[] results = new int[threadCount];
for (int i = 0; i < threadCount; ++i)
{
int threadId = i;
int length = threadCount * i + iter + 1;
threads[threadId] = new Thread(() => { results[threadId] = StrangeMethod(length, null, null, null, null); });
threads[threadId].Start();
}
for (int i = 0; i < threadCount; ++i)
{
threads[i].Join();
sum += results[i];
}
Console.WriteLine("Finished an iteration, the sum is: " + sum);
}
return 100;
}
}
it fails with the same error with a ~30% repro rate in a ~2-minute run, I could hit it in VS and collect info that is necessary.
Windows x64 last update, (amd ryzen), checked CoreCLR, release libraries, compiled with \git\runtime\.dotnet\dotnet.exe msbuild /p:TargetArchitecture=x64 /p:Configuration=Checked /p:LibrariesConfiguration=Release.
stack in the thread where the assert happens
KernelBase.dll!wil::details::DebugBreak(void) Unknown
coreclr.dll!DbgAssertDialog(const char * szFile, int iLine, const char * szExpr) Line 700 C++
> coreclr.dll!Thread::RedirectThreadAtHandledJITCase(void(*)() pTgt) Line 3200 C++
coreclr.dll!Thread::CheckForAndDoRedirect(void(*)() pRedirectTarget) Line 3228 C++
coreclr.dll!Thread::CheckForAndDoRedirectForGC() Line 3335 C++
coreclr.dll!ThreadSuspend::SuspendRuntime(ThreadSuspend::SUSPEND_REASON reason) Line 3982 C++
coreclr.dll!ThreadSuspend::SuspendEE(ThreadSuspend::SUSPEND_REASON reason) Line 6137 C++
coreclr.dll!GCToEEInterface::SuspendEE(SUSPEND_REASON reason) Line 28 C++
coreclr.dll!WKS::GCHeap::GarbageCollectGeneration(unsigned int gen, gc_reason reason) Line 37395 C++
coreclr.dll!WKS::GCHeap::GarbageCollect(int generation, bool low_memory_p, int mode) Line 36619 C++
coreclr.dll!GCInterface::Collect(int generation, int mode) Line 1011 C++
00007ff9a3c6b53a() Unknown
System.Private.CoreLib.dll!00007ffa1c4a206a() Unknown
System.Private.CoreLib.dll!00007ffa1c4acba0() Unknown
System.Private.CoreLib.dll!00007ffa1c4a21fd() Unknown
coreclr.dll!CallDescrWorkerInternal() Line 100 Unknown
coreclr.dll!MethodDescCallSite::CallTargetWorker(const unsigned __int64 * pArguments, unsigned __int64 * pReturnValue, int cbReturnValue) Line 552 C++
coreclr.dll!ThreadNative::KickOffThread_Worker(void * ptr) Line 247 C++
coreclr.dll!ManagedThreadBase_DispatchInner(ManagedThreadCallState * pCallState) Line 7337 C++
coreclr.dll!ManagedThreadBase_DispatchMiddle(ManagedThreadCallState * pCallState) Line 7381 C++
coreclr.dll!``ManagedThreadBase_DispatchOuter'::`11'::__Body::Run'::`5'::__Body::Run(Param * pParam) Line 7540 C++
coreclr.dll!`ManagedThreadBase_DispatchOuter'::`11'::__Body::Run(ManagedThreadBase_DispatchOuter::__l2::TryArgs * pArgs) Line 7542 C++
coreclr.dll!ManagedThreadBase_DispatchOuter(ManagedThreadCallState * pCallState) Line 7564 C++
coreclr.dll!ManagedThreadBase_FullTransition(void(*)(void *) pTarget, void * args, UnhandledExceptionLocation filterType) Line 7588 C++
coreclr.dll!ThreadNative::KickOffThread(void * pass) Line 327 C++
kernel32.dll!BaseThreadInitThunk
() Unknown
ntdll.dll!RtlUserThreadStart
() Unknown
threads
Not Flagged 110804 0 Worker Thread coreclr.dll!DebuggerRCThread::ThreadProcStatic coreclr.dll!DebuggerRCThread::MainLoop
Not Flagged 57628 0 Main Thread Main Thread coreclr.dll!CrstBase::Enter
Not Flagged 99244 0 Worker Thread ntdll.dll!TppWorkerThread ntdll.dll!NtWaitForWorkViaWorkerFactory
Not Flagged 88748 0 Worker Thread ntdll.dll!TppWorkerThread ntdll.dll!NtWaitForWorkViaWorkerFactory
Not Flagged 112384 0 Worker Thread ntdll.dll!TppWorkerThread ntdll.dll!NtWaitForWorkViaWorkerFactory
Not Flagged 38804 0 Worker Thread coreclr.dll!DiagnosticServer::DiagnosticsServerThread coreclr.dll!IpcStream::DiagnosticsIpc::Poll
Not Flagged 52320 0 Worker Thread .NET Finalizer [Inline Frame] coreclr.dll!CLREventWaitHelper2
Not Flagged > 99252 0 Worker Thread coreclr.dll!ThreadNative::KickOffThread coreclr.dll!DbgAssertDialog
Not Flagged 26848 0 Worker Thread coreclr.dll!ThreadNative::KickOffThread [Inline Frame] coreclr.dll!CLREventWaitHelper2
Not Flagged 12148 0 Worker Thread coreclr.dll!ThreadNative::KickOffThread [Inline Frame] coreclr.dll!CLREventWaitHelper2
Not Flagged 111308 0 Worker Thread coreclr.dll!ThreadNative::KickOffThread [Inline Frame] coreclr.dll!CLREventWaitHelper2
Not Flagged 10540 0 Worker Thread coreclr.dll!ThreadNative::KickOffThread 00007ff9a3c6bb7d
sandreenko
All 16 comments
PTAL @jkotas @janvorli
sandreenko
on 11 Aug 2020
It does not repro for me.
This assert suggests that we are hitting Windows OS bug. @mangod9 @jeffschwMSFT
Can you check the state of the thread that we are trying to call SetThreadContext on and see whether there is any pattern betweeen multiple hits? E.g. does it only happen when the target thread is executing specific instructions?
jkotas
on 11 Aug 2020
@sandreenko perhaps you can share a dump since you can repro more consistently?
mangod9
on 11 Aug 2020
@sandreenko perhaps you can share a dump since you can repro more consistently?
Sure, I can't attach it here because it is too big so I placed it in OneDrive, please tell me if it is not working. MinDumpWithHeap(VS)+symbols
Can you check the state of the thread that we are trying to call SetThreadContext on and see whether there is any pattern between multiple hits? E.g. does it only happen when the target thread is executing specific instructions?
I have checked 5 hits and they were interrupted on the different instructions in StrangeMethod.
hit 1:
this->m_OSThreadId: 10540
asm for that thread:
00007FF9A3C6BB76 movsxd r8,ecx
00007FF9A3C6BB79 mov dword ptr [rax+r8*4],ecx
-> 00007FF9A3C6BB7D inc ecx
00007FF9A3C6BB7F cmp ecx,esi
00007FF9A3C6BB81 jl 00007FF9A3C6BB76
00007ff9a3c6bb7d() Unknown
00007ff9a3c6b496() Unknown
System.Private.CoreLib.dll!00007ffa1c4a206a() Unknown
System.Private.CoreLib.dll!00007ffa1c4acba0() Unknown
System.Private.CoreLib.dll!00007ffa1c4a21fd() Unknown
coreclr.dll!CallDescrWorkerInternal() Line 100 Unknown
coreclr.dll!MethodDescCallSite::CallTargetWorker(const unsigned __int64 * pArguments, unsigned __int64 * pReturnValue, int cbReturnValue) Line 552 C++
coreclr.dll!ThreadNative::KickOffThread_Worker(void * ptr) Line 247 C++
hit 2:
this->m_OSThreadId 23548
00007FFD6092BB76 movsxd r8,ecx
00007FFD6092BB79 mov dword ptr [rax+r8*4],ecx
-> 00007FFD6092BB7D inc ecx
00007FFD6092BB7F cmp ecx,esi
00007FFD6092BB81 jl 00007FFD6092BB76
00007ffd6092bb7d() Unknown
00007ffd6092b496() Unknown
System.Private.CoreLib.dll!00007ffdbf6d206a() Unknown
System.Private.CoreLib.dll!00007ffdbf6dcba0() Unknown
System.Private.CoreLib.dll!00007ffdbf6d21fd() Unknown
hit 3:
this->m_OSThreadId 22716 unsigned __int64
00007FFD7327BB8C add edx,dword ptr [rax+r8*4]
00007FFD7327BB90 cmp edx,0FA0h
-> 00007FFD7327BB96 je 00007FFD7327BBC1
00007FFD7327BB98 inc ecx
00007FFD7327BB9A cmp ecx,esi
00007ffd7327bb96() Unknown
00007ffd7327b496() Unknown
System.Private.CoreLib.dll!00007ffdd187206a() Unknown
System.Private.CoreLib.dll!00007ffdd187cba0() Unknown
System.Private.CoreLib.dll!00007ffdd18721fd() Unknown
sandreenko
on 11 Aug 2020
I cant repro the issue either. Like Jan mentions it appears that windows call to SetThreadContext is failing, perhaps it could log GetLastError to diagnose why its failing.
mangod9
on 12 Aug 2020
GetLastError value is not recognized, checked that I can repro it on my second machine, so it is not something wrong with my main one, branch that I am using (https://github.com/dotnet/runtime/compare/master...sandreenko:GScookieExp)
sandreenko
on 12 Aug 2020
Could please try to reproduce it with https://github.com/jkotas/runtime/commit/292f45b664345b2aa29a04d9e23e60bc4e754bc3 and capture the dump? It will allow us to get the last error.
jkotas
on 12 Aug 2020
Could please try to reproduce it with jkotas@292f45b and capture the dump? It will allow us to get the last error.
Sure, the value is 87(dec)
# for decimal 87 / hex 0x57
XNS_INTERNAL_ERROR bugcodes.h
NMERR_FRAME_HAS_NO_CAPTURE netmon.h
ERROR_INVALID_PARAMETER winerror.h
# The parameter is incorrect.
LDAP_FILTER_ERROR winldap.h
Weird that its ERROR_INVALID_PARAMETER. Looking at the previous dump you shared the handle and context appear to be valid. The thread is suspended too:
0:007> ~
0 Id: 69f0.58b0 Suspend: 0 Teb: 000000d7`c94e0000 Unfrozen
1 Id: 69f0.7b5c Suspend: 0 Teb: 000000d7`c94e2000 Unfrozen
2 Id: 69f0.6524 Suspend: 0 Teb: 000000d7`c94e4000 Unfrozen
3 Id: 69f0.707c Suspend: 0 Teb: 000000d7`c94e6000 Unfrozen
4 Id: 69f0.6ad8 Suspend: 0 Teb: 000000d7`c94e8000 Unfrozen
5 Id: 69f0.4aa0 Suspend: 0 Teb: 000000d7`c94ea000 Unfrozen
6 Id: 69f0.24e8 Suspend: 0 Teb: 000000d7`c94ec000 Unfrozen ""
. 7 Id: 69f0.1120 Suspend: 0 Teb: 000000d7`c94ee000 Unfrozen
8 Id: 69f0.6718 Suspend: 0 Teb: 000000d7`c9492000 Unfrozen
**9 Id: 69f0.6724 Suspend: 1 Teb: 000000d7`c9496000 Unfrozen**
@sandreenko what version of Windows do you have? I think that I've seen Windows adding some extra sanity checks for the context relatively recently. E.g. the context flags are validated using RtlpSanitizeContextFlags and that one can return STATUS_INVALID_PARAMETER.
janvorli
on 12 Aug 2020
Looking at the pCtx->ContextFlags, its value is: 0xc010001b, so it has these extra flags set besides the regular ones:
define AMD64_CONTEXT_EXCEPTION_REQUEST 0x40000000L
define AMD64_CONTEXT_EXCEPTION_REPORTING 0x80000000L
I can see that the context flag validation fails if the AMD64_CONTEXT_EXCEPTION_REPORTING is set on the context.
janvorli
on 12 Aug 2020
I can see that the context flag validation fails if the AMD64_CONTEXT_EXCEPTION_REPORTING is set on the context.
It seems to work fine most of the time. It fails only once in a while.
I was able to reproduce this as well after leaving the repro run overnight. I have regular fully patched Windows version. kernelbase.dll version is 10.0.19041.423.
jkotas
on 12 Aug 2020
@janvorli discovered that this may be caused by using SP that points to uncommitted stack page.
Here is a more reliable repro:
using System;
using System.Threading;
using System.Runtime.CompilerServices;
class Program
{
static volatile bool never;
static int stackAlloc = 10000;
[MethodImplAttribute(MethodImplOptions.NoInlining)]
static void Dummy(long a0, long a1, long a2, long a3, long a4, long a5, long a6, long a7, long a8, long a9,
long a10, long a11, long a12, long a13, long a14, long a15, long a16, long a17, long a18, long a19,
long a20, long a21, long a22, long a23, long a24, long a25, long a26, long a27, long a28, long a29,
long a30, long a31)
{
}
static int Loop(int count)
{
int fact = 0;
for (int i = 0; i < count; i++) fact *= i;
if (never) Dummy(0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0);
return fact;
}
static void Work()
{
Span<byte> s = stackalloc byte[Interlocked.Add(ref stackAlloc, 16*8)];
s[0] = 42;
Loop(1000000000);
new Thread(Work).Start();
}
public static int Main()
{
new Thread(() => { for (;;) { GC.Collect(); Thread.Sleep(1); } });
for (int i = 0; i < 100; i++)
new Thread(Work).Start();
Work();
return 100;
}
}
So this doesnt appear to be a .net regression, but something which needs to be fixed due to stricter validation from windows?
mangod9
on 12 Aug 2020
This issue will be fixed in future Windows releases.
jkotas
on 13 Aug 2020
For internal reference, the Windows OS fix is PR #5041554 in os.2020 collection. "Rtl: Accept stack pointers in the guard page range.".
jkotas
on 13 Aug 2020
Related issues
GitAntoinee
路
3Comments
aggieben
路
3Comments
jzabroski
路
3Comments
matty-hall
路
3Comments
EgorBo
路
3Comments
Most helpful comment
@janvorli discovered that this may be caused by using SP that points to uncommitted stack page.
Here is a more reliable repro: