Mono: [netcore] Investigate CI failures related to SafeHandle
Quite often the CI jobs fail on the following error:
System.ObjectDisposedException : Safe handle has been closed.\nObject name: 'SafeHandle'.
Stack trace
at System.Runtime.InteropServices.SafeHandle.InternalRelease(Boolean disposeOrFinalizeOperation)
at System.Runtime.InteropServices.SafeHandle.DangerousRelease()
at Internal.Cryptography.HashProviderDispenser.HmacHashProvider.TryFinalizeHashAndReset(Span`1 destination, Int32& bytesWritten)
at Internal.Cryptography.HMACCommon.TryFinalizeHashAndReset(Span`1 destination, Int32& bytesWritten)
at System.Security.Cryptography.HMACSHA256.TryHashFinal(Span`1 destination, Int32& bytesWritten)
at System.Security.Cryptography.HashAlgorithm.TryComputeHash(ReadOnlySpan`1 source, Span`1 destination, Int32& bytesWritten)
at System.Security.Cryptography.Rfc2898DeriveBytes.Func()
at System.Security.Cryptography.Rfc2898DeriveBytes.GetBytes(Int32 cb)
at System.Security.Cryptography.PasswordBasedEncryption.Encrypt(ReadOnlySpan`1 password, ReadOnlySpan`1 passwordBytes, SymmetricAlgorithm cipher, Boolean isPkcs12, ReadOnlySpan`1 source, PbeParameters pbeParameters, ReadOnlySpan`1 salt, Byte[] destination, Span`1 ivDest)
at System.Security.Cryptography.KeyFormatHelper.WriteEncryptedPkcs8(ReadOnlySpan`1 password, ReadOnlySpan`1 passwordBytes, AsnWriter pkcs8Writer, PbeParameters pbeParameters)
at System.Security.Cryptography.KeyFormatHelper.WriteEncryptedPkcs8(ReadOnlySpan`1 passwordBytes, AsnWriter pkcs8Writer, PbeParameters pbeParameters)
at System.Security.Cryptography.DSA.TryExportEncryptedPkcs8PrivateKey(ReadOnlySpan`1 passwordBytes, PbeParameters pbeParameters, Span`1 destination, Int32& bytesWritten)
at System.Security.Cryptography.Dsa.Tests.DSAKeyFileTests.<>c__DisplayClass23_0.<ReadBase64EncryptedPkcs8>b__2(DSA dsa, Span`1 destination, Int32& written)
at System.Security.Cryptography.Dsa.Tests.DSAKeyFileTests.ReadWriteKey(String base64, DSAParameters& expected, ReadKeyAction readAction, Func`2 writeArrayFunc, WriteKeyToSpanFunc writeSpanFunc, Boolean isEncrypted)
at System.Security.Cryptography.Dsa.Tests.DSAKeyFileTests.ReadBase64EncryptedPkcs8(String base64EncPkcs8, Byte[] passwordBytes, PbeParameters pbeParameters, DSAParameters& expected)
at System.Security.Cryptography.Dsa.Tests.DSAKeyFileTests.ReadWriteDsa1024EncryptedPkcs8_PasswordBytes()
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
There's a history on the Azure Pipelines that shows it's occurring quite often.
Note that there is no explicit call to SafeHandle.DangerousRelease in the given code path and it's likely coming from the P/Invoke marshalling code. CoreCLR silently ignores exceptions from DangerousRelease in marshalling (https://github.com/dotnet/coreclr/blob/7adc7249f71679b6a1727e17cd15df4db5d621b3/src/System.Private.CoreLib/src/System/StubHelpers.cs#L1646-L1652).
@jkotas, is there any rationale for ignoring the exceptions from SafeHandle.DangerousRelease in the marshalling code? It seems it could hide legitimate bugs or race conditions. Unfortunately the code predates the history on GitHub.
GC
filipnavara
All 29 comments
I think @baulig made some in this space but not sure if they were ported to netcore
marek-safar
on 16 Sep 2019
is there any rationale for ignoring the exceptions from SafeHandle.DangerousRelease in the marshalling code
It was the behavior 15 years ago when it was introduced. I agree that it is less than ideal behavior. I have opened https://github.com/dotnet/coreclr/issues/26730 to have this tracked.
jkotas
on 16 Sep 2019
Thanks! I will try to do a CI run on CoreCLR with the catch removed to see if the same CoreFX test(s) would fail there too.
filipnavara
on 16 Sep 2019
These errors keep happening on the CI quite regularly. I still didn't manage to reproduce them locally but in all cases the affected code path ended up in HashProviderDispenser.
filipnavara
on 19 Sep 2019
I tried various combinations of Debug/Release builds, OpenSSL 1.0.2n/1.1 on Ubuntu 18.04, each of them for 100 consecutive runs. It never reproduced for me locally.
filipnavara
on 19 Sep 2019
👍1
Thanks a lot for investigating this. I've also seen it happen on CI but was never able to reproduce it locally.
baulig
on 19 Sep 2019
I finally managed to reproduce it on stock Ubuntu 16.04 running under WSL2. On fourth run two of the tests failed.
Stack trace with line numbers:
System.Security.Cryptography.Rsa.Tests.RSAKeyFileTests.ReadEncryptedRsa1032 [FAIL]
System.ObjectDisposedException : Safe handle has been closed.
Object name: 'SafeHandle'.
Stack Trace:
/home/filipnavara/mono/netcore/System.Private.CoreLib/shared/System/Runtime/InteropServices/SafeHandle.cs(252,0): at System.Runtime.InteropServices.SafeHandle.InternalRelease(Boolean disposeOrFinalizeOperation)
/home/filipnavara/mono/netcore/System.Private.CoreLib/shared/System/Runtime/InteropServices/SafeHandle.cs(173,0): at System.Runtime.InteropServices.SafeHandle.DangerousRelease()
/_/src/System.Security.Cryptography.Algorithms/src/Internal/Cryptography/HashProviderDispenser.Unix.cs(160,0): at Internal.Cryptography.HashProviderDispenser.HmacHashProvider.TryFinalizeHashAndReset(Span`1 destination, Int32& bytesWritten)
/_/src/System.Security.Cryptography.Algorithms/src/Internal/Cryptography/HMACCommon.cs(71,0): at Internal.Cryptography.HMACCommon.TryFinalizeHashAndReset(Span`1 destination, Int32& bytesWritten)
/_/src/System.Security.Cryptography.Algorithms/src/System/Security/Cryptography/HMACSHA384.cs(75,0): at System.Security.Cryptography.HMACSHA384.TryHashFinal(Span`1 destination, Int32& bytesWritten)
/_/src/System.Security.Cryptography.Primitives/src/System/Security/Cryptography/HashAlgorithm.cs(65,0): at System.Security.Cryptography.HashAlgorithm.TryComputeHash(ReadOnlySpan`1 source, Span`1 destination, Int32& bytesWritten)
/_/src/System.Security.Cryptography.Algorithms/src/System/Security/Cryptography/Rfc2898DeriveBytes.cs(280,0): at System.Security.Cryptography.Rfc2898DeriveBytes.Func()
/_/src/System.Security.Cryptography.Algorithms/src/System/Security/Cryptography/Rfc2898DeriveBytes.cs(191,0): at System.Security.Cryptography.Rfc2898DeriveBytes.GetBytes(Int32 cb)
/_/src/Common/src/System/Security/Cryptography/PasswordBasedEncryption.cs(379,0): at System.Security.Cryptography.PasswordBasedEncryption.Encrypt(ReadOnlySpan`1 password, ReadOnlySpan`1 passwordBytes, SymmetricAlgorithm cipher, Boolean isPkcs12, ReadOnlySpan`1 source, PbeParameters pbeParameters, ReadOnlySpan`1 salt, Byte[] destination, Span`1 ivDest)
/_/src/Common/src/System/Security/Cryptography/KeyFormatHelper.cs(348,0): at System.Security.Cryptography.KeyFormatHelper.WriteEncryptedPkcs8(ReadOnlySpan`1 password, ReadOnlySpan`1 passwordBytes, AsnWriter pkcs8Writer, PbeParameters pbeParameters)
/_/src/Common/src/System/Security/Cryptography/KeyFormatHelper.cs(298,0): at System.Security.Cryptography.KeyFormatHelper.WriteEncryptedPkcs8(ReadOnlySpan`1 password, AsnWriter pkcs8Writer, PbeParameters pbeParameters)
/_/src/System.Security.Cryptography.Algorithms/src/System/Security/Cryptography/RSA.cs(418,0): at System.Security.Cryptography.RSA.TryExportEncryptedPkcs8PrivateKey(ReadOnlySpan`1 password, PbeParameters pbeParameters, Span`1 destination, Int32& bytesWritten)
/_/src/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/RSAKeyFileTests.cs(1265,0): at System.Security.Cryptography.Rsa.Tests.RSAKeyFileTests.<>c__DisplayClass31_0.<ReadBase64EncryptedPkcs8>b__2(RSA rsa, Span`1 destination, Int32& written)
/_/src/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/RSAKeyFileTests.cs(1420,0): at System.Security.Cryptography.Rsa.Tests.RSAKeyFileTests.ReadWriteKey(String base64, RSAParameters& expected, ReadKeyAction readAction, Func`2 writeArrayFunc, WriteKeyToSpanFunc writeSpanFunc, Boolean isEncrypted)
/_/src/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/RSAKeyFileTests.cs(1258,0): at System.Security.Cryptography.Rsa.Tests.RSAKeyFileTests.ReadBase64EncryptedPkcs8(String base64EncPkcs8, String password, PbeParameters pbeParameters, RSAParameters& expected)
/_/src/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/RSAKeyFileTests.cs(516,0): at System.Security.Cryptography.Rsa.Tests.RSAKeyFileTests.ReadEncryptedRsa1032()
/home/filipnavara/mono/netcore/System.Private.CoreLib/src/System.Reflection/RuntimeMethodInfo.cs(339,0): at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
I've got two native crashes too. For the first one I didn't have a debugger installed. For the second one I got this:
System.Security.Cryptography.Encryption.Aes.Tests.AesContractTests.InvalidIVSizes(invalidIvSize: 536870928, skipOnNetfx: True) [STARTING]
=================================================================
Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================
=================================================================
Native stacktrace:
=================================================================
0x7f0f62b1bd73 - /home/filipnavara/mono/netcore/shared/Microsoft.NETCore.App/5.0.0-alpha1.19452.1/libcoreclr.so : (null)
0x7f0f62b1c0ec - /home/filipnavara/mono/netcore/shared/Microsoft.NETCore.App/5.0.0-alpha1.19452.1/libcoreclr.so : (null)
0x7f0f62aadbd0 - /home/filipnavara/mono/netcore/shared/Microsoft.NETCore.App/5.0.0-alpha1.19452.1/libcoreclr.so : (null)
0x7f0f62a28c9e - /home/filipnavara/mono/netcore/shared/Microsoft.NETCore.App/5.0.0-alpha1.19452.1/libcoreclr.so : (null)
0x7f0f6430d390 - /lib/x86_64-linux-gnu/libpthread.so.0 : (null)
0x7f0f62c7c18a - /home/filipnavara/mono/netcore/shared/Microsoft.NETCore.App/5.0.0-alpha1.19452.1/libcoreclr.so : (null)
0x7f0f62c7ccc5 - /home/filipnavara/mono/netcore/shared/Microsoft.NETCore.App/5.0.0-alpha1.19452.1/libcoreclr.so : (null)
0x7f0f62c7dc2b - /home/filipnavara/mono/netcore/shared/Microsoft.NETCore.App/5.0.0-alpha1.19452.1/libcoreclr.so : (null)
0x7f0f62c81980 - /home/filipnavara/mono/netcore/shared/Microsoft.NETCore.App/5.0.0-alpha1.19452.1/libcoreclr.so : (null)
0x7f0f62c81b7b - /home/filipnavara/mono/netcore/shared/Microsoft.NETCore.App/5.0.0-alpha1.19452.1/libcoreclr.so : (null)
0x7f0f62c84ab9 - /home/filipnavara/mono/netcore/shared/Microsoft.NETCore.App/5.0.0-alpha1.19452.1/libcoreclr.so : (null)
0x7f0f62c71969 - /home/filipnavara/mono/netcore/shared/Microsoft.NETCore.App/5.0.0-alpha1.19452.1/libcoreclr.so : (null)
0x7f0f62c61efc - /home/filipnavara/mono/netcore/shared/Microsoft.NETCore.App/5.0.0-alpha1.19452.1/libcoreclr.so : (null)
0x7f0f62bfb31b - /home/filipnavara/mono/netcore/shared/Microsoft.NETCore.App/5.0.0-alpha1.19452.1/libcoreclr.so : (null)
0x7f0f62c23799 - /home/filipnavara/mono/netcore/shared/Microsoft.NETCore.App/5.0.0-alpha1.19452.1/libcoreclr.so : (null)
0x7f0f62bfd7de - /home/filipnavara/mono/netcore/shared/Microsoft.NETCore.App/5.0.0-alpha1.19452.1/libcoreclr.so : (null)
0x7f0f62bfd89f - /home/filipnavara/mono/netcore/shared/Microsoft.NETCore.App/5.0.0-alpha1.19452.1/libcoreclr.so : (null)
0x7f0f62bac551 - /home/filipnavara/mono/netcore/shared/Microsoft.NETCore.App/5.0.0-alpha1.19452.1/libcoreclr.so : (null)
0x4056fe38 - Unknown
=================================================================
Telemetry Dumper:
=================================================================
Pkilling 0x7f0f58d21700 from 0x7f0f531b5700
Pkilling 0x7f0f527ff700 from 0x7f0f531b5700
Pkilling 0x7f0f523fd700 from 0x7f0f531b5700
Pkilling 0x7f0f51ffb700 from 0x7f0f531b5700
Pkilling 0x7f0f53bba700 from 0x7f0f531b5700
Pkilling 0x7f0f537b8700 from 0x7f0f531b5700
Pkilling 0x7f0f533b6700 from 0x7f0f531b5700
Pkilling 0x7f0f606e2700 from 0x7f0f531b5700
Pkilling 0x7f0f525fe700 from 0x7f0f531b5700
Pkilling 0x7f0f521fc700 from 0x7f0f531b5700
Pkilling 0x7f0f58b1c700 from 0x7f0f531b5700
Pkilling 0x7f0f64731740 from 0x7f0f531b5700
Pkilling 0x7f0f539b9700 from 0x7f0f531b5700
Pkilling 0x7f0f535b7700 from 0x7f0f531b5700
Could not exec mono-hang-watchdog, expected on path '/usr/local/etc/../bin/mono-hang-watchdog' (errno 2)
Entering thread summarizer pause from 0x7f0f531b5700
Finished thread summarizer pause from 0x7f0f531b5700.
Waiting for dumping threads to resume
=================================================================
External Debugger Dump:
=================================================================
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib/python2.7/dist-packages/lldb/__init__.py", line 123, in <module>
import six
ImportError: No module named six
(lldb) command source -s 0 '/tmp/mono-gdb-commands.5867'
Executing commands in '/tmp/mono-gdb-commands.5867'.
(lldb) process attach --pid 5867
Process 5867 stopped
* thread #1: tid = 5867, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'dotnet', stop reason = signal SIGSTOP
frame #0: 0x00007f0f634c2826 libc.so.6`sigsuspend + 86
libc.so.6`sigsuspend:
-> 0x7f0f634c2826 <+86>: cmpq $-0x1000, %rax ; imm = 0xFFFFFFFFFFFFF000
0x7f0f634c282c <+92>: ja 0x7f0f634c2843 ; <+115>
0x7f0f634c282e <+94>: movl %edx, %edi
0x7f0f634c2830 <+96>: movl %eax, 0xc(%rsp)
thread #2: tid = 5868, 0x00007f0f64309360 libpthread.so.0`__pthread_cond_wait + 192, name = 'SGen worker', stop reason = signal SIGSTOP
frame #0: 0x00007f0f64309360 libpthread.so.0`__pthread_cond_wait + 192
libpthread.so.0`__pthread_cond_wait:
-> 0x7f0f64309360 <+192>: movl (%rsp), %edi
0x7f0f64309363 <+195>: callq 0x7f0f6430c1c0 ; __pthread_disable_asynccancel
0x7f0f64309368 <+200>: movq 0x8(%rsp), %rdi
0x7f0f6430936d <+205>: movl $0x1, %esi
thread #3: tid = 5869, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'Finalizer', stop reason = signal SIGSTOP
frame #0: 0x00007f0f634c2826 libc.so.6`sigsuspend + 86
libc.so.6`sigsuspend:
-> 0x7f0f634c2826 <+86>: cmpq $-0x1000, %rax ; imm = 0xFFFFFFFFFFFFF000
0x7f0f634c282c <+92>: ja 0x7f0f634c2843 ; <+115>
0x7f0f634c282e <+94>: movl %edx, %edi
0x7f0f634c2830 <+96>: movl %eax, 0xc(%rsp)
thread #4: tid = 5870, 0x00007f0f6430b827 libpthread.so.0`do_futex_wait.constprop.3 + 55, name = 'dotnet', stop reason = signal SIGSTOP
frame #0: 0x00007f0f6430b827 libpthread.so.0`do_futex_wait.constprop.3 + 55
libpthread.so.0`do_futex_wait.constprop.3:
-> 0x7f0f6430b827 <+55>: cmpq $-0x1000, %rax ; imm = 0xFFFFFFFFFFFFF000
0x7f0f6430b82d <+61>: ja 0x7f0f6430b848 ; <+88>
0x7f0f6430b82f <+63>: movl %r12d, %edi
0x7f0f6430b832 <+66>: callq 0x7f0f6430c1c0 ; __pthread_disable_asynccancel
thread #5: tid = 5873, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'dotnet', stop reason = signal SIGSTOP
frame #0: 0x00007f0f634c2826 libc.so.6`sigsuspend + 86
libc.so.6`sigsuspend:
-> 0x7f0f634c2826 <+86>: cmpq $-0x1000, %rax ; imm = 0xFFFFFFFFFFFFF000
0x7f0f634c282c <+92>: ja 0x7f0f634c2843 ; <+115>
0x7f0f634c282e <+94>: movl %edx, %edi
0x7f0f634c2830 <+96>: movl %eax, 0xc(%rsp)
thread #6: tid = 5874, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'dotnet', stop reason = signal SIGSTOP
frame #0: 0x00007f0f634c2826 libc.so.6`sigsuspend + 86
libc.so.6`sigsuspend:
-> 0x7f0f634c2826 <+86>: cmpq $-0x1000, %rax ; imm = 0xFFFFFFFFFFFFF000
0x7f0f634c282c <+92>: ja 0x7f0f634c2843 ; <+115>
0x7f0f634c282e <+94>: movl %edx, %edi
0x7f0f634c2830 <+96>: movl %eax, 0xc(%rsp)
thread #7: tid = 5875, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'Thread Pool Wor', stop reason = signal SIGSTOP
frame #0: 0x00007f0f634c2826 libc.so.6`sigsuspend + 86
libc.so.6`sigsuspend:
-> 0x7f0f634c2826 <+86>: cmpq $-0x1000, %rax ; imm = 0xFFFFFFFFFFFFF000
0x7f0f634c282c <+92>: ja 0x7f0f634c2843 ; <+115>
0x7f0f634c282e <+94>: movl %edx, %edi
0x7f0f634c2830 <+96>: movl %eax, 0xc(%rsp)
thread #8: tid = 5876, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'Thread Pool Wor', stop reason = signal SIGSTOP
frame #0: 0x00007f0f634c2826 libc.so.6`sigsuspend + 86
libc.so.6`sigsuspend:
-> 0x7f0f634c2826 <+86>: cmpq $-0x1000, %rax ; imm = 0xFFFFFFFFFFFFF000
0x7f0f634c282c <+92>: ja 0x7f0f634c2843 ; <+115>
0x7f0f634c282e <+94>: movl %edx, %edi
0x7f0f634c2830 <+96>: movl %eax, 0xc(%rsp)
thread #9: tid = 5877, 0x00007f0f6430b827 libpthread.so.0`do_futex_wait.constprop.3 + 55, name = 'dotnet', stop reason = signal SIGSTOP
frame #0: 0x00007f0f6430b827 libpthread.so.0`do_futex_wait.constprop.3 + 55
libpthread.so.0`do_futex_wait.constprop.3:
-> 0x7f0f6430b827 <+55>: cmpq $-0x1000, %rax ; imm = 0xFFFFFFFFFFFFF000
0x7f0f6430b82d <+61>: ja 0x7f0f6430b848 ; <+88>
0x7f0f6430b82f <+63>: movl %r12d, %edi
0x7f0f6430b832 <+66>: callq 0x7f0f6430c1c0 ; __pthread_disable_asynccancel
thread #10: tid = 5878, 0x00007f0f6430b827 libpthread.so.0`do_futex_wait.constprop.3 + 55, name = 'dotnet', stop reason = signal SIGSTOP
frame #0: 0x00007f0f6430b827 libpthread.so.0`do_futex_wait.constprop.3 + 55
libpthread.so.0`do_futex_wait.constprop.3:
-> 0x7f0f6430b827 <+55>: cmpq $-0x1000, %rax ; imm = 0xFFFFFFFFFFFFF000
0x7f0f6430b82d <+61>: ja 0x7f0f6430b848 ; <+88>
0x7f0f6430b82f <+63>: movl %r12d, %edi
0x7f0f6430b832 <+66>: callq 0x7f0f6430c1c0 ; __pthread_disable_asynccancel
thread #11: tid = 5879, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'dotnet', stop reason = signal SIGSTOP
frame #0: 0x00007f0f634c2826 libc.so.6`sigsuspend + 86
libc.so.6`sigsuspend:
-> 0x7f0f634c2826 <+86>: cmpq $-0x1000, %rax ; imm = 0xFFFFFFFFFFFFF000
0x7f0f634c282c <+92>: ja 0x7f0f634c2843 ; <+115>
0x7f0f634c282e <+94>: movl %edx, %edi
0x7f0f634c2830 <+96>: movl %eax, 0xc(%rsp)
thread #12: tid = 5880, 0x00007f0f6430cf7b libpthread.so.0`__waitpid + 107, name = 'dotnet', stop reason = signal SIGSTOP
frame #0: 0x00007f0f6430cf7b libpthread.so.0`__waitpid + 107
libpthread.so.0`__waitpid:
-> 0x7f0f6430cf7b <+107>: cmpq $-0x1000, %rax ; imm = 0xFFFFFFFFFFFFF000
0x7f0f6430cf81 <+113>: ja 0x7f0f6430cf9c ; <+140>
0x7f0f6430cf83 <+115>: movl %r8d, %edi
0x7f0f6430cf86 <+118>: movl %eax, 0xc(%rsp)
thread #13: tid = 5881, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'Thread Pool Wor', stop reason = signal SIGSTOP
frame #0: 0x00007f0f634c2826 libc.so.6`sigsuspend + 86
libc.so.6`sigsuspend:
-> 0x7f0f634c2826 <+86>: cmpq $-0x1000, %rax ; imm = 0xFFFFFFFFFFFFF000
0x7f0f634c282c <+92>: ja 0x7f0f634c2843 ; <+115>
0x7f0f634c282e <+94>: movl %edx, %edi
0x7f0f634c2830 <+96>: movl %eax, 0xc(%rsp)
thread #14: tid = 5882, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'Thread Pool Wor', stop reason = signal SIGSTOP
frame #0: 0x00007f0f634c2826 libc.so.6`sigsuspend + 86
libc.so.6`sigsuspend:
-> 0x7f0f634c2826 <+86>: cmpq $-0x1000, %rax ; imm = 0xFFFFFFFFFFFFF000
0x7f0f634c282c <+92>: ja 0x7f0f634c2843 ; <+115>
0x7f0f634c282e <+94>: movl %edx, %edi
0x7f0f634c2830 <+96>: movl %eax, 0xc(%rsp)
thread #15: tid = 5883, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'Thread Pool Wor', stop reason = signal SIGSTOP
frame #0: 0x00007f0f634c2826 libc.so.6`sigsuspend + 86
libc.so.6`sigsuspend:
-> 0x7f0f634c2826 <+86>: cmpq $-0x1000, %rax ; imm = 0xFFFFFFFFFFFFF000
0x7f0f634c282c <+92>: ja 0x7f0f634c2843 ; <+115>
0x7f0f634c282e <+94>: movl %edx, %edi
0x7f0f634c2830 <+96>: movl %eax, 0xc(%rsp)
thread #16: tid = 5884, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'Thread Pool Wor', stop reason = signal SIGSTOP
frame #0: 0x00007f0f634c2826 libc.so.6`sigsuspend + 86
libc.so.6`sigsuspend:
-> 0x7f0f634c2826 <+86>: cmpq $-0x1000, %rax ; imm = 0xFFFFFFFFFFFFF000
0x7f0f634c282c <+92>: ja 0x7f0f634c2843 ; <+115>
0x7f0f634c282e <+94>: movl %edx, %edi
0x7f0f634c2830 <+96>: movl %eax, 0xc(%rsp)
thread #17: tid = 5885, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'Thread Pool Wor', stop reason = signal SIGSTOP
frame #0: 0x00007f0f634c2826 libc.so.6`sigsuspend + 86
libc.so.6`sigsuspend:
-> 0x7f0f634c2826 <+86>: cmpq $-0x1000, %rax ; imm = 0xFFFFFFFFFFFFF000
0x7f0f634c282c <+92>: ja 0x7f0f634c2843 ; <+115>
0x7f0f634c282e <+94>: movl %edx, %edi
0x7f0f634c2830 <+96>: movl %eax, 0xc(%rsp)
Executable module set to "/home/filipnavara/mono/netcore/dotnet".
Architecture set to: x86_64-pc-linux.
(lldb) thread list
Process 5867 stopped
* thread #1: tid = 5867, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'dotnet', stop reason = signal SIGSTOP
thread #2: tid = 5868, 0x00007f0f64309360 libpthread.so.0`__pthread_cond_wait + 192, name = 'SGen worker', stop reason = signal SIGSTOP
thread #3: tid = 5869, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'Finalizer', stop reason = signal SIGSTOP
thread #4: tid = 5870, 0x00007f0f6430b827 libpthread.so.0`do_futex_wait.constprop.3 + 55, name = 'dotnet', stop reason = signal SIGSTOP
thread #5: tid = 5873, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'dotnet', stop reason = signal SIGSTOP
thread #6: tid = 5874, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'dotnet', stop reason = signal SIGSTOP
thread #7: tid = 5875, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'Thread Pool Wor', stop reason = signal SIGSTOP
thread #8: tid = 5876, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'Thread Pool Wor', stop reason = signal SIGSTOP
thread #9: tid = 5877, 0x00007f0f6430b827 libpthread.so.0`do_futex_wait.constprop.3 + 55, name = 'dotnet', stop reason = signal SIGSTOP
thread #10: tid = 5878, 0x00007f0f6430b827 libpthread.so.0`do_futex_wait.constprop.3 + 55, name = 'dotnet', stop reason = signal SIGSTOP
thread #11: tid = 5879, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'dotnet', stop reason = signal SIGSTOP
thread #12: tid = 5880, 0x00007f0f6430cf7b libpthread.so.0`__waitpid + 107, name = 'dotnet', stop reason = signal SIGSTOP
thread #13: tid = 5881, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'Thread Pool Wor', stop reason = signal SIGSTOP
thread #14: tid = 5882, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'Thread Pool Wor', stop reason = signal SIGSTOP
thread #15: tid = 5883, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'Thread Pool Wor', stop reason = signal SIGSTOP
thread #16: tid = 5884, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'Thread Pool Wor', stop reason = signal SIGSTOP
thread #17: tid = 5885, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'Thread Pool Wor', stop reason = signal SIGSTOP
(lldb) thread backtrace all
error: libcoreclr.so 0x004c0ee5: DW_TAG_variable has an invalid location:
DW_OP_const8u(0x0000000000000000)
DW_OP_lit28
error: libcoreclr.so 0x005ca957: DW_TAG_variable has an invalid location:
DW_OP_lit25
DW_OP_reg12
error: libcoreclr.so 0x005cad72: DW_TAG_variable has an invalid location:
DW_OP_reg20
DW_OP_reg12
* thread #1: tid = 5867, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'dotnet', stop reason = signal SIGSTOP
* frame #0: 0x00007f0f634c2826 libc.so.6`sigsuspend + 86
frame #1: 0x00007f0f62cc8eb7 libcoreclr.so`suspend_signal_handler(_dummy=<unavailable>, info=<unavailable>, context=0x00007ffdcfab9940) + 215 at mono-threads-posix-signals.c:200
frame #2: 0x00007f0f6430d390 libpthread.so.0`??? + 1
frame #3: 0x00007f0f6430935e libpthread.so.0`__pthread_cond_wait + 190
frame #4: 0x00007f0f62cbb065 libcoreclr.so`mono_os_cond_timedwait + 5 at mono-os-mutex.h:177
frame #5: 0x00007f0f62cbb060 libcoreclr.so`mono_os_cond_timedwait(cond=0x0000000000c05520, mutex=0x0000000000c054f8, timeout_ms=<unavailable>) + 224 at mono-os-mutex.c:32
frame #6: 0x00007f0f62c24d99 libcoreclr.so`mono_w32handle_timedwait_signal_handle + 43 at mono-coop-mutex.h:100
frame #7: 0x00007f0f62c24d6e libcoreclr.so`mono_w32handle_timedwait_signal_handle at w32handle.c:652
frame #8: 0x00007f0f62c24d6e libcoreclr.so`mono_w32handle_timedwait_signal_handle(handle_data=0x0000000000c054e8, timeout=4294967295, alerted=0x00007ffdcfab9fd0, poll=0) + 142 at w32handle.c:767
frame #9: 0x00007f0f62c263af libcoreclr.so`mono_w32handle_wait_one(handle=<unavailable>, timeout=4294967295, alertable=<unavailable>) + 431 at w32handle.c:882
frame #10: 0x00007f0f62c26b88 libcoreclr.so`mono_w32handle_wait_multiple(handles=0x00007ffdcfaba7b0, nhandles=1, waitall=0, timeout=4294967295, alertable=1, error=<unavailable>) + 1496 at w32handle.c:1002
frame #11: 0x00007f0f62c0c97b libcoreclr.so`ves_icall_System_Threading_WaitHandle_Wait_internal(handles=0x00007ffdcfaba7b0, numhandles=1, waitall=<unavailable>, timeout=-1, error=0x00007ffdcfaba720) + 219 at threads.c:2266
frame #12: 0x00007f0f62bb7a29 libcoreclr.so`ves_icall_System_Threading_WaitHandle_Wait_internal_raw(a0=0x00007ffdcfaba7b0, a1=1, a2=<unavailable>, a3=-1, error=0x00007ffdcfaba720) + 105 at icall-def-netcore.h:541
frame #13: 0x00000000405ac98f
frame #14: 0x00000000405ac194
frame #15: 0x0000000040577758
frame #16: 0x0000000040548fd4
frame #17: 0x00000000404dce00
frame #18: 0x00000000404dcf42
frame #19: 0x00007f0f62a28246 libcoreclr.so`mono_jit_runtime_invoke(method=<unavailable>, obj=0x0000000000000000, params=0xffffffffffffffff, exc=<unavailable>, error=0x0000000000000000) + 1878 at mini-runtime.c:3185
frame #20: 0x00007f0f62bf237c libcoreclr.so`do_runtime_invoke(method=0x0000000000c15488, obj=<unavailable>, params=<unavailable>, exc=<unavailable>, error=0x00007ffdcfabb0c0) + 60 at object.c:3042
frame #21: 0x00007f0f62bf5195 libcoreclr.so`do_exec_main_checked(method=0x0000000000c15488, args=<unavailable>, error=0x00007ffdcfabb0c0) + 117 at object.c:5159
frame #22: 0x00007f0f62a7c972 libcoreclr.so`mono_jit_exec + 50 at driver.c:1322
frame #23: 0x00007f0f62a7c940 libcoreclr.so`mono_jit_exec(domain=0x0000000000b841c0, assembly=<unavailable>, argc=11, argv=0x0000000000bc20c0) + 336 at driver.c:1267
frame #24: 0x00007f0f62a7df80 libcoreclr.so`mono_main + 3984 at driver.c:1404
frame #25: 0x00007f0f62a7de41 libcoreclr.so`mono_main(argc=13, argv=0x0000000000bc20b0) + 3665 at driver.c:2624
frame #26: 0x00007f0f62b1c74d libcoreclr.so`coreclr_execute_assembly(hostHandle=<unavailable>, domainId=<unavailable>, argc=10, argv=0x0000000000bc16c0, managedAssemblyPath="/home/filipnavara/mono/netcore/corefx/tests/extracted/System.Security.Cryptography.Algorithms.Tests/xunit.console.dll", exitCode=0x00007ffdcfabb360) + 157 at main-core.c:284
frame #27: 0x00007f0f62ff8303 libhostpolicy.so`??? + 755
frame #28: 0x00007f0f62ff8b5c libhostpolicy.so`corehost_main + 204
frame #29: 0x00007f0f6325cbf0 libhostfxr.so`??? + 1536
frame #30: 0x00007f0f6325b783 libhostfxr.so`??? + 691
thread #2: tid = 5868, 0x00007f0f64309360 libpthread.so.0`__pthread_cond_wait + 192, name = 'SGen worker', stop reason = signal SIGSTOP
frame #0: 0x00007f0f64309360 libpthread.so.0`__pthread_cond_wait + 192
frame #1: 0x00007f0f62caa72b libcoreclr.so`thread_func + 19 at mono-os-mutex.h:177
frame #2: 0x00007f0f62caa718 libcoreclr.so`thread_func + 77 at sgen-thread-pool.c:165
frame #3: 0x00007f0f62caa6cb libcoreclr.so`thread_func(data=<unavailable>) + 779 at sgen-thread-pool.c:196
frame #4: 0x00007f0f643036ba libpthread.so.0`start_thread + 202
frame #5: 0x00007f0f6359441d libc.so.6`clone + 109
thread #3: tid = 5869, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'Finalizer', stop reason = signal SIGSTOP
frame #0: 0x00007f0f634c2826 libc.so.6`sigsuspend + 86
frame #1: 0x00007f0f62cc8eb7 libcoreclr.so`suspend_signal_handler(_dummy=<unavailable>, info=<unavailable>, context=0x00007f0f606e1780) + 215 at mono-threads-posix-signals.c:200
frame #2: 0x00007f0f6430d390 libpthread.so.0`??? + 1
frame #3: 0x00007f0f6430c26b libpthread.so.0`__lll_lock_wait + 27
frame #4: 0x00007f0f64305dbd libpthread.so.0`__GI___pthread_mutex_lock + 125
frame #5: 0x00007f0f62c7da76 libcoreclr.so`sgen_gc_invoke_finalizers + 5 at mono-os-mutex.h:103
frame #6: 0x00007f0f62c7da71 libcoreclr.so`sgen_gc_invoke_finalizers + 34 at mono-coop-mutex.h:54
frame #7: 0x00007f0f62c7da4f libcoreclr.so`sgen_gc_invoke_finalizers [inlined] sgen_gc_lock at sgen-gc.c:3773
frame #8: 0x00007f0f62c7da4f libcoreclr.so`sgen_gc_invoke_finalizers + 239 at sgen-gc.c:2744
frame #9: 0x00007f0f62c548c9 libcoreclr.so`finalizer_thread [inlined] mono_runtime_do_background_work + 262 at gc.c:930
frame #10: 0x00007f0f62c547c3 libcoreclr.so`finalizer_thread(unused=<unavailable>) + 147 at gc.c:969
frame #11: 0x00007f0f62c0f07e libcoreclr.so`start_wrapper + 250 at threads.c:1233
frame #12: 0x00007f0f62c0ef84 libcoreclr.so`start_wrapper(data=0x0000000000c1e5c0) + 68 at threads.c:1306
frame #13: 0x00007f0f643036ba libpthread.so.0`start_thread + 202
frame #14: 0x00007f0f6359441d libc.so.6`clone + 109
thread #4: tid = 5870, 0x00007f0f6430b827 libpthread.so.0`do_futex_wait.constprop.3 + 55, name = 'dotnet', stop reason = signal SIGSTOP
frame #0: 0x00007f0f6430b827 libpthread.so.0`do_futex_wait.constprop.3 + 55
frame #1: 0x00007f0f6430b8d4 libpthread.so.0`__new_sem_wait_slow.constprop.0 + 84
frame #2: 0x00007f0f6430b97a libpthread.so.0`__new_sem_wait + 58
frame #3: 0x00007f0f62cc3d13 libcoreclr.so`mono_thread_info_suspend_lock_with_info + 12 at mono-os-semaphore.h:203
frame #4: 0x00007f0f62cc3d07 libcoreclr.so`mono_thread_info_suspend_lock_with_info(info=<unavailable>) + 103 at mono-threads.c:1418
frame #5: 0x00007f0f62cc5e0e libcoreclr.so`mono_thread_info_attach + 255 at mono-threads.c:496
frame #6: 0x00007f0f62cc5d0f libcoreclr.so`mono_thread_info_attach + 143 at mono-threads.c:702
frame #7: 0x00007f0f62c0b9bd libcoreclr.so`mono_thread_attach(domain=0x0000000000b841c0) + 61 at threads.c:1524
frame #8: 0x00007f0f62c0e5f0 libcoreclr.so`mono_threads_attach_coop_internal(domain=0x0000000000b841c0, cookie=0x00007f0f59551eb8, stackdata=0x00007f0f59551e80) + 304 at threads.c:6008
frame #9: 0x00007f0f62c0e63c libcoreclr.so`mono_threads_attach_coop(domain=<unavailable>, dummy=<unavailable>) + 44 at threads.c:6045
thread #5: tid = 5873, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'dotnet', stop reason = signal SIGSTOP
frame #0: 0x00007f0f634c2826 libc.so.6`sigsuspend + 86
frame #1: 0x00007f0f62cc8eb7 libcoreclr.so`suspend_signal_handler(_dummy=<unavailable>, info=<unavailable>, context=0x00007f0f58b1aa40) + 215 at mono-threads-posix-signals.c:200
frame #2: 0x00007f0f6430d390 libpthread.so.0`??? + 1
frame #3: 0x00007f0f6430935e libpthread.so.0`__pthread_cond_wait + 190
frame #4: 0x00007f0f62cbb065 libcoreclr.so`mono_os_cond_timedwait + 5 at mono-os-mutex.h:177
frame #5: 0x00007f0f62cbb060 libcoreclr.so`mono_os_cond_timedwait(cond=0x0000000000c05590, mutex=0x0000000000c05568, timeout_ms=<unavailable>) + 224 at mono-os-mutex.c:32
frame #6: 0x00007f0f62c24d99 libcoreclr.so`mono_w32handle_timedwait_signal_handle + 43 at mono-coop-mutex.h:100
frame #7: 0x00007f0f62c24d6e libcoreclr.so`mono_w32handle_timedwait_signal_handle at w32handle.c:652
frame #8: 0x00007f0f62c24d6e libcoreclr.so`mono_w32handle_timedwait_signal_handle(handle_data=0x0000000000c05558, timeout=4294967295, alerted=0x00007f0f58b1b0b0, poll=0) + 142 at w32handle.c:767
frame #9: 0x00007f0f62c263af libcoreclr.so`mono_w32handle_wait_one(handle=<unavailable>, timeout=4294967295, alertable=<unavailable>) + 431 at w32handle.c:882
frame #10: 0x00007f0f62c26b88 libcoreclr.so`mono_w32handle_wait_multiple(handles=0x00007f0f58b1b890, nhandles=1, waitall=0, timeout=4294967295, alertable=1, error=<unavailable>) + 1496 at w32handle.c:1002
frame #11: 0x00007f0f62c0c97b libcoreclr.so`ves_icall_System_Threading_WaitHandle_Wait_internal(handles=0x00007f0f58b1b890, numhandles=1, waitall=<unavailable>, timeout=-1, error=0x00007f0f58b1b800) + 219 at threads.c:2266
frame #12: 0x00007f0f62bb7a29 libcoreclr.so`ves_icall_System_Threading_WaitHandle_Wait_internal_raw(a0=0x00007f0f58b1b890, a1=1, a2=<unavailable>, a3=-1, error=0x00007f0f58b1b800) + 105 at icall-def-netcore.h:541
frame #13: 0x00000000405ac98f
frame #14: 0x00000000405ac194
frame #15: 0x00000000405ad353
frame #16: 0x00000000405acc88
frame #17: 0x00000000405ac558
thread #6: tid = 5874, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'dotnet', stop reason = signal SIGSTOP
frame #0: 0x00007f0f634c2826 libc.so.6`sigsuspend + 86
frame #1: 0x00007f0f62cc8eb7 libcoreclr.so`suspend_signal_handler(_dummy=<unavailable>, info=<unavailable>, context=0x00007f0f58d20700) + 215 at mono-threads-posix-signals.c:200
frame #2: 0x00007f0f6430d390 libpthread.so.0`??? + 1
frame #3: 0x00007f0f64309709 libpthread.so.0`__pthread_cond_timedwait + 297
frame #4: 0x00007f0f62cbb00b libcoreclr.so`mono_os_cond_timedwait(cond=0x00007f0f62fdc100, mutex=0x00007f0f62fdc140, timeout_ms=500) + 139 at mono-os-mutex.c:75
frame #5: 0x00007f0f62cc6904 libcoreclr.so`mono_thread_info_sleep + 55 at mono-coop-mutex.h:100
frame #6: 0x00007f0f62cc68cd libcoreclr.so`mono_thread_info_sleep + 137 at mono-threads.c:1634
frame #7: 0x00007f0f62cc6844 libcoreclr.so`mono_thread_info_sleep(ms=500, alerted=0x00007f0f58d20df4) + 52 at mono-threads.c:1666
frame #8: 0x00007f0f62b69f32 libcoreclr.so`monitor_thread(unused=<unavailable>) + 242 at threadpool-worker-default.c:708
frame #9: 0x00007f0f62c0f07e libcoreclr.so`start_wrapper + 250 at threads.c:1233
frame #10: 0x00007f0f62c0ef84 libcoreclr.so`start_wrapper(data=0x00007f0f4c004fa0) + 68 at threads.c:1306
frame #11: 0x00007f0f643036ba libpthread.so.0`start_thread + 202
frame #12: 0x00007f0f6359441d libc.so.6`clone + 109
thread #7: tid = 5875, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'Thread Pool Wor', stop reason = signal SIGSTOP
frame #0: 0x00007f0f634c2826 libc.so.6`sigsuspend + 86
frame #1: 0x00007f0f62cc8eb7 libcoreclr.so`suspend_signal_handler(_dummy=<unavailable>, info=<unavailable>, context=0x00007f0f53bb8900) + 215 at mono-threads-posix-signals.c:200
frame #2: 0x00007f0f6430d390 libpthread.so.0`??? + 1
frame #3: 0x00007f0f64309709 libpthread.so.0`__pthread_cond_timedwait + 297
frame #4: 0x00007f0f62cbb00b libcoreclr.so`mono_os_cond_timedwait(cond=0x0000000000c05670, mutex=0x0000000000c05648, timeout_ms=12000) + 139 at mono-os-mutex.c:75
frame #5: 0x00007f0f62c24d99 libcoreclr.so`mono_w32handle_timedwait_signal_handle + 43 at mono-coop-mutex.h:100
frame #6: 0x00007f0f62c24d6e libcoreclr.so`mono_w32handle_timedwait_signal_handle at w32handle.c:652
frame #7: 0x00007f0f62c24d6e libcoreclr.so`mono_w32handle_timedwait_signal_handle(handle_data=0x0000000000c05638, timeout=12000, alerted=0x00007f0f53bb8fb0, poll=0) + 142 at w32handle.c:767
frame #8: 0x00007f0f62c2640e libcoreclr.so`mono_w32handle_wait_one(handle=<unavailable>, timeout=12000, alertable=<unavailable>) + 526 at w32handle.c:892
frame #9: 0x00007f0f62c26b88 libcoreclr.so`mono_w32handle_wait_multiple(handles=0x00007f0f53bb9790, nhandles=1, waitall=0, timeout=12000, alertable=1, error=<unavailable>) + 1496 at w32handle.c:1002
frame #10: 0x00007f0f62c0c97b libcoreclr.so`ves_icall_System_Threading_WaitHandle_Wait_internal(handles=0x00007f0f53bb9790, numhandles=1, waitall=<unavailable>, timeout=12000, error=0x00007f0f53bb9700) + 219 at threads.c:2266
frame #11: 0x00007f0f62bb7a29 libcoreclr.so`ves_icall_System_Threading_WaitHandle_Wait_internal_raw(a0=0x00007f0f53bb9790, a1=1, a2=<unavailable>, a3=12000, error=0x00007f0f53bb9700) + 105 at icall-def-netcore.h:541
frame #12: 0x00000000405ac98f
frame #13: 0x000000004060bd8c
frame #14: 0x000000004060aa4f
frame #15: 0x0000000040605f90
frame #16: 0x00007f0f62a28246 libcoreclr.so`mono_jit_runtime_invoke(method=<unavailable>, obj=0x0000000000000000, params=0x0000000000002ee0, exc=<unavailable>, error=0x0000000000000000) + 1878 at mini-runtime.c:3185
frame #17: 0x00007f0f62bf237c libcoreclr.so`do_runtime_invoke(method=0x0000000000c00f68, obj=<unavailable>, params=<unavailable>, exc=<unavailable>, error=0x00007f0f53bb9ce0) + 60 at object.c:3042
frame #18: 0x00007f0f62c1035a libcoreclr.so`worker_callback + 80 at threadpool-netcore.c:205
frame #19: 0x00007f0f62c1030a libcoreclr.so`worker_callback + 730 at threadpool-netcore.c:291
frame #20: 0x00007f0f62b694e6 libcoreclr.so`worker_thread(unused=<unavailable>) + 1430 at threadpool-worker-default.c:500
frame #21: 0x00007f0f62c0f07e libcoreclr.so`start_wrapper + 250 at threads.c:1233
frame #22: 0x00007f0f62c0ef84 libcoreclr.so`start_wrapper(data=0x00007f0f4c004fa0) + 68 at threads.c:1306
frame #23: 0x00007f0f643036ba libpthread.so.0`start_thread + 202
frame #24: 0x00007f0f6359441d libc.so.6`clone + 109
thread #8: tid = 5876, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'Thread Pool Wor', stop reason = signal SIGSTOP
frame #0: 0x00007f0f634c2826 libc.so.6`sigsuspend + 86
frame #1: 0x00007f0f62cc8eb7 libcoreclr.so`suspend_signal_handler(_dummy=<unavailable>, info=<unavailable>, context=0x00007f0f539b8700) + 215 at mono-threads-posix-signals.c:200
frame #2: 0x00007f0f6430d390 libpthread.so.0`??? + 1
frame #3: 0x00007f0f6430ba15 libpthread.so.0`do_futex_wait + 69
frame #4: 0x00007f0f6430badf libpthread.so.0`__new_sem_wait_slow + 95
frame #5: 0x00007f0f6430bb92 libpthread.so.0`sem_timedwait + 66
frame #6: 0x00007f0f62b69312 libcoreclr.so`worker_thread + 151 at mono-os-semaphore.h:252
frame #7: 0x00007f0f62b6927b libcoreclr.so`worker_thread + 29 at mono-coop-semaphore.h:55
frame #8: 0x00007f0f62b6925e libcoreclr.so`worker_thread + 256 at threadpool-worker-default.c:391
frame #9: 0x00007f0f62b6915e libcoreclr.so`worker_thread(unused=<unavailable>) + 526 at threadpool-worker-default.c:490
frame #10: 0x00007f0f62c0f07e libcoreclr.so`start_wrapper + 250 at threads.c:1233
frame #11: 0x00007f0f62c0ef84 libcoreclr.so`start_wrapper(data=0x00007f0f480079b0) + 68 at threads.c:1306
frame #12: 0x00007f0f643036ba libpthread.so.0`start_thread + 202
frame #13: 0x00007f0f6359441d libc.so.6`clone + 109
thread #9: tid = 5877, 0x00007f0f6430b827 libpthread.so.0`do_futex_wait.constprop.3 + 55, name = 'dotnet', stop reason = signal SIGSTOP
frame #0: 0x00007f0f6430b827 libpthread.so.0`do_futex_wait.constprop.3 + 55
frame #1: 0x00007f0f6430b8d4 libpthread.so.0`__new_sem_wait_slow.constprop.0 + 84
frame #2: 0x00007f0f6430b97a libpthread.so.0`__new_sem_wait + 58
frame #3: 0x00007f0f62cc3f82 libcoreclr.so`mono_thread_info_wait_for_resume + 8 at mono-os-semaphore.h:203
frame #4: 0x00007f0f62cc3f7a libcoreclr.so`mono_thread_info_wait_for_resume(info=0x00007f0f440008c0) + 10 at mono-threads.c:232
frame #5: 0x00007f0f62cc9638 libcoreclr.so`mono_threads_state_poll_with_info(info=0x00007f0f440008c0) + 280 at mono-threads-coop.c:158
frame #6: 0x00000000405c4391
frame #7: 0x00000000406b10c4
frame #8: 0x00000000407a08a4
frame #9: 0x00007f0f62a28246 libcoreclr.so`mono_jit_runtime_invoke(method=<unavailable>, obj=0x0000000000000000, params=0x00007f0f44001930, exc=<unavailable>, error=0x0000000000000037) + 1878 at mini-runtime.c:3185
frame #10: 0x00007f0f62bf237c libcoreclr.so`do_runtime_invoke(method=0x00007f0f540b0380, obj=<unavailable>, params=<unavailable>, exc=<unavailable>, error=0x00007f0f537b46b0) + 60 at object.c:3042
frame #11: 0x00007f0f62bff601 libcoreclr.so`mono_runtime_try_invoke_array(method=<unavailable>, obj=0x0000000000000000, params=0x00007f0f5885cd90, exc=0x0000000000000000, error=0x00007f0f537b46b0) + 2065 at object.c:5605
frame #12: 0x00007f0f62ba2508 libcoreclr.so`ves_icall_InternalInvoke(method_handle=<unavailable>, this_arg_handle=<unavailable>, params_handle=<unavailable>, exception_out=MonoExceptionHandleOut @ rbx, error=0x00007f0f537b46b0) + 1016 at icall.c:3738
frame #13: 0x00007f0f62baee29 libcoreclr.so`ves_icall_InternalInvoke_raw(a0=0x00007f0f537b4740, a1=0x00007f0f537b4748, a2=0x00007f0f537b4750, a3=0x00007f0f537b4738, error=0x00007f0f537b46b0) + 105 at icall-def-netcore.h:261
frame #14: 0x0000000040570c8c
frame #15: 0x000000004056fffa
frame #16: 0x0000000040652790
frame #17: 0x000000004065269c
frame #18: 0x0000000040652060
frame #19: 0x0000000040651f18
frame #20: 0x00000000406517b8
frame #21: 0x0000000040651674
frame #22: 0x0000000040650878
frame #23: 0x000000004065073c
frame #24: 0x000000004064e114
frame #25: 0x000000004064e00c
frame #26: 0x000000004064d928
frame #27: 0x000000004064d7b0
frame #28: 0x000000004064c62a
frame #29: 0x000000004064c11c
frame #30: 0x000000004064bf7c
frame #31: 0x000000004064b150
frame #32: 0x000000004064adac
frame #33: 0x0000000040648034
frame #34: 0x0000000040647d1c
frame #35: 0x0000000040644560
frame #36: 0x00000000406440c8
frame #37: 0x0000000040643f2c
frame #38: 0x0000000040643f9c
frame #39: 0x0000000040643730
frame #40: 0x00000000406432b4
thread #10: tid = 5878, 0x00007f0f6430b827 libpthread.so.0`do_futex_wait.constprop.3 + 55, name = 'dotnet', stop reason = signal SIGSTOP
frame #0: 0x00007f0f6430b827 libpthread.so.0`do_futex_wait.constprop.3 + 55
frame #1: 0x00007f0f6430b8d4 libpthread.so.0`__new_sem_wait_slow.constprop.0 + 84
frame #2: 0x00007f0f6430b97a libpthread.so.0`__new_sem_wait + 58
frame #3: 0x00007f0f62cc3f82 libcoreclr.so`mono_thread_info_wait_for_resume + 8 at mono-os-semaphore.h:203
frame #4: 0x00007f0f62cc3f7a libcoreclr.so`mono_thread_info_wait_for_resume(info=0x00007f0f380008c0) + 10 at mono-threads.c:232
frame #5: 0x00007f0f62cc9638 libcoreclr.so`mono_threads_state_poll_with_info(info=0x00007f0f380008c0) + 280 at mono-threads-coop.c:158
thread #11: tid = 5879, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'dotnet', stop reason = signal SIGSTOP
frame #0: 0x00007f0f634c2826 libc.so.6`sigsuspend + 86
frame #1: 0x00007f0f62cc8eb7 libcoreclr.so`suspend_signal_handler(_dummy=<unavailable>, info=<unavailable>, context=0x00007f0f533b08c0) + 215 at mono-threads-posix-signals.c:200
frame #2: 0x00007f0f6430d390 libpthread.so.0`??? + 1
frame #3: 0x00007f0f53c747c2 libcrypto.so.1.0.0`BN_sub + 2
frame #4: 0x00007f0f53c7ada4 libcrypto.so.1.0.0`BN_mod_lshift_quick + 132
frame #5: 0x00007f0f53c902a7 libcrypto.so.1.0.0`??? + 615
frame #6: 0x00007f0f53c92ee1 libcrypto.so.1.0.0`??? + 705
frame #7: 0x00007f0f53c936f9 libcrypto.so.1.0.0`??? + 1049
frame #8: 0x00007f0f53c8f140 libcrypto.so.1.0.0`EC_POINT_mul + 64
frame #9: 0x00007f0f53cc521f libcrypto.so.1.0.0`??? + 191
frame #10: 0x00007f0f53c9c400 libcrypto.so.1.0.0`??? + 80
frame #11: 0x00007f0f53c9c565 libcrypto.so.1.0.0`??? + 213
frame #12: 0x00007f0f584d1833 System.Security.Cryptography.Native.OpenSsl.so`CryptoNative_EvpPKeyDeriveSecretAgreement(secret="", secretLength=<unavailable>, ctx=<unavailable>) + 67 at pal_evp_pkey_ecdh.c:46 [opt]
frame #13: 0x00000000407fb8a7
thread #12: tid = 5880, 0x00007f0f6430cf7b libpthread.so.0`__waitpid + 107, name = 'dotnet', stop reason = signal SIGSTOP
frame #0: 0x00007f0f6430cf7b libpthread.so.0`__waitpid + 107
frame #1: 0x00007f0f62b1bfa9 libcoreclr.so`dump_native_stacktrace(mctx=0x00007f0f5804b920, signal=<unavailable>) + 665 at mini-posix.c:1074
frame #2: 0x00007f0f62b1c0ec libcoreclr.so`mono_dump_native_crash_info(signal=<unavailable>, mctx=0x00007f0f5804b920, info=<unavailable>) + 12 at mini-posix.c:1120
frame #3: 0x00007f0f62aadbd0 libcoreclr.so`mono_handle_native_crash(signal="SIGSEGV", mctx=0x00007f0f5804b920, info=0x00007f0f5804bbf0) + 208 at mini-exceptions.c:3356
frame #4: 0x00007f0f62a28c9e libcoreclr.so`mono_sigsegv_signal_handler_debug(_dummy=11, _info=0x00007f0f5804bbf0, context=0x00007f0f5804bac0, debug_fault_addr=<unavailable>) + 782 at mini-runtime.c:3365
frame #5: 0x00007f0f6430d390 libpthread.so.0`??? + 1
frame #6: 0x00007f0f62c7c18a libcoreclr.so`pin_objects_in_nursery + 10 at sgen-gc.h:770
frame #7: 0x00007f0f62c7c180 libcoreclr.so`pin_objects_in_nursery + 233 at sgen-gc.c:630
frame #8: 0x00007f0f62c7c097 libcoreclr.so`pin_objects_in_nursery(do_scan_objects=0, ctx=<unavailable>) + 39 at sgen-gc.c:712
frame #9: 0x00007f0f62c7ccc5 libcoreclr.so`major_copy_or_mark_from_roots(gc_thread_gray_queue=0x00007f0f531b0d10, old_next_pin_slot=0x0000000000000000, mode=COPY_OR_MARK_FROM_ROOTS_FINISH_CONCURRENT, object_ops_nopar=0x00007f0f62fe15a8, object_ops_par=0x0000000000000000) + 517 at sgen-gc.c:2022
frame #10: 0x00007f0f62c7dc2b libcoreclr.so`major_finish_collection(gc_thread_gray_queue=0x00007f0f531b0d10, reason="finishing", is_overflow=0, old_next_pin_slot=18446744073709551615, forced=0) + 139 at sgen-gc.c:2205
frame #11: 0x00007f0f62c81980 libcoreclr.so`sgen_perform_collection_inner + 144 at sgen-gc.c:2460
frame #12: 0x00007f0f62c818f0 libcoreclr.so`sgen_perform_collection_inner(requested_size=536870960, generation_to_collect=1, reason="Finish concurrent collection", forced_serial=0, stw=1) + 976 at sgen-gc.c:2549
frame #13: 0x00007f0f62c81b7b libcoreclr.so`sgen_ensure_free_space [inlined] sgen_perform_collection(stw=<unavailable>, forced_serial=<unavailable>, reason=<unavailable>, generation_to_collect=<unavailable>, requested_size=<unavailable>) + 123 at sgen-gc.c:2644
frame #14: 0x00007f0f62c81b69 libcoreclr.so`sgen_ensure_free_space(size=536870960, generation=<unavailable>) + 105 at sgen-gc.c:2510
frame #15: 0x00007f0f62c84ab9 libcoreclr.so`sgen_los_alloc_large_inner(vtable=0x0000000000dffdb0, size=536870960) + 89 at sgen-los.c:380
frame #16: 0x00007f0f62c71969 libcoreclr.so`sgen_alloc_obj_nolock(vtable=0x0000000000dffdb0, size=536870960) + 217 at sgen-alloc.c:183
frame #17: 0x00007f0f62c61efc libcoreclr.so`mono_gc_alloc_vector(vtable=0x0000000000dffdb0, size=536870960, max_length=536870928) + 172 at sgen-mono.c:1322
frame #18: 0x00007f0f62bfb31b libcoreclr.so`mono_array_new_full_checked(domain=0x0000000000b841c0, array_class=0x0000000000db9cc0, lengths=0x00007f0f531b0fa0, lower_bounds=0x0000000000000000, error=0x00007f0f531b10a0) + 459 at object.c:6419
frame #19: 0x00007f0f62c23799 libcoreclr.so`mono_array_new_full_handle(domain=<unavailable>, array_class=<unavailable>, lengths=<unavailable>, lower_bounds=<unavailable>, error=<unavailable>) + 9 at handle.c:396
frame #20: 0x00007f0f62bfd7de libcoreclr.so`mono_array_clone_in_domain(domain=0x0000000000b841c0, array_handle=MonoArrayHandle @ rbx, error=0x00007f0f531b10a0) + 446 at object.c:6225
frame #21: 0x00007f0f62bfd89f libcoreclr.so`mono_object_clone_handle(obj=MonoObjectHandle @ rbx, error=0x00007f0f531b10a0) + 63 at object.c:6142
frame #22: 0x00007f0f62bac551 libcoreclr.so`ves_icall_System_Object_MemberwiseClone_raw [inlined] ves_icall_System_Object_MemberwiseClone(error=0x00007f0f531b10a0, this_obj=<unavailable>) + 81 at icall.c:1320
frame #23: 0x00007f0f62bac54c libcoreclr.so`ves_icall_System_Object_MemberwiseClone_raw(a0=0x00007f0f531b1128, error=0x00007f0f531b10a0) + 76 at icall-def-netcore.h:177
frame #24: 0x000000004056fe38
frame #25: 0x0000000040807408
frame #26: 0x000000004080768c
frame #27: 0x00007f0f62a28246 libcoreclr.so`mono_jit_runtime_invoke(method=<unavailable>, obj=0x0000000000000000, params=0x00007f0f30001930, exc=<unavailable>, error=0x00007f0f624a3f30) + 1878 at mini-runtime.c:3185
frame #28: 0x00007f0f62bf237c libcoreclr.so`do_runtime_invoke(method=0x00007f0f5413d2f8, obj=<unavailable>, params=<unavailable>, exc=<unavailable>, error=0x00007f0f531b1700) + 60 at object.c:3042
frame #29: 0x00007f0f62bff601 libcoreclr.so`mono_runtime_try_invoke_array(method=<unavailable>, obj=0x0000000000000000, params=0x00007f0f620a9b00, exc=0x0000000000000000, error=0x00007f0f531b1700) + 2065 at object.c:5605
frame #30: 0x00007f0f62ba2508 libcoreclr.so`ves_icall_InternalInvoke(method_handle=<unavailable>, this_arg_handle=<unavailable>, params_handle=<unavailable>, exception_out=MonoExceptionHandleOut @ rbx, error=0x00007f0f531b1700) + 1016 at icall.c:3738
frame #31: 0x00007f0f62baee29 libcoreclr.so`ves_icall_InternalInvoke_raw(a0=0x00007f0f531b1790, a1=0x00007f0f531b1798, a2=0x00007f0f531b17a0, a3=0x00007f0f531b1788, error=0x00007f0f531b1700) + 105 at icall-def-netcore.h:261
frame #32: 0x0000000040570c8c
frame #33: 0x000000004056fffa
frame #34: 0x0000000040652790
frame #35: 0x000000004065269c
frame #36: 0x0000000040652060
frame #37: 0x0000000040651f18
frame #38: 0x00000000406517b8
frame #39: 0x0000000040651674
frame #40: 0x0000000040650878
frame #41: 0x000000004065073c
frame #42: 0x000000004064e114
frame #43: 0x000000004064e00c
frame #44: 0x000000004064d928
frame #45: 0x000000004064d7b0
frame #46: 0x000000004064c62a
frame #47: 0x000000004064c11c
frame #48: 0x000000004064bf7c
frame #49: 0x000000004064b150
frame #50: 0x000000004064adac
frame #51: 0x0000000040648034
frame #52: 0x0000000040647d1c
frame #53: 0x00000000406a9244
frame #54: 0x00000000406a8f14
frame #55: 0x00000000406440c8
frame #56: 0x0000000040643f2c
frame #57: 0x0000000040642fdb
frame #58: 0x0000000040642684
frame #59: 0x000000004064250c
frame #60: 0x0000000040640ed0
frame #61: 0x0000000040640cac
frame #62: 0x000000004063f532
frame #63: 0x000000004063e9f0
frame #64: 0x000000004063e8dc
frame #65: 0x000000004063b398
frame #66: 0x000000004063b29c
frame #67: 0x000000004063a5a1
frame #68: 0x0000000040639ff4
frame #69: 0x0000000040639eec
frame #70: 0x0000000040634898
frame #71: 0x000000004063479c
frame #72: 0x0000000040633c91
frame #73: 0x00000000405acc88
frame #74: 0x0000000040633654
frame #75: 0x000000004062cf70
frame #76: 0x00000000405ad353
frame #77: 0x00000000405acc88
frame #78: 0x00000000405ac558
thread #13: tid = 5881, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'Thread Pool Wor', stop reason = signal SIGSTOP
frame #0: 0x00007f0f634c2826 libc.so.6`sigsuspend + 86
frame #1: 0x00007f0f62cc8eb7 libcoreclr.so`suspend_signal_handler(_dummy=<unavailable>, info=<unavailable>, context=0x00007f0f527fe700) + 215 at mono-threads-posix-signals.c:200
frame #2: 0x00007f0f6430d390 libpthread.so.0`??? + 1
frame #3: 0x00007f0f6430ba15 libpthread.so.0`do_futex_wait + 69
frame #4: 0x00007f0f6430badf libpthread.so.0`__new_sem_wait_slow + 95
frame #5: 0x00007f0f6430bb92 libpthread.so.0`sem_timedwait + 66
frame #6: 0x00007f0f62b69312 libcoreclr.so`worker_thread + 151 at mono-os-semaphore.h:252
frame #7: 0x00007f0f62b6927b libcoreclr.so`worker_thread + 29 at mono-coop-semaphore.h:55
frame #8: 0x00007f0f62b6925e libcoreclr.so`worker_thread + 256 at threadpool-worker-default.c:391
frame #9: 0x00007f0f62b6915e libcoreclr.so`worker_thread(unused=<unavailable>) + 526 at threadpool-worker-default.c:490
frame #10: 0x00007f0f62c0f07e libcoreclr.so`start_wrapper + 250 at threads.c:1233
frame #11: 0x00007f0f62c0ef84 libcoreclr.so`start_wrapper(data=0x00007f0f400021e0) + 68 at threads.c:1306
frame #12: 0x00007f0f643036ba libpthread.so.0`start_thread + 202
frame #13: 0x00007f0f6359441d libc.so.6`clone + 109
thread #14: tid = 5882, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'Thread Pool Wor', stop reason = signal SIGSTOP
frame #0: 0x00007f0f634c2826 libc.so.6`sigsuspend + 86
frame #1: 0x00007f0f62cc8eb7 libcoreclr.so`suspend_signal_handler(_dummy=<unavailable>, info=<unavailable>, context=0x00007f0f525fd700) + 215 at mono-threads-posix-signals.c:200
frame #2: 0x00007f0f6430d390 libpthread.so.0`??? + 1
frame #3: 0x00007f0f6430ba15 libpthread.so.0`do_futex_wait + 69
frame #4: 0x00007f0f6430badf libpthread.so.0`__new_sem_wait_slow + 95
frame #5: 0x00007f0f6430bb92 libpthread.so.0`sem_timedwait + 66
frame #6: 0x00007f0f62b69312 libcoreclr.so`worker_thread + 151 at mono-os-semaphore.h:252
frame #7: 0x00007f0f62b6927b libcoreclr.so`worker_thread + 29 at mono-coop-semaphore.h:55
frame #8: 0x00007f0f62b6925e libcoreclr.so`worker_thread + 256 at threadpool-worker-default.c:391
frame #9: 0x00007f0f62b6915e libcoreclr.so`worker_thread(unused=<unavailable>) + 526 at threadpool-worker-default.c:490
frame #10: 0x00007f0f62c0f07e libcoreclr.so`start_wrapper + 250 at threads.c:1233
frame #11: 0x00007f0f62c0ef84 libcoreclr.so`start_wrapper(data=0x00007f0f3c260470) + 68 at threads.c:1306
frame #12: 0x00007f0f643036ba libpthread.so.0`start_thread + 202
frame #13: 0x00007f0f6359441d libc.so.6`clone + 109
thread #15: tid = 5883, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'Thread Pool Wor', stop reason = signal SIGSTOP
frame #0: 0x00007f0f634c2826 libc.so.6`sigsuspend + 86
frame #1: 0x00007f0f62cc8eb7 libcoreclr.so`suspend_signal_handler(_dummy=<unavailable>, info=<unavailable>, context=0x00007f0f523fc700) + 215 at mono-threads-posix-signals.c:200
frame #2: 0x00007f0f6430d390 libpthread.so.0`??? + 1
frame #3: 0x00007f0f6430ba15 libpthread.so.0`do_futex_wait + 69
frame #4: 0x00007f0f6430badf libpthread.so.0`__new_sem_wait_slow + 95
frame #5: 0x00007f0f6430bb92 libpthread.so.0`sem_timedwait + 66
frame #6: 0x00007f0f62b69312 libcoreclr.so`worker_thread + 151 at mono-os-semaphore.h:252
frame #7: 0x00007f0f62b6927b libcoreclr.so`worker_thread + 29 at mono-coop-semaphore.h:55
frame #8: 0x00007f0f62b6925e libcoreclr.so`worker_thread + 256 at threadpool-worker-default.c:391
frame #9: 0x00007f0f62b6915e libcoreclr.so`worker_thread(unused=<unavailable>) + 526 at threadpool-worker-default.c:490
frame #10: 0x00007f0f62c0f07e libcoreclr.so`start_wrapper + 250 at threads.c:1233
frame #11: 0x00007f0f62c0ef84 libcoreclr.so`start_wrapper(data=0x00007f0f3c2636e0) + 68 at threads.c:1306
frame #12: 0x00007f0f643036ba libpthread.so.0`start_thread + 202
frame #13: 0x00007f0f6359441d libc.so.6`clone + 109
thread #16: tid = 5884, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'Thread Pool Wor', stop reason = signal SIGSTOP
frame #0: 0x00007f0f634c2826 libc.so.6`sigsuspend + 86
frame #1: 0x00007f0f62cc8eb7 libcoreclr.so`suspend_signal_handler(_dummy=<unavailable>, info=<unavailable>, context=0x00007f0f521fb700) + 215 at mono-threads-posix-signals.c:200
frame #2: 0x00007f0f6430d390 libpthread.so.0`??? + 1
frame #3: 0x00007f0f6430ba15 libpthread.so.0`do_futex_wait + 69
frame #4: 0x00007f0f6430badf libpthread.so.0`__new_sem_wait_slow + 95
frame #5: 0x00007f0f6430bb92 libpthread.so.0`sem_timedwait + 66
frame #6: 0x00007f0f62b69312 libcoreclr.so`worker_thread + 151 at mono-os-semaphore.h:252
frame #7: 0x00007f0f62b6927b libcoreclr.so`worker_thread + 29 at mono-coop-semaphore.h:55
frame #8: 0x00007f0f62b6925e libcoreclr.so`worker_thread + 256 at threadpool-worker-default.c:391
frame #9: 0x00007f0f62b6915e libcoreclr.so`worker_thread(unused=<unavailable>) + 526 at threadpool-worker-default.c:490
frame #10: 0x00007f0f62c0f07e libcoreclr.so`start_wrapper + 250 at threads.c:1233
frame #11: 0x00007f0f62c0ef84 libcoreclr.so`start_wrapper(data=0x00007f0f34002240) + 68 at threads.c:1306
frame #12: 0x00007f0f643036ba libpthread.so.0`start_thread + 202
frame #13: 0x00007f0f6359441d libc.so.6`clone + 109
thread #17: tid = 5885, 0x00007f0f634c2826 libc.so.6`sigsuspend + 86, name = 'Thread Pool Wor', stop reason = signal SIGSTOP
frame #0: 0x00007f0f634c2826 libc.so.6`sigsuspend + 86
frame #1: 0x00007f0f62cc8eb7 libcoreclr.so`suspend_signal_handler(_dummy=<unavailable>, info=<unavailable>, context=0x00007f0f51ffa700) + 215 at mono-threads-posix-signals.c:200
frame #2: 0x00007f0f6430d390 libpthread.so.0`??? + 1
frame #3: 0x00007f0f6430ba15 libpthread.so.0`do_futex_wait + 69
frame #4: 0x00007f0f6430badf libpthread.so.0`__new_sem_wait_slow + 95
frame #5: 0x00007f0f6430bb92 libpthread.so.0`sem_timedwait + 66
frame #6: 0x00007f0f62b69312 libcoreclr.so`worker_thread + 151 at mono-os-semaphore.h:252
frame #7: 0x00007f0f62b6927b libcoreclr.so`worker_thread + 29 at mono-coop-semaphore.h:55
frame #8: 0x00007f0f62b6925e libcoreclr.so`worker_thread + 256 at threadpool-worker-default.c:391
frame #9: 0x00007f0f62b6915e libcoreclr.so`worker_thread(unused=<unavailable>) + 526 at threadpool-worker-default.c:490
frame #10: 0x00007f0f62c0f07e libcoreclr.so`start_wrapper + 250 at threads.c:1233
frame #11: 0x00007f0f62c0ef84 libcoreclr.so`start_wrapper(data=0x00007f0f28002240) + 68 at threads.c:1306
frame #12: 0x00007f0f643036ba libpthread.so.0`start_thread + 202
frame #13: 0x00007f0f6359441d libc.so.6`clone + 109
(lldb) detach
Process 5867 detached
(lldb) quit
=================================================================
Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x7f0f62c7c18a):0x7f0f62c7c17a eb 6d 0f 1f 40 00 48 83 e7 f8 0f 84 68 02 00 00 [email protected]...
0x7f0f62c7c18a 48 8b 3f 4c 89 e6 48 83 e7 f8 e8 17 f0 ff ff 48 H.?L..H........H
0x7f0f62c7c19a 8d 58 07 48 83 e3 f8 49 83 7c 24 08 ff 48 89 da .X.H...I.|$..H..
0x7f0f62c7c1aa 74 35 8b 35 6a bd 35 00 85 f6 74 1a 4c 89 e7 e8 t5.5j.5...t.L...
=================================================================
Managed Stacktrace:
=================================================================
at <unknown> <0xffffffff>
at System.Object:MemberwiseClone <0x00097>
at System.Array:Clone <0x0002b>
at Internal.Cryptography.Helpers:CloneByteArray <0x0003b>
at Internal.Cryptography.AesImplementation:CreateDecryptor <0x00037>
at <>c__DisplayClass4_0:<InvalidIVSizes>b__1 <0x0003b>
at Xunit.Record:Exception <0x00050>
at System.Security.Cryptography.Encryption.Aes.Tests.AesContractTests:InvalidIVSizes <0x00407>
at <Module>:runtime_invoke_void_int_byte <0x0009b>
at <unknown> <0xffffffff>
at System.Reflection.RuntimeMethodInfo:InternalInvoke <0x000db>
at System.Reflection.RuntimeMethodInfo:Invoke <0x0013b>
at System.Reflection.MethodBase:Invoke <0x00049>
at Xunit.Sdk.TestInvoker`1:CallTestMethod <0x00043>
at <<InvokeTestMethodAsync>b__1>d:MoveNext <0x00397>
at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000df>
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start <0x0002b>
at <>c__DisplayClass48_1:<InvokeTestMethodAsync>b__1 <0x000ef>
at <AggregateAsync>d__4:MoveNext <0x000e2>
at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000df>
at Xunit.Sdk.ExecutionTimer:AggregateAsync <0x000f7>
at <>c__DisplayClass48_1:<InvokeTestMethodAsync>b__0 <0x0015b>
at <RunAsync>d__9:MoveNext <0x00095>
at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000d7>
at Xunit.Sdk.ExceptionAggregator:RunAsync <0x000f3>
at <InvokeTestMethodAsync>d__48:MoveNext <0x00333>
at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000e7>
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002b>
at Xunit.Sdk.TestInvoker`1:InvokeTestMethodAsync <0x0011b>
at Xunit.Sdk.XunitTestInvoker:InvokeTestMethodAsync <0x000e7>
at <<RunAsync>b__47_0>d:MoveNext <0x00706>
at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000f3>
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002b>
at Xunit.Sdk.TestInvoker`1:<RunAsync>b__47_0 <0x000fb>
at <RunAsync>d__10`1:MoveNext <0x000c0>
at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000d7>
at Xunit.Sdk.ExceptionAggregator:RunAsync <0x000ff>
at Xunit.Sdk.TestInvoker`1:RunAsync <0x000f3>
at Xunit.Sdk.XunitTestRunner:InvokeTestMethodAsync <0x000db>
at <InvokeTestAsync>d__4:MoveNext <0x001e9>
at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000eb>
at Xunit.Sdk.XunitTestRunner:InvokeTestAsync <0x000fb>
at <>c__DisplayClass43_0:<RunAsync>b__0 <0x00038>
at <RunAsync>d__10`1:MoveNext <0x000a4>
at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000df>
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002b>
at Xunit.Sdk.ExceptionAggregator:RunAsync <0x0013f>
at <RunAsync>d__43:MoveNext <0x004eb>
at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000f3>
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002b>
at Xunit.Sdk.TestRunner`1:RunAsync <0x0011b>
at <RunTestAsync>d__11:MoveNext <0x0020f>
at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x00103>
at Xunit.Sdk.XunitTheoryTestCaseRunner:RunTestAsync <0x000c3>
at <RunAsync>d__19:MoveNext <0x0036f>
at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000e7>
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002b>
at Xunit.Sdk.TestCaseRunner`1:RunAsync <0x00117>
at Xunit.Sdk.XunitTheoryTestCase:RunAsync <0x000af>
at Xunit.Sdk.XunitTestMethodRunner:RunTestCaseAsync <0x0009a>
at <RunTestCasesAsync>d__32:MoveNext <0x001bf>
at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000f3>
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002b>
at Xunit.Sdk.TestMethodRunner`1:RunTestCasesAsync <0x0011b>
at <RunAsync>d__31:MoveNext <0x001ad>
at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000df>
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002b>
at Xunit.Sdk.TestMethodRunner`1:RunAsync <0x00113>
at Xunit.Sdk.XunitTestClassRunner:RunTestMethodAsync <0x000e7>
at <RunTestMethodsAsync>d__38:MoveNext <0x00991>
at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000ff>
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002b>
at Xunit.Sdk.TestClassRunner`1:RunTestMethodsAsync <0x00127>
at <RunAsync>d__37:MoveNext <0x0039e>
at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000e7>
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002b>
at Xunit.Sdk.TestClassRunner`1:RunAsync <0x00117>
at Xunit.Sdk.XunitTestCollectionRunner:RunTestClassAsync <0x000f3>
at <RunTestClassesAsync>d__28:MoveNext <0x00410>
at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000f3>
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002b>
at Xunit.Sdk.TestCollectionRunner`1:RunTestClassesAsync <0x0011b>
at <RunAsync>d__27:MoveNext <0x0039b>
at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000e7>
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002b>
at Xunit.Sdk.TestCollectionRunner`1:RunAsync <0x00117>
at Xunit.Sdk.XunitTestAssemblyRunner:RunTestCollectionAsync <0x000bf>
at <>c__DisplayClass14_2:<RunTestCollectionsAsync>b__2 <0x00060>
at System.Threading.Tasks.Task`1:InnerInvoke <0x00078>
at <>c:<.cctor>b__274_0 <0x0002f>
at System.Threading.ExecutionContext:RunInternal <0x00101>
at System.Threading.Tasks.Task:ExecuteWithThreadLocal <0x00277>
at System.Threading.Tasks.Task:ExecuteEntry <0x000c3>
at <>c:<.cctor>b__8_0 <0x00053>
at Xunit.Sdk.MaxConcurrencySyncContext:RunOnSyncContext <0x0004f>
at <>c__DisplayClass11_0:<WorkerThreadProc>b__0 <0x0004b>
at System.Threading.ExecutionContext:RunInternal <0x00101>
at System.Threading.ExecutionContext:Run <0x0006f>
at System.Object:lambda_method <0x00082>
at Xunit.Sdk.ExecutionContextHelper:Run <0x0005a>
at Xunit.Sdk.MaxConcurrencySyncContext:WorkerThreadProc <0x001e3>
at <>c:<QueueUserWorkItem>b__5_0 <0x00066>
at System.Threading.Tasks.Task:InnerInvoke <0x00092>
at <>c:<.cctor>b__274_0 <0x0002f>
at System.Threading.ExecutionContext:RunInternal <0x00101>
at System.Threading.Tasks.Task:ExecuteWithThreadLocal <0x00277>
at System.Threading.Tasks.Task:ExecuteEntryUnsafe <0x00097>
at <>c:<.cctor>b__10_0 <0x00057>
at System.Threading.Thread:StartCallback <0x000fb>
at System.Object:runtime_invoke_void__this__ <0x00091>
=================================================================
Aborted
Makefile:212: recipe for target 'run-tests-corefx-System.Security.Cryptography.Algorithms.Tests' failed
I have collected dozens of failures and SIGSEGVs so far. Looks like a memory corruption.
Not sure if it's related but I can reliably make it SIGSEGV with interpreter, on both Ubuntu 16.04 and 18.04 with MONO_ENV_OPTIONS="--interp" make run-tests-corefx-System.Security.Cryptography.Algorithms.Tests FIXTURE="-verbose -parallel none":
Managed stack trace:
at <unknown> <0xffffffff>
at Crypto:EvpDigestUpdate <0x0005c>
at Crypto:EvpDigestUpdate <0x00024>
at EvpHashProvider:AppendHashData <0x00018>
at Implementation:HashCore <0x00014>
at System.Security.Cryptography.HashAlgorithm:TryComputeHash <0x00048>
at Internal.Cryptography.AsymmetricAlgorithmHelpers:TryHashData <0x00036>
at DSAOpenSsl:TryHashData <0x0001e>
Native stack trace:
#0 0x00007f8b99bef23a in __waitpid (pid=pid@entry=1355, stat_loc=stat_loc@entry=0x7f8b99e31f64, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1 0x00007f8b9833d785 in dump_native_stacktrace (mctx=mctx@entry=0x7f8b99e32920, signal=0x7f8b984fec1e "SIGSEGV") at mini-posix.c:1074
#2 0x00007f8b9833d80c in mono_dump_native_crash_info (signal=signal@entry=0x7f8b984fec1e "SIGSEGV", mctx=mctx@entry=0x7f8b99e32920, info=info@entry=0x7f8b99e32bf0) at mini-posix.c:1120
#3 0x00007f8b982d1eb0 in mono_handle_native_crash (signal=signal@entry=0x7f8b984fec1e "SIGSEGV", mctx=mctx@entry=0x7f8b99e32920, info=info@entry=0x7f8b99e32bf0) at mini-exceptions.c:3356
#4 0x00007f8b9824f91e in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0x7f8b99e32bf0, context=0x7f8b99e32ac0, debug_fault_addr=<optimized out>) at mini-runtime.c:3365
#5 <signal handler called>
#6 0x00007f8b8d77f7ed in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
#7 0x00007f8b8d780643 in SHA1_Update () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
#8 0x00007f8b9833fd73 in do_icall (sig=sig@entry=0x2d8e488, op=op@entry=556, sp=0x7fff6d16b058, sp@entry=0x7fff6d16b068, ptr=ptr@entry=0x7f8b8dd295e0 <CryptoNative_EvpDigestUpdate>, save_last_error=save_last_error@entry=0) at interp/interp.c:1906
#9 0x00007f8b9834004d in do_icall_wrapper (frame=0x7fff6d16b230, sig=0x2d8e488, op=556, sp=0x7fff6d16b068, ptr=0x7f8b8dd295e0 <CryptoNative_EvpDigestUpdate>, save_last_error=0) at interp/interp.c:1975
#10 0x00007f8b98345650 in interp_exec_method_full (frame=0x7fff6d16b230, context=0xd58a1c7, clause_args=0x798ae28b, error=0xfb3f9d81) at interp/interp.c:3485
#11 0x00007f8b983433b9 in interp_exec_method_full (frame=0x7fff6d16b340, context=0xd58a1c7, clause_args=0x798ae28b, error=0xfb3f9d81) at interp/interp.c:3582
#12 0x00007f8b983433b9 in interp_exec_method_full (frame=0x7fff6d16b440, context=0xd58a1c7, clause_args=0x798ae28b, error=0xfb3f9d81) at interp/interp.c:3582
#13 0x00007f8b98343477 in interp_exec_method_full (frame=0x7fff6d16b580, context=0xd58a1c7, clause_args=0x798ae28b, error=0xfb3f9d81) at interp/interp.c:3595
#14 0x00007f8b98343477 in interp_exec_method_full (frame=0x7fff6d16b6c0, context=0xd58a1c7, clause_args=0x798ae28b, error=0xfb3f9d81) at interp/interp.c:3595
#15 0x00007f8b983433b9 in interp_exec_method_full (frame=0x7fff6d16b7f0, context=0xd58a1c7, clause_args=0x798ae28b, error=0xfb3f9d81) at interp/interp.c:3582
#16 0x00007f8b983433b9 in interp_exec_method_full (frame=0x7fff6d16ba10, context=0xd58a1c7, clause_args=0x798ae28b, error=0xfb3f9d81) at interp/interp.c:3582
#17 0x00007f8b983433b9 in interp_exec_method_full (frame=0x7fff6d16bb40, context=0xd58a1c7, clause_args=0x798ae28b, error=0xfb3f9d81) at interp/interp.c:3582
#18 0x00007f8b983433b9 in interp_exec_method_full (frame=0x7fff6d16bd10, context=0xd58a1c7, clause_args=0x798ae28b, error=0xfb3f9d81) at interp/interp.c:3582
The issue with interpreter is likely orthogonal to the original issue. What happens with the interpreter is the following:
- The managed prototype of the interop call is defined as
private static extern int EvpDigestUpdate(SafeEvpMdCtxHandle ctx, ref byte d, int cnt);. The last parameter is of typeint. - The native prototype is defined as
int32_t CryptoNative_EvpDigestUpdate(EVP_MD_CTX* ctx, const void* d, size_t cnt);. Last parameter is of typesize_t, which is 64-bit unsigned integer. - When the actual call happens I can see in the stack trace that the parameters are
CryptoNative_EvpDigestUpdate (ctx=0x2c87630, d=0x7f3e5ed99850, cnt=140733193388108). The last value is0x7FFF0000004C, while the correct value should have been just0x4C.
/cc @lewurm @BrzVlad
filipnavara
on 19 Sep 2019
The signature needs to be fixed to match. Since c# has no size_t, probably a long/int64_t should be used.
vargaz
on 19 Sep 2019
@vargaz I will file a CoreFX issue for that (https://github.com/dotnet/corefx/issues/41219). The behavior still differs between JIT and Interpreter and I would like to know why.
filipnavara
on 19 Sep 2019
I spent the whole night debugging it and I made it reliably crash on my machine with specific set of random seed (to force xunit test order) and MONO_GC_DEBUG options. I reduced it to about 30 tests in System.Security.Cryptography.DeriveBytesTests.* that were enough to trigger the runtime crash in something that looked like memory corruption.
Today I decided to recompile the runtime as debug to get better stack traces. The crashes instantly went away and I was not able to reproduce them anymore. It's possible that the older GCC on Ubuntu 16.04 miscompiles something.
filipnavara
on 20 Sep 2019
❤1
Smaller repro for the original problem:
using System;
using System.Security.Cryptography;
using System.Threading.Tasks;
namespace HelloWorld
{
class Program
{
static void Main(string[] args)
{
Parallel.For(0, 100, i =>
{
var _salt = new byte[1];
using (var _hmac = new HMACSHA1(new byte[0]))
{
Span<byte> uiSpan = stackalloc byte[64];
uiSpan = uiSpan.Slice(0, 20);
_hmac.TryComputeHash(_salt, uiSpan, out int bytesWritten);
}
});
}
}
}
Replace the HelloWorld sample with it and run MONO_ENV_OPTIONS="--debug" MONO_GC_DEBUG="collect-before-allocs" make run-sample.
filipnavara
on 21 Sep 2019
And here's a version of the repro that removes several layers of the CoreFX code and calls directly to native library. Error checking and all unnecessary code is omitted.
using System;
using System.Threading.Tasks;
using System.Runtime.InteropServices;
using System.Diagnostics;
internal static partial class Interop
{
internal static class Libraries
{
internal const string CryptoNative = "System.Security.Cryptography.Native.OpenSsl";
}
internal sealed class SafeHmacCtxHandle : SafeHandle
{
private SafeHmacCtxHandle() :
base(IntPtr.Zero, ownsHandle: true)
{
}
protected override bool ReleaseHandle()
{
Interop.Crypto.HmacDestroy(handle);
SetHandle(IntPtr.Zero);
return true;
}
public override bool IsInvalid
{
get { return handle == IntPtr.Zero; }
}
}
internal static partial class Crypto
{
[DllImport(Libraries.CryptoNative, EntryPoint = "CryptoNative_EvpSha1")]
internal static extern IntPtr EvpSha1();
[DllImport(Libraries.CryptoNative, EntryPoint = "CryptoNative_HmacCreate")]
internal extern static SafeHmacCtxHandle HmacCreate(ref byte key, int keyLen, IntPtr md);
[DllImport(Libraries.CryptoNative, EntryPoint = "CryptoNative_HmacDestroy")]
internal extern static void HmacDestroy(IntPtr ctx);
[DllImport(Libraries.CryptoNative, EntryPoint = "CryptoNative_HmacFinal")]
internal extern static int HmacFinal(SafeHmacCtxHandle ctx, ref byte data, ref int len);
}
}
namespace HelloWorld
{
public class HMACSHA1 : IDisposable
{
protected byte[] KeyValue;
internal Interop.SafeHmacCtxHandle _hmacCtx;
public HMACSHA1(byte[] key)
{
_hmacCtx = Interop.Crypto.HmacCreate(ref MemoryMarshal.GetReference(new Span<byte>(key)), key.Length, Interop.Crypto.EvpSha1());
KeyValue = new byte[1];
}
public unsafe bool TryFinalizeHashAndReset(Span<byte> destination, out int bytesWritten)
{
int length = destination.Length;
Interop.Crypto.HmacFinal(_hmacCtx, ref MemoryMarshal.GetReference(destination), ref length);
bytesWritten = length;
return true;
}
public bool TryComputeHash(ReadOnlySpan<byte> source, Span<byte> destination, out int bytesWritten)
{
return TryFinalizeHashAndReset(destination, out bytesWritten);
}
public void Dispose()
{
if (_hmacCtx != null)
{
_hmacCtx.Dispose();
_hmacCtx = null;
}
}
}
class Program
{
static void Main(string[] args)
{
Parallel.For(0, 1000, j =>
{
using (var _hmac = new HMACSHA1(new byte[0]))
{
Span<byte> uiSpan = stackalloc byte[20];
_hmac.TryComputeHash(null, uiSpan, out int bytesWritten);
}
});
}
}
}
To rule out the corruption comes from native code you can use this shim:
#include <stdint.h>
#define DLLEXPORT __attribute__ ((__visibility__ ("default")))
DLLEXPORT const void* CryptoNative_EvpSha1(void)
{
return (void*)42;
}
DLLEXPORT void* CryptoNative_HmacCreate(const uint8_t* key, int32_t keyLen, const void* md)
{
}
DLLEXPORT void CryptoNative_HmacDestroy(void* ctx)
{
}
DLLEXPORT int32_t CryptoNative_HmacFinal(void* ctx, uint8_t* data, int32_t* len)
{
return 0;
}
You need to change CryptoNative constant above to native.so and compile this code with gcc -shared -fpic sample/HelloWorld/native.c -o sample/HelloWorld/bin/netcoreapp3.0/native.so.
filipnavara
on 21 Sep 2019
Not sure if it will help but I managed to capture GC log at level 5 with the repro program above: https://gist.github.com/filipnavara/ad556d29bce4c6960976f3df7c2cc31f
filipnavara
on 23 Sep 2019
Smaller repro:
using System;
using System.Threading;
using System.Threading.Tasks;
using System.Runtime.InteropServices;
using System.Diagnostics;
internal sealed class SafeHmacCtxHandle : SafeHandle
{
public SafeHmacCtxHandle() :
base(IntPtr.Zero, ownsHandle: true)
{
}
protected override bool ReleaseHandle()
{
return true;
}
public override bool IsInvalid
{
get { return handle == IntPtr.Zero; }
}
}
namespace HelloWorld
{
public class HMACSHA1 : IDisposable
{
protected byte[] KeyValue;
internal SafeHmacCtxHandle _hmacCtx;
[DllImport("native", EntryPoint = "CryptoNative_HmacFinal")]
internal extern static int HmacFinal(SafeHmacCtxHandle ctx, ref byte data, ref int length);
public HMACSHA1()
{
_hmacCtx = new SafeHmacCtxHandle ();
KeyValue = new byte[1];
}
public void TryFinalizeHashAndReset(Span<byte> destination, out int bytesWritten)
{
int len2 = 0;
HmacFinal(_hmacCtx, ref MemoryMarshal.GetReference(destination), ref len2);
bytesWritten = 0;
}
public void Dispose()
{
}
}
class Program
{
static void Main(string[] args)
{
Console.WriteLine ("X");
while (true) {
Parallel.For(0, 1000, j => {
using (var _hmac = new HMACSHA1()) {
Span<byte> uiSpan = stackalloc byte[20];
GC.Collect();
int bytesWritten;
_hmac.TryFinalizeHashAndReset(uiSpan, out bytesWritten);
GC.Collect();
}
});
}
}
}
}
This contains only one pinvoke, creates the safe handle in managed code, still fails the same way.
vargaz
on 24 Sep 2019
Another version:
using System;
using System.Threading;
using System.Threading.Tasks;
using System.Runtime.InteropServices;
using System.Diagnostics;
sealed class SafeHmacCtxHandle : SafeHandle
{
public SafeHmacCtxHandle() :
base(IntPtr.Zero, ownsHandle: true) {
}
protected override bool ReleaseHandle() {
return true;
}
public override bool IsInvalid {
get { return handle == IntPtr.Zero; }
}
}
public class HMACSHA1 : IDisposable
{
protected byte[] KeyValue;
internal SafeHmacCtxHandle _hmacCtx;
[DllImport("native", EntryPoint = "CryptoNative_HmacFinal")]
internal extern static int HmacFinal(SafeHmacCtxHandle ctx, ref byte data, ref int length);
public HMACSHA1() {
_hmacCtx = new SafeHmacCtxHandle ();
KeyValue = new byte[1];
}
public void Doit(int destination, int bytesWritten) {
int len2 = 0;
byte b = 0;
HmacFinal(_hmacCtx, ref b, ref len2);
}
public void Dispose() {
}
static void Main(string[] args) {
Console.WriteLine ("X");
while (true) {
Parallel.For(0, 1000, j => {
using (var hmac = new HMACSHA1()) {
GC.Collect();
int bytesWritten = 0 ;
hmac.Doit(bytesWritten, bytesWritten);
GC.Collect();
}
});
}
}
}
This one doesn't contain Span's anymore.
vargaz
on 24 Sep 2019
Here's a version with P/Invoke in only one thread:
using System;
using System.Threading;
using System.Threading.Tasks;
using System.Runtime.InteropServices;
using System.Diagnostics;
sealed class SafeHmacCtxHandle : SafeHandle
{
public SafeHmacCtxHandle() :
base(IntPtr.Zero, ownsHandle: true) {
}
protected override bool ReleaseHandle() {
return true;
}
public override bool IsInvalid {
get { return handle == IntPtr.Zero; }
}
}
public class HMACSHA1
{
internal SafeHmacCtxHandle _hmacCtx;
[DllImport("native", EntryPoint = "CryptoNative_HmacFinal")]
internal extern static int HmacFinal(SafeHmacCtxHandle ctx, ref byte data, ref int length);
public HMACSHA1() {
_hmacCtx = new SafeHmacCtxHandle ();
var y = new byte[1];
}
public void Doit(int destination, int bytesWritten) {
int len2 = 0;
byte b = 0;
HmacFinal(_hmacCtx, ref b, ref len2);
}
static void Main(string[] args) {
var t = new Thread(() => { while (true) { GC.Collect(); var x = new byte[1]; Thread.Yield(); } });
t.Start();
while (true) {
var hmac = new HMACSHA1();
var x = new byte[1];
hmac.Doit(0, 0);
}
}
}
Can we re-label the issue? area-Runtime: GC + area-Runtime: Interop seems more appropriate. (/cc @marek-safar)
filipnavara
on 24 Sep 2019
@vargaz are you looking into this issue?
marek-safar
on 24 Sep 2019
I tried recompiling on the same system with clang -O2 and the problem goes away. I can try to narrow it down to some specific optimization or file but it's pretty time consuming.
Update: I failed to narrow it to specific optimization. Specifying individual optimization passes that should be equal to -O1 produces different code from -O1 that doesn't exhibit the error. I also tried updating to GCC 7.4 (gcc-7 (Ubuntu 7.4.0-1ubuntu1~16.04~ppa1) 7.4.0) and the issue persists.
filipnavara
on 25 Sep 2019
We followed up on the issue on Gitter and I'd like to write down the summary for further reference. Firstly, big thanks to @vargaz, @BrzVlad and @jaykrell for helping to narrow the problem down and for offering a plausible explanation for the bug.
After some deliberation we narrowed down the problem to a single function - mono_threads_enter_gc_safe_region_unbalanced_with_info - that behaves differently when compiled with optimizations.
When the GC safe mode is being entered Mono saves the context with current register values to the thread info. This information is used by the GC for pinning objects referenced by the registers or the stack.
If the information saved in the context is incorrect the GC may not realize that some object (SafeHandle in this case) needs to be pinned and it will happily move it as the heap is compacted. This results in the original object still being referenced but the memory is already reclaimed and can be reused for other allocations.
So, how would the context information get wrong values? And why does it happen only with GCC and optimizations turned on?
The answer lies on this line. The register values are saved to the context very late in the process of the transition to the GC safe state. We need to ensure that all "callee saved" registers properly represent the state at the time the state transition happens but at this point we are already three calls down the stack in a middle of the function.
The compiler is allowed to save some of these registers to a stack and restore them only on the function exit. By the time the registers are saved in mono_thread_state_init some of the original values may be pushed on the stack and they will not be present in the registers. Once we return from mono_threads_enter_gc_safe_region_unbalanced_with_info the stack is unwound and the references disappear. We end up in an inconsistent state where all the registers that the compiler decided to save to the stack are suddenly invisible to the garbage collector.
The fix is not obvious. You can guarantee internally consistent stack + context if you wrap a frame around the whole P/Invoke and ensure that we never go back in the stack until we return from the GC safe mode. In fact, by the nature of stack scanning it would not even be necessary to ensure the correct context being captured because just saving it on the stack would be sufficient.
I tried to start a less intrusive fix in https://github.com/mono/mono/pull/17110 where I ensure the context is captured as soon as possible. By its nature it's still not safe but it works with current GCC/Clang compilers.
filipnavara
on 28 Sep 2019
👀1
For the record, both the optimized and unoptimized version of mono_threads_enter_gc_safe_region_unbalanced_with_info clobbers one of the registers. It just happens to be a different one.
The prolog of -O0 version:
123: 55 push %rbp
124: 48 89 e5 mov %rsp,%rbp
127: 53 push %rbx
128: 48 83 ec 08 sub $0x8,%rsp
The prolog of -O1 version:
11a: 55 push %rbp
11b: 48 89 e5 mov %rsp,%rbp
11e: 48 83 ec 10 sub $0x10,%rsp
122: 48 89 7d f8 mov %rdi,-0x8(%rbp)
https://github.com/mono/mono/pull/17150 seems to resolve the issue, at least using the above repro running locally on Ubuntu 16.04.
lateralusX
on 2 Oct 2019
I haven't seen the SafeHandle failure on our mono-netcore CI runs anymore after #17150 was merged. Closing :)
akoeplinger
on 7 Oct 2019
Fair enough. I'll open a new issue for the lack of unit tests for GC safe/unsafe transition :)
filipnavara
on 7 Oct 2019
Was this page helpful?
0 / 5 - 0 ratings
Related issues
kumpera
路
3Comments
alxchk
路
4Comments
decriptor
路
4Comments
mkArtak
路
3Comments
NattyNarwhal
路
4Comments