Runtime: System.Security.Cryptography.OpenSsl.Tests failing on CI runs

Created on 25 Jan 2020 · 32Comments · Source: dotnet/runtime

Tests are crashing on several OSX runs

===========================================================================================================
/private/tmp/helix/working/BB380A7A/w/CADB0A98/e /private/tmp/helix/working/BB380A7A/w/CADB0A98/e
  Discovering: System.Security.Cryptography.OpenSsl.Tests (method display = ClassAndMethod, method display options = None)
No usable version of libssl was found
./RunTests.sh: line 161: 65451 Abort trap: 6           "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Security.Cryptography.OpenSsl.Tests.runtimeconfig.json --depsfile System.Security.Cryptography.OpenSsl.Tests.deps.json xunit.console.dll System.Security.Cryptography.OpenSsl.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing -notrait category=nonnetcoreapptests -notrait category=nonosxtests $RSP_FILE
/private/tmp/helix/working/BB380A7A/w/CADB0A98/e
----- end Thu Jan 23 05:44:06 PST 2020 ----- exit code 134 ----------------------------------------------------------
exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.

Builds

Test Runs

netcoreapp5.0-OSX-Release-x64-CoreCLR_release-OSX.1013.Amd64.Open
netcoreapp5.0-OSX-Release-x64-CoreCLR_release-OSX.1014.Amd64.Open
netcoreapp5.0-OSX-Release-x64-Mono_release-OSX.1014.Amd64.Open

area-Infrastructure blocking-clean-ci untriaged

Source

jaredpar

Most helpful comment

Apologies for the delay in getting this resolved, we had some miscommunication on our side and we are going through the rest of the machines to fix them. ETA is tomorrow morning.

JpratherMS on 6 Feb 2020

🚀2

All 32 comments

This was supposed to be fixed by @ilyas1974 already. OpenSSL was not installed correctly. He installed it now in all the OSX machines. @ilyas1974 could you please follow up in this?

safern on 25 Jan 2020

Yes, I will follow up on what is going on with these systems.

ilyas1974 on 27 Jan 2020

@ilyas1974 I haven't seen this failure anymore, did you follow up? I just want to confirm we can close this issue.

safern on 31 Jan 2020

Apologize for not updating this thread early. DDFUN is currently working on going though all the systems to verify this functionality. Their work item for this task can be for at http://ddweb/TechEase/TicketEdit.aspx?id=1636188. I believe a majority of the systems have OpenSSL configured properly - there are a few that do not (what will be resolved with the associated work item.

ilyas1974 on 31 Jan 2020

Apologize for not updating this thread early.

No problem. Would you mind getting back whenever this is finished so that we can close this issue?

safern on 31 Jan 2020

Hit by #31641

jkotas on 3 Feb 2020

This is being hit in rolling CI builds as well:
https://helix.dot.net/api/2019-06-17/jobs/6afd25d6-b672-4525-bcb3-92be7581046a/workitems/System.Security.Cryptography.OpenSsl.Tests/console

@ilyas1974 what's the status on this?

safern on 3 Feb 2020

Reaching out to DDFUN as they are performing the work to get the current status on this.

ilyas1974 on 3 Feb 2020

We have reached out to the tech and are currently waiting for a reply. I should have an update for you on the current status by tomorrow morning.

/cc @JpratherMS

ilyas1974 on 4 Feb 2020

@JpratherMS, is there an update on the status of the OpenSSL work? Last I heard there were only about 6 systems that were not configured correctly. If we can get that list, I can remove them from rotation until DDFUN is able to fix the OpenSSL issues.

ilyas1974 on 4 Feb 2020

The affected machines have been fixed, please let me know if this issue persists.

JpratherMS on 4 Feb 2020

Thanks @JpratherMS and @ilyas1974 -- I'm going to close this issue and if someone hits it again we can re-open.

safern on 4 Feb 2020

@JpratherMS just happened again on: dci-mac-build-103.local machine. I'll re-open this until it is clear if there are some machines with this issue still.

https://helix.dot.net/api/2019-06-17/jobs/fd9733fb-53cf-41b8-a9ce-f450b21ed229/workitems/System.Security.Cryptography.OpenSsl.Tests/console

safern on 4 Feb 2020

Thanks for reaching out, we found an issue with our process in discovering the ones that are not configured correctly. We are working on resolving that now, I will update again when that is completed.

JpratherMS on 4 Feb 2020

❤1

Data from the last 100 builds (CI and PR)

Builds

|Build|Pull Request | Test Failure Count|
| --- | --- | --- |
|#507937|#1787|10|
|#507961|#31734|1|
|#508096|Rolling|1|
|#508266|#2087|1|
|#508472|#2275|1|
|#508547|#31765|1|
|#508623|#31807|1|
|#508780|#31765|1|
|#508787|#27540|1|
|#508835|#5523|1|
|#508862|#2140|1|
|#508923|#31819|1|
|#509133|#31826|1|
|#509169|#31802|1|
|#509342|#1612|20|
|#509347|#2259|1|
|#509362|Rolling|1|
|#509423|#1787|5|
|#509654|#31838|5|
|#509747|Rolling|1|
|#510085|#31854|1|
|#510294|#31814|1|

Configurations

netcoreapp5.0-OSX-Debug-x64-CoreCLR_checked-OSX.1013.Amd64.Open
netcoreapp5.0-OSX-Debug-x64-CoreCLR_release-OSX.1013.Amd64.Open
netcoreapp5.0-OSX-Debug-x64-CoreCLR_release-OSX.1014.Amd64.Open
netcoreapp5.0-OSX-Debug-x64-Mono_release-OSX.1013.Amd64.Open
netcoreapp5.0-OSX-Debug-x64-Mono_release-OSX.1014.Amd64.Open
netcoreapp5.0-OSX-Release-x64-Mono_release-OSX.1014.Amd64.Open

Helix Logs

jaredpar on 5 Feb 2020

🚀1

I am seeing other System.Security.Cryptography.* tests failing on netcoreapp5.0-OSX-Debug-x64-Mono_release-OSX.1014.Amd64.Open with (what looks like) the same error:
https://helix.dot.net/api/2019-06-17/jobs/6da71858-a172-4863-a07e-c2be8c8dd875/workitems/System.Security.Cryptography.Algorithms.Tests/console

===========================================================================================================
/private/tmp/helix/working/A6850955/w/BD6409D4/e /private/tmp/helix/working/A6850955/w/BD6409D4/e
  Discovering: System.Security.Cryptography.Algorithms.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Security.Cryptography.Algorithms.Tests (found 1234 of 1239 test cases)
  Starting:    System.Security.Cryptography.Algorithms.Tests (parallel test collections = on, max threads = 4)
No usable version of libssl was found

…
…
…

=================================================================
    Managed Stacktrace:
=================================================================
      at <unknown> <0xffffffff>
      at System.Security.Cryptography.AesCcm:Encrypt <0x00112>
      at System.Security.Cryptography.AesCcm:Encrypt <0x00462>
      at System.Security.Cryptography.Algorithms.Tests.AesCcmTests:AesCcmNistTestsTamperCiphertext <0x00312>
      at <Module>:runtime_invoke_void_object <0x000b0>
      at <unknown> <0xffffffff>
      at System.Reflection.RuntimeMethodInfo:InternalInvoke <0x000b8>
      at System.Reflection.RuntimeMethodInfo:Invoke <0x0016a>
      at System.Reflection.MethodBase:Invoke <0x00058>
      at Xunit.Sdk.TestInvoker`1:CallTestMethod <0x0005a>
      at <<InvokeTestMethodAsync>b__1>d:MoveNext <0x0040a>
      at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0010a>
      at System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start <0x00042>
      at <>c__DisplayClass48_1:<InvokeTestMethodAsync>b__1 <0x000de>
      at <AggregateAsync>d__4:MoveNext <0x000f5>
      at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0010a>
      at Xunit.Sdk.ExecutionTimer:AggregateAsync <0x000e2>
      at <>c__DisplayClass48_1:<InvokeTestMethodAsync>b__0 <0x0018a>
      at <RunAsync>d__9:MoveNext <0x000a0>
      at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x00102>
      at Xunit.Sdk.ExceptionAggregator:RunAsync <0x000da>
      at <InvokeTestMethodAsync>d__48:MoveNext <0x003ba>
      at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x00112>
      at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00042>
      at Xunit.Sdk.TestInvoker`1:InvokeTestMethodAsync <0x0010e>
      at Xunit.Sdk.XunitTestInvoker:InvokeTestMethodAsync <0x0013a>
      at <<RunAsync>b__47_0>d:MoveNext <0x00779>
      at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0011a>
      at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00042>
      at Xunit.Sdk.TestInvoker`1:<RunAsync>b__47_0 <0x000e6>
      at <RunAsync>d__10`1:MoveNext <0x000cb>
      at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x00102>
      at Xunit.Sdk.ExceptionAggregator:RunAsync <0x000fa>
      at Xunit.Sdk.TestInvoker`1:RunAsync <0x0012a>
      at Xunit.Sdk.XunitTestRunner:InvokeTestMethodAsync <0x0010a>
      at <InvokeTestAsync>d__4:MoveNext <0x00210>
      at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x00112>
      at Xunit.Sdk.XunitTestRunner:InvokeTestAsync <0x000ea>
      at <>c__DisplayClass43_0:<RunAsync>b__0 <0x00043>
      at <RunAsync>d__10`1:MoveNext <0x000ab>
      at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0010a>
      at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00042>
      at Xunit.Sdk.ExceptionAggregator:RunAsync <0x0015e>
      at <RunAsync>d__43:MoveNext <0x005b2>
      at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0011a>
      at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00042>
      at Xunit.Sdk.TestRunner`1:RunAsync <0x0010e>
      at <RunTestAsync>d__11:MoveNext <0x00242>
      at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0012a>
      at Xunit.Sdk.XunitTheoryTestCaseRunner:RunTestAsync <0x000c2>
      at <RunAsync>d__19:MoveNext <0x003ca>
      at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x00112>
      at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00042>
      at Xunit.Sdk.TestCaseRunner`1:RunAsync <0x0010e>
      at Xunit.Sdk.XunitTheoryTestCase:RunAsync <0x000ea>
      at Xunit.Sdk.XunitTestMethodRunner:RunTestCaseAsync <0x000bd>
      at <RunTestCasesAsync>d__32:MoveNext <0x001f5>
      at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0011a>
      at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00042>
      at Xunit.Sdk.TestMethodRunner`1:RunTestCasesAsync <0x0010e>
      at <RunAsync>d__31:MoveNext <0x001f8>
      at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0010a>
      at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00042>
      at Xunit.Sdk.TestMethodRunner`1:RunAsync <0x00106>
      at Xunit.Sdk.XunitTestClassRunner:RunTestMethodAsync <0x0012a>
      at <RunTestMethodsAsync>d__38:MoveNext <0x00b00>
      at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0012a>
      at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00042>
      at Xunit.Sdk.TestClassRunner`1:RunTestMethodsAsync <0x0011e>
      at <RunAsync>d__37:MoveNext <0x00405>
      at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x00112>
      at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00042>
      at Xunit.Sdk.TestClassRunner`1:RunAsync <0x0010e>
      at Xunit.Sdk.XunitTestCollectionRunner:RunTestClassAsync <0x00132>
      at <RunTestClassesAsync>d__28:MoveNext <0x004ef>
      at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0011a>
      at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00042>
      at Xunit.Sdk.TestCollectionRunner`1:RunTestClassesAsync <0x0010e>
      at <RunAsync>d__27:MoveNext <0x00402>
      at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x00112>
      at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00042>
      at Xunit.Sdk.TestCollectionRunner`1:RunAsync <0x0010e>
      at Xunit.Sdk.XunitTestAssemblyRunner:RunTestCollectionAsync <0x0010a>
      at <>c__DisplayClass14_2:<RunTestCollectionsAsync>b__2 <0x00067>
      at System.Threading.Tasks.Task`1:InnerInvoke <0x00097>
      at <>c:<.cctor>b__274_0 <0x00036>
      at System.Threading.ExecutionContext:RunInternal <0x00118>
      at System.Threading.Tasks.Task:ExecuteWithThreadLocal <0x00312>
      at System.Threading.Tasks.Task:ExecuteEntry <0x000f2>
      at <>c:<.cctor>b__8_0 <0x0006a>
      at Xunit.Sdk.MaxConcurrencySyncContext:RunOnSyncContext <0x0006e>
      at <>c__DisplayClass11_0:<WorkerThreadProc>b__0 <0x0005a>
      at System.Threading.ExecutionContext:RunInternal <0x00118>
      at System.Threading.ExecutionContext:Run <0x00072>
      at System.Object:lambda_method2 <0x0008a>
      at Xunit.Sdk.ExecutionContextHelper:Run <0x00081>
      at Xunit.Sdk.MaxConcurrencySyncContext:WorkerThreadProc <0x00262>
      at <>c:<QueueUserWorkItem>b__5_0 <0x00076>
      at System.Threading.Tasks.Task:InnerInvoke <0x000b5>
      at <>c:<.cctor>b__274_0 <0x00036>
      at System.Threading.ExecutionContext:RunInternal <0x00118>
      at System.Threading.Tasks.Task:ExecuteWithThreadLocal <0x00312>
      at System.Threading.Tasks.Task:ExecuteEntryUnsafe <0x000ba>
      at <>c:<.cctor>b__10_0 <0x0006a>
      at System.Threading.Thread:StartCallback <0x0012c>
      at System.Object:runtime_invoke_void__this__ <0x000b0>
=================================================================
./RunTests.sh: line 161: 32067 Abort trap: 6           (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Security.Cryptography.Algorithms.Tests.runtimeconfig.json --depsfile System.Security.Cryptography.Algorithms.Tests.deps.json xunit.console.dll System.Security.Cryptography.Algorithms.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing -notrait category=nonnetcoreapptests -notrait category=nonosxtests $RSP_FILE
/private/tmp/helix/working/A6850955/w/BD6409D4/e
----- end Tue Feb 4 18:00:28 PST 2020 ----- exit code 134 ----------------------------------------------------------
exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.

From https://github.com/dotnet/runtime/pull/31772 Libraries Test Run release mono OSX x64 Debug leg in https://dev.azure.com/dnceng/public/_build/results?buildId=507900&view=logs&j=c6f8dc49-92a1-5760-c098-ba97b8142bfb&t=22b0078b-0469-5ba6-8725-2121fdbae049

https://dev.azure.com/dnceng/public/_build/results?buildId=507900&view=ms.vss-test-web.build-test-results-tab&runId=16121704&paneView=debug&resultId=172169

ahsonkhan on 5 Feb 2020

Yes it is the same error, mono just crashes in a different way and prints the whole call stack, but it is caused by the same: No usable version of libssl was found.

Thanks @ahsonkhan

safern on 5 Feb 2020

👍1

Looks like right now this is failing 14% of all runtime jobs: CI and PRs. Particularly though it's for OSX10.13 and OSX10.14. Think at this point we need to pull those configurations until we can get the underlying issue fixed.

jaredpar on 6 Feb 2020

Think at this point we need to pull those configurations until we can get the underlying issue fixed.

I was already working on that, talked with @JpratherMS offline and he hopes the work to update the machines will be done by EOD. He will update if so, if it is not ready I'll put up my PR to disable the affected tests on OSX.

safern on 6 Feb 2020

👍1

Apologies for the delay in getting this resolved, we had some miscommunication on our side and we are going through the rest of the machines to fix them. ETA is tomorrow morning.

JpratherMS on 6 Feb 2020

🚀2

If this isn't fixed by EOD, I'd really appreciate disabling the tests. I'm hitting this constantly, even sometimes on multiple re-runs. I hit it just a little while ago, so at least as of this afternoon it is still not fixed.