Runtime: Infrastructure - Status/Health

[jashook]

Overview

Please use these queries to discover issues

• Blocking CI
• Blocking Official Build
• Blocking CI Optional
• Blocking Outerloop

Blocking CI

|Status|Issue|Build Count|
|---|---|---|
|:fire:|OSX machines are de-provisioned during CI / PR runs leading to failures|130|
|:fire:|Inability to unzip assets during build on Unix x64|121|
|:fire:|Builds legs getting abandoned due to agent notification issues|64|
|:fire:| System.Threading.Tasks.Tests timed out on net5.0-Linux-Debug-arm64-Mo ...|63|
|:fire:|HTTP2 test intermittently failing CI|56|
|:fire:|Can_Run_App_With_StatiHost installer test failure in CI|41|
|:fire:|Bundle_Is_Extracted test failing intermittently on Linux|26|
|:fire:|Bundle_extraction_is_reused intermittently fails with "text file is busy"|10|
|:fire:|System.Diagnostics.Process.Tests timing out on MacOS with Mono|9|
|:warning:|Errors installing the SDK during builds|9|
|:fire:|ClientAndServer_OneOrBothUseDefault_Ok - fails with NullReferenceException|7|
|:fire:|System.Net.Sockets.Tests.LocalEndPointTestIPv4Sync.TcpAcceptSocket_Whe ...|2|
|:fire:|QuaternionTests failures in release/5.0 on ARM64 CoreCLR Checked Alpine.|2|
|:fire:|Test failure: System.Numerics.Tests.Vector3Tests.Vector3ReflectTest|2|
|:fire:|Build iOS x64 Release AllSubsets_Mono failing in all CI jobs |N/A|
|:fire:|Failing test - GC/Features/HeapExpansion/bestfit-finalize/bestfit-fina ...|N/A|
|:fire:|Disabled Job - Libraries Test Run release mono windows x64 Release|N/A|
|:fire:|System.Collections.Concurrent.Tests crashing in CI|N/A|
|:fire:|ios test run on simulator got cancelled|N/A|
|:fire:|Test Failure JIT.jit64 JIT.Methodical on windows arm64 Checked no_tier ...|N/A|
|:fire:|Test failure: System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_ ...|N/A|
|:fire:|Some iOS tests are failing to launch test app|N/A|
|:fire:|build failure: Failed to download or parse releases-index.json with error|N/A|
|:fire:|InterruptInFinallyBlockTest_SkipOnDesktopFramework failed on Linux Deb ...|N/A|
|:fire:|Make mono runtime tests stop using patching|N/A|
|:fire:|System.Threading.Tasks.Tests.TaskAwaiterTests timeout: Libraries Test ...|N/A|
|:fire:|APK deletion/installation failure: Android arm64 Release AllSubsets_Mono|N/A|
|:fire:|Segfault during restore: CoreCLR Product Build Linux_musl arm release|N/A|
|:fire:|runtime-live-build timed out|N/A|
|:fire:|ConnectAsync_CancellationRequestedAfterConnect_ThrowsOperationCanceled ...|N/A|
|:fire:|Work items crashing on arm64-mono_interpreter|N/A|
|:fire:|Tests timing out with incomplete log|N/A|
|:fire:|System.Net.Mail.Tests.SmtpClientTest.TestMailDeliveryAsync hung|N/A|
|:fire:|SafeHandle use-after-dispose in FileSystemWatcher on OSX|N/A|
|:fire:|Exception in test infrastructure - System.InvalidOperationException: C ...|N/A|

Blocking Official Build

|Status|Issue|Build Count|
|---|---|---|
|:fire:|OSX machines are de-provisioned during CI / PR runs leading to failures|130|
|:warning:|Errors installing the SDK during builds|9|

Blocking CI Optional

|Status|Issue|Build Count|
|---|---|---|

Blocking Outerloop

|Status|Issue|Build Count|
|---|---|---|
|:fire:|Runtime Tests Linux_musl arm fails with "The "FindDotNetCliPackage" ta ...|43|
|:fire:|DirectoryInfoWrapper throws DirectoryNotFoundException when directory ...|N/A|
|:fire:|Test failure: readytorun\crossgen2\crossgen2smoke_donotalwaysusecros ...|N/A|
|:fire:|Test failure: baseservices\threading\threadpool\unregister\unregis ...|N/A|

Goals

1. A minimum 95% passing rate for the runtime pipeline

Resources

1. runtime pipeline analytics

[jashook]

/cc @dotnet/coreclr-infra

[jashook]

/cc @dotnet/jit-contrib

[jkotas]

https://github.com/dotnet/coreclr/issues/26057 Failed to resolve SDK 'Microsoft.DotNet.Helix.Sdk'

[jkotas]

dotnet/coreclr#27453 Test Infrastructure Failure: Access to the path ... is denied

[jashook]

Summary of the week of 21-Oct-2019

Problems (cross out signifies the problem is fixed)

1. OSX build machines failing to take work
2. Failed to resolve SDK 'Microsoft.DotNet.Helix.Sdk' dotnet/coreclr#26057
3. Linux arm musl build fails
4. Clang 5.0 throws a stack trace occasionally when building arm or arm64 targets

[sandreenko]

@jashook does this issue need a new owner while you are on vacation?

[jashook]

I assume the ownership is now shared.

/cc @trylek @ViktorHofer @jkoritzinsky @dagood @jaredpar

[jkotas]

Libraries Build Windows_NT x86 Release leg is failing 100% of time. See https://github.com/dotnet/runtime/pull/967

[trylek]

Interesting. To me it looks like a code issue rather than an infra hiccup though:

Fatal error. 0xC0000005
   at DynamicClass.WriteTypeWithDateTimeOffsetTypePropertyToJson(System.Runtime.Serialization.XmlWriterDelegator, System.Object, System.Runtime.Serialization.Json.XmlObjectSerializerWriteContextComplexJson, System.Runtime.Serialization.ClassDataContract, System.Xml.XmlDictionaryString[])
   at System.Runtime.Serialization.Json.JsonClassDataContract.WriteJsonValueCore(System.Runtime.Serialization.XmlWriterDelegator, System.Object, System.Runtime.Serialization.Json.XmlObjectSerializerWriteContextComplexJson, System.RuntimeTypeHandle)
   at System.Runtime.Serialization.Json.JsonDataContract.WriteJsonValue(System.Runtime.Serialization.XmlWriterDelegator, System.Object, System.Runtime.Serialization.Json.XmlObjectSerializerWriteContextComplexJson, System.RuntimeTypeHandle)
   at System.Runtime.Serialization.Json.DataContractJsonSerializerImpl.WriteJsonValue(System.Runtime.Serialization.Json.JsonDataContract, System.Runtime.Serialization.XmlWriterDelegator, System.Object, 

Perhaps some traditional shenanigans regarding time zone settings on the test machines?

[ViktorHofer]

cc @ahsonkhan @steveharter

[mikedn]

That looks like it came from my #737, I'm pretty sure I know what the problem is. The odd thing is that the CI was green when the PR was merged and it's not very clear why.

[henrikse55]

Looks like the same CI (Libraries Build Windows_NT x86 Release) failed in my PR #842

https://helix.dot.net/api/2019-06-17/jobs/068da4f0-9282-4e18-a1de-c2baaecf32b0/workitems/System.Runtime.Serialization.Json.Tests/console

Fatal error. 0xC0000005
   at DynamicClass.WriteTypeWithDateTimeOffsetTypePropertyToJson(System.Runtime.Serialization.XmlWriterDelegator, System.Object, System.Runtime.Serialization.Json.XmlObjectSerializerWriteContextComplexJson, System.Runtime.Serialization.ClassDataContract, System.Xml.XmlDictionaryString[])
   at System.Runtime.Serialization.Json.JsonClassDataContract.WriteJsonValueCore(System.Runtime.Serialization.XmlWriterDelegator, System.Object, System.Runtime.Serialization.Json.XmlObjectSerializerWriteContextComplexJson, System.RuntimeTypeHandle)
   at System.Runtime.Serialization.Json.JsonDataContract.WriteJsonValue(System.Runtime.Serialization.XmlWriterDelegator, System.Object, System.Runtime.Serialization.Json.XmlObjectSerializerWriteContextComplexJson, System.RuntimeTypeHandle)

[safern]

That looks like it came from my #737, I'm pretty sure I know what the problem is. The odd thing is that the CI was green when the PR was merged and it's not very clear why.

The reason why your PR was green is because of the current state we’re in.

1. In order to achieve building live live, we needed to disable running the libraries tests on coreclr PRs.
2. The only pipeline that runs libraries tests is runtime-libraries which is conditioned to only run when the change includes a change to src/libraries/*. Since your change only touched coreclr, it didn’t run in your PR.

I’m working on fixing this and moving to a single pipeline that always run and libraries tests should run always when coreclr or libraries are touched.

[jkotas]

CoreCLR Test Run Windows_NT arm legs are failing 100% for all PRs currently (https://github.com/dotnet/runtime/issues/1097)

[jkotas]

129 Non-deterministic failure hit by CoreCLR tests: Assert failure(PID 2664 [0x00000a68], Thread: 3612 [0x0e1c]): pMethodDesc->GetCallCounter()->IsCallCountingEnabled(pMethodDesc)

[safern]

Updated issue for failure in CoreCLR Pri0 Test Run Windows_NT x64 checked tests timing out:

cmdLine:C:\h\w\A995095C\w\B9DA09DC\e\tracing\eventpipe\providervalidation\providervalidation\providervalidation.cmd Timed Out
      Test Harness Exitcode is : -100
      To run the test:
      > set CORE_ROOT=C:\h\w\A995095C\p
      > C:\h\w\A995095C\w\B9DA09DC\e\tracing\eventpipe\providervalidation\providervalidation\providervalidation.cmd
      Expected: True
      Actual:   False

cc: @josalem is working on fixing it.

[jkotas]

https://github.com/dotnet/runtime/issues/2209: Unable to pull image mcr.microsoft.com/...

[jkotas]

A lot of PRs are failing with:
Unhandled exception. System.IO.FileLoadException: Could not load file or assembly 'System.Runtime.CompilerServices.Unsafe, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The located assembly's manifest definition does not match the assembly reference. (0x80131040 (FUSION_E_REF_DEF_MISMATCH))\nFile name: 'System.Runtime.CompilerServices.Unsafe, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'

https://github.com/dotnet/runtime/pull/2344 is reverting the change that introduced the problem.

[safern]

I just saw an issue related to helix in one of my PRs:

https://github.com/dotnet/core-eng/issues/8694

Adding to the description.

[jkotas]

It is not convenient to keep updating this issue with all intermittent test failures hit by the CI. I have started marking issues that are intermittently causing CI failures with blocking-clean-ci label, This label did exist in the repo, but it was not used for a while - time to start using it again.

Query: https://github.com/dotnet/runtime/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+label%3Ablocking-clean-ci+

[jaredpar]

It is not convenient to keep updating this issue with all intermittent test failures hit by the CI

Agree. Going forward I would prefer this be more of a status page for the repository. A place to visit to quickly check if you're running into a known issue and link to a place to find more information.

I have started marking issues that are intermittently causing CI failures with blocking-clean-ci label, This label did exist in the repo, but it was not used for a while - time to start using it again.

+1

[sandreenko]

Link #32835

[danmosemsft]

Unpinning for a bit

[jaredpar]

@danmosemsft why? This is pinned so that devs can find active infra issues easily.

[danmosemsft]

Because we can only pin 3 issues and I added a new one. Which do you want to drop? :)

[jkotas]

FWIW, I find the permanently pinned issues distracting. I am actively forcing myself to avoid clicking on the "x" button because it would unpin the issue for everybody. I have done it several times by accident. Muscle memory: you see "x" next to a thing that you do not want to see anymore, so you automatically click it to make it go away. I wish github allowed me to hide the pinned issues that I have seen hundred times already.

[danmosemsft]

I hear you - I don't know we have any better way to communicate with the community. Unless we add something to the top of the readme, which may not be noticed.

[jkotas]

I am wondering who is the target audience for the pinned issues. The pinned issues communicate the following currently:

• We have 12 overarching themes that we are working on in .NET 5. Should this rather be mentioned next to the roadmap link in the readme?
• We have 30 different sources of CI and official build flakiness
• We are using 5.0 as the milestone for .NET 5.0 issues
• We are going to rename the test directory in two weeks

Are these the most important things we want everybody in the community to know?

[danmosemsft]

😁
Maybe the stickies should only be for announcements (sticks around for a week or two only) and anything else should be linked from the readme. My guess is that nobody reads the readme after they've read it once, of course.

[sandreenko]

Looks like outerloop job is not in a good shape right now (pipeline),
all Linux_musl x64/arm64 send tests are failing with:

2020-08-22T21:30:49.2246346Z   Uploading payloads for Job on (Alpine.312.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-20200602002622-e06dc59...
2020-08-22T21:30:49.2279177Z /__w/1/s/.packages/microsoft.dotnet.helix.sdk/5.0.0-beta.20407.3/tools/Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(47,5): error : Correlation Payload '/__w/1/s/artifacts/tests/coreclr/Linux_musl.x64.Checked/Tests/Core_Root/' not found. [/__w/1/s/src/coreclr/tests/helixpublishwitharcade.proj]
2020-08-22T21:30:49.2404592Z ##[error].packages/microsoft.dotnet.helix.sdk/5.0.0-beta.20407.3/tools/Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(47,5): error : (NETCORE_ENGINEERING_TELEMETRY=Build) Correlation Payload '/__w/1/s/artifacts/tests/coreclr/Linux_musl.x64.Checked/Tests/Core_Root/' not found.

I have not found a separate issue about that, could somebody take a look?

[trylek]

I'm taking a look now.

[ViktorHofer]

@jaredpar I don't think we have updated this issue for quite some time. Do we have some better to track now? Should we close this?

[jaredpar]

The issue itself gets updated about every 15 minutes. The reason it appears static is that we're not following up on the individual issues here: closing for instance when the issues are resolved. My first instinct would be to focus on that.

[ViktorHofer]

Great to hear that the issue is still updated. Should we pin it again after closing stale issues?

[jaredpar]

Think so.