Runtime: Runtime Tests Linux_musl arm fails with "The "FindDotNetCliPackage" task failed unexpectedly."

[sandreenko]

Full log:

  Starting Azure Pipelines Test Run CoreCLR Linux arm Checked no_tiered_compilation @ (Alpine.312.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.12-helix-arm32v7-20200908125213-5bece88
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20560.1/tools/dotnet-cli/DotNetCli.targets(5,5): error MSB4018: The "FindDotNetCliPackage" task failed unexpectedly. [/__w/1/s/src/tests/Common/helixpublishwitharcade.proj]
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20560.1/tools/dotnet-cli/DotNetCli.targets(5,5): error MSB4018: System.Net.Http.HttpRequestException: Response status code does not indicate success: 404 (Not Found). [/__w/1/s/src/tests/Common/helixpublishwitharcade.proj]
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20560.1/tools/dotnet-cli/DotNetCli.targets(5,5): error MSB4018:    at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode() [/__w/1/s/src/tests/Common/helixpublishwitharcade.proj]
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20560.1/tools/dotnet-cli/DotNetCli.targets(5,5): error MSB4018:    at Microsoft.DotNet.Helix.Sdk.FindDotNetCliPackage.ExecuteAsync() in /_/src/Microsoft.DotNet.Helix/Sdk/FindDotNetCliPackage.cs:line 61 [/__w/1/s/src/tests/Common/helixpublishwitharcade.proj]
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20560.1/tools/dotnet-cli/DotNetCli.targets(5,5): error MSB4018:    at Microsoft.DotNet.Helix.Sdk.FindDotNetCliPackage.Execute() in /_/src/Microsoft.DotNet.Helix/Sdk/FindDotNetCliPackage.cs:line 45 [/__w/1/s/src/tests/Common/helixpublishwitharcade.proj]
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20560.1/tools/dotnet-cli/DotNetCli.targets(5,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [/__w/1/s/src/tests/Common/helixpublishwitharcade.proj]
/__w/1/s/.packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20560.1/tools/dotnet-cli/DotNetCli.targets(5,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [/__w/1/s/src/tests/Common/helixpublishwitharcade.proj]
##[error].packages/microsoft.dotnet.helix.sdk/6.0.0-beta.20560.1/tools/dotnet-cli/DotNetCli.targets(5,5): error MSB4018: (NETCORE_ENGINEERING_TELEMETRY=Helix) The "FindDotNetCliPackage" task failed unexpectedly.
System.Net.Http.HttpRequestException: Response status code does not indicate success: 404 (Not Found).
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at Microsoft.DotNet.Helix.Sdk.FindDotNetCliPackage.ExecuteAsync() in /_/src/Microsoft.DotNet.Helix/Sdk/FindDotNetCliPackage.cs:line 61
   at Microsoft.DotNet.Helix.Sdk.FindDotNetCliPackage.Execute() in /_/src/Microsoft.DotNet.Helix/Sdk/FindDotNetCliPackage.cs:line 45
   at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()

An example:
https://dev.azure.com/dnceng/public/_build/results?buildId=882549&view=logs&j=22eeac8c-742e-5aad-c88c-3907a882a5f0&t=8e36944f-99e4-5f3f-4391-f6a5a224ec79

It looks like the last item that blocks at least some green runs after we disabled individual failing tests, should we disable the job?

Link #702

Runfo Tracking Issue: Linux Musl ARM Outerloop can't find dotnet cli package
|Definition|Build|Kind|Job Name|
|---|---|---|---|
|runtime-coreclr outerloop|883740|PR 43568|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|883740|PR 43568|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|883616|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|883616|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|883329|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|883329|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|883054|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|883054|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|882700|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|882700|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|882549|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|882549|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|882457|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|882457|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|882395|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|882395|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|882355|PR 43024|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|882355|PR 43024|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|882296|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|882296|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|882019|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|882019|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|881709|PR 44398|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|881709|PR 44398|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|881709|PR 44398|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|881709|PR 44398|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|881617|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|881617|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|881165|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|881165|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|880789|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|880789|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|880667|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|880667|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|880556|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|880556|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|880421|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|880421|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|880132|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|880132|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|879993|PR 44420|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|879993|PR 44420|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|879839|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|879839|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|879663|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|879663|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|879361|PR 44404|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|879361|PR 44404|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|879292|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|879292|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|878969|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|878969|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|878747|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|878747|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|878616|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|878616|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|878518|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|878518|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|878313|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|878313|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|878101|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|878101|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|877802|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|877802|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|877700|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|877700|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|877608|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|877608|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|877458|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|877458|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|877408|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|877408|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|877217|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|877217|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|876908|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|876908|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|876688|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|876688|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|876501|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|876501|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|876259|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|876259|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|876144|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|876144|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|875915|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|
|runtime-coreclr outerloop|875915|Rolling|CoreCLR Pri1 Runtime Tests Run Linux_musl arm checked|

Build Result Summary
|Day Hit Count|Week Hit Count|Month Hit Count|
|---|---|---|
|0|0|25|

[sandreenko]

PTAL @tommcdon @trylek @dotnet/runtime-infrastructure

[hoyosjs]

Turning this into a tracking issue. It seems to only affect linux-musl-arm outerloop pipelines, but there's 106 hits over 7 days, so it seems somewhat deterministic.

[trylek]

Hmm, that's weird. @janvorli, it looks like we're actually running the linux_musl arm tests in outerloop runs only, not in PR / CI runs. Did you try to run the outerloop pipeline when introducing the leg i.o.w. do we know if this ever worked?

[safern]

It seems like it is trying to download it from: https://dotnetcli.azureedge.net/dotnet/Runtime/5.0.0-rc.2.20475.5/dotnet-runtime-5.0.0-rc.2.20475.5-linux-musl-arm.tar.gz

I wonder if we shipped a runtime for linux-musl-arm for that version of the runtime.

[janvorli]

We are running the libraries linux_musl_arm tests in the CI. I can see the leg in the recent CIs. It seems that run is executed only if coreclr was changed.

[janvorli]

Here is an example of a log from such a test run:
https://dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_apis/build/builds/872722/logs/1053

[safern]

This works: https://dotnetcli.azureedge.net/dotnet/Runtime/5.0.0/dotnet-runtime-5.0.0-linux-musl-arm.tar.gz

It seems like we're using the BundledNetCoreAppVersion which is defined by the sdk used to build... in this case 5.0.0-rc.2.*

I suggest changing this: https://github.com/dotnet/runtime/blob/b37f10a504f20faef2558b6d79180411ad0603c3/src/tests/Common/helixpublishwitharcade.proj#L97 to 5.0.0 in the meantime until we update our SDK to use 5.0.0 in our global.json

[safern]

Btw, @janvorli, it seems like Alpine ARM is not included in the release notes for 5.0.0: https://github.com/dotnet/core/blob/master/release-notes/5.0/5.0.0/5.0.0.md#net-500---november-10-2020

[trylek]

@janvorli - I believe the log you mention corresponds to a "libraries" Linux_musl arm run, the run that's failing in the outerloop pipeline is a "coreclr" Linux_musl arm run; these two still use somewhat separate scripts and so the properties are likely forked, i.o.w. we should probably do what Santi suggests for the time being.

[janvorli]

That's what I said - we are running only the libraries tests in the CI.

[trylek]

I see, sorry about missing that bit in your previous response.

[janvorli]

it seems like Alpine ARM is not included in the release notes for 5.0.0:

It is intentional, I have found only last week that the aspnetcore packages are not built for 5.0. I have thought I've seen 5.0-RC2 ones being produced after I've merged in my change to aspnetcore, but I've obviously made a mistake and seen something else. So we will fix that for 5.0.1.

[safern]

Sounds good. Didn't know if it was intentional or missed. Thanks for the update.