Runtime: Many Android JIT tests failing in CI

[ViktorHofer]

https://dev.azure.com/dnceng/public/_build/results?buildId=873257&view=ms.vss-test-web.build-test-results-tab

Configuration: Mono Android x64 Release @ Ubuntu.1804.Amd64.Android.Open

@akoeplinger @steveisok is this known?

Runfo Tracking Issue: Many Android JIT tests failing in CI
|Build|Definition|Kind|Run Name|Console|Core Dump|Test Results|Run Client|
|---|---|---|---|---|---|---|---|
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||
|904959|runtime|PR 44560|Mono Android arm64 Release @ Windows.10.Amd64.Android.Open|||||

Displaying 100 of 4613 results

Build Result Summary
|Day Hit Count|Week Hit Count|Month Hit Count|
|---|---|---|
|0|6|71|

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ViktorHofer]

cc @SamMonoRT @vargaz

[msftbot[bot]]

Tagging subscribers to this area:
See info in area-owners.md if you want to be subscribed.

[ViktorHofer]

@danmosemsft the bot isn't mentioning anyone, intentional?

[steveisok]

So, https://dev.azure.com/dnceng/public/_build/results?buildId=873257&view=ms.vss-test-web.build-test-results-tab is a result of an apk installation failure. That impacted 50+ tests on two runs from https://github.com/dotnet/runtime/pull/38457

It looks like work items completed on the machine that failed afterwards, so my guess is that adb is in a bad state and may be corrected upon the work item finishing. We can look into making this more resilient.

The other issues linked in by runfo appear to be different. My guess on some would adb corruption occurring at different points in time.

[danmosemsft]

@danmosemsft the bot isn't mentioning anyone, intentional?

The owner is @vargaz -- @vargaz do you want notifications for this area label?

[akoeplinger]

@ViktorHofer all of the failures from https://github.com/dotnet/runtime/pull/43740 are known, we're iterating on replacing OpenSSL on Android there.

[ViktorHofer]

@steveisok what do you think would be a reasonable follow-up action? The day hit count is concerningly high. Assuming that disabling the Android leg in CI is a no go, should we move the job into a different pipeline that doesn't fail the build: https://github.com/dotnet/runtime/issues/44031?

cc @safern @jkoritzinsky for opinions

[fanyang-mono]

I wonder why this only happened to runtime tests? Theoretically, library tests should be affected as well.

[steveisok]

I'm trying to sort through the recent data first and that may help what a good follow up action will be.

[safern]

I wonder why this only happened to runtime tests? Theoretically, library tests should be affected as well.

I've also seen it by libraries tests as well...

[steveisok]

The libraries failures that I've seen are legit OOM crashes as seen in one adb log

The runtime tests are a bit of a head scratcher, but I think I've seen enough instability across the board where it makes sense to move the simulator runs into a different pipeline.

I think if https://github.com/dotnet/xharness/issues/354 gets fixed, the runs will be better.

[safern]

@steveisok reached out and gathered data, we will be moving the Android builds that use a simulator into the new build we will add for staging until xharness is stable enough to run tests on an Android simulator.

[steveisok]

Alex put up a PR to bump the simulator RAM size to 3GB.

https://dev.azure.com/dnceng/internal/_git/dotnet-helix-machines/pullrequest/11307?_a=files

[ViktorHofer]

Still seeing a day hit count of about 10. What are our mitigation plans at this point? Moving the whole Android suite to the staging pipeline (which doesn't exist yet)?

cc @safern @aik-jahoda

[safern]

What are our mitigation plans at this point? Moving the whole Android suite to the staging pipeline (which doesn't exist yet)?

Yeah. I should have that done by today 😄

[steveisok]

Definitely seeing a bit more instability today. Moving to the staging pipeline will help.

[ViktorHofer]

I see tons of Android failures in many PRs today :(

[safern]

Note that I just merged my staging pipeline PR which moved the Android tests that use the emulator.