Runtime: JIT/Regression/JitBlue/GitHub_7147/GitHub_7147/GitHub_7147.sh

[jashook]

Arm32 innerloop test failure.

Return code:      1
Raw output file:      /root/helix/work/workitem/JIT/Regression/Reports/JIT.Regression/JitBlue/GitHub_7147/GitHub_7147/GitHub_7147.output.txt
Raw output:
BEGIN EXECUTION
/root/helix/work/correlation/corerun GitHub_7147.dll ''
Expected: 100
Actual: 108
END EXECUTION - FAILED
Test Harness Exitcode is : 1
To run the test:
> set CORE_ROOT=/root/helix/work/correlation
> /root/helix/work/workitem/JIT/Regression/JitBlue/GitHub_7147/GitHub_7147/GitHub_7147.sh

This seems to have started happening while the arm32 queue was offline earlier this week.

category:testing
theme:testing
skill-level:beginner
cost:small

[jashook]

/cc @dotnet/jit-contrib

[mikedn]

Hmm, this test comes from my PR that was merged yesterday: dotnet/coreclr#20650

[jashook]

Opened https://github.com/dotnet/coreclr/pull/26336 to disable it on arm32. Seems better then reverting :)

[jashook]

Can I assign the issue to you @mikedn?

[mikedn]

Sure, I'm looking into it.

[mikedn]

I'm not sure what's going on. For
```C#
bool b = false;

        for (int i = 0; i < k; ++i)
        {
            // Sub-expressions "x / y" and "c.f" are both invariant
            // w.r.t. this loop, and can be hoisted.  Since each can
            // raise an exception, and the exceptions have different
            // types, their relative order must be preserved -- the
            // hoisted copy of "x / y" must be evaluated before the
            // hoisted copy of "c.f"
            b = (x / y < i + c.f);
        }

        return b;

I see the following ARM32 being generated:
```asm
G_M63130_IG02:
00000C  2000           movs    r0, 0
00000E  2600           movs    r6, 0
000010  2C00           cmp     r4, 0
000012  DD11           ble     SHORT G_M63130_IG06
000014  4608           mov     r0, r1
000016  4611           mov     r1, r2
000018  F24E 7300      movw    r3, 0xe700
00001C  F6C7 0337      movt    r3, 0x7837
000020  4798           blx     r3               // CORINFO_HELP_DIV

G_M63130_IG03:
000022  686B           ldr     r3, [r5+4]
000024  18F3           adds    r3, r6, r3
000026  4283           cmp     r3, r0
000028  DC01           bgt     SHORT G_M63130_IG04
00002A  2300           movs    r3, 0
00002C  E000           b       SHORT G_M63130_IG05

This looks as expected, the division was hoisted out of the loop so it should throw DivideByZeroException. Yet the 108 result indicates that a different exception was thrown.

I've searched for the test results for my PR and it looks to me that the test has ran and passed:

https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-coreclr-refs-pull-20650-merge-0fc1857600b14e7f97/JIT.Regression/testResults.xml?sv=2018-03-28&sr=c&sig=4u%2BxCxyTPYy3h2tOLrB%2BVdaSOAVpvKNWeW1zrvjdzMY%3D&se=2019-09-01T18%3A51%3A05Z&sp=rl

<collection time="0.514" skipped="0" failed="0" passed="1" total="1" name="Test collection for JIT_Regression._JitBlue_GitHub_7147_GitHub_7147_GitHub_7147_">
<test time="0.5137482" name="JIT/Regression/JitBlue/GitHub_7147/GitHub_7147/GitHub_7147.sh" result="Pass" method="_JitBlue_GitHub_7147_GitHub_7147_GitHub_7147_sh" type="JIT_Regression._JitBlue_GitHub_7147_GitHub_7147_GitHub_7147_">
<output>
<![CDATA[

Return code:      0
Raw output file:      /root/helix/work/workitem/JIT/Regression/Reports/JIT.Regression/JitBlue/GitHub_7147/GitHub_7147/GitHub_7147.output.txt
Raw output:
BEGIN EXECUTION
/root/helix/work/correlation/corerun GitHub_7147.dll ''
Expected: 100
Actual: 100
END EXECUTION - PASSED
Test Harness Exitcode is : 0
To run the test:
> set CORE_ROOT=/root/helix/work/correlation
> /root/helix/work/workitem/JIT/Regression/JitBlue/GitHub_7147/GitHub_7147/GitHub_7147.sh
]]>
</output>
<traits>
<trait name="TestGroup" value="JIT.Regression"/>
</traits>
</test>
</collection>

Is CI working properly? I've seen soo many random failures that I don't exactly trust it.

[BruceForstall]

Is CI working properly? I've seen soo many random failures that I don't exactly trust it.

Well, we _just_ replaced the arm32 machines being used, and @jashook told me the old ones had some problems. So maybe we need to trigger new runs and see what happens?

https://github.com/dotnet/coreclr/pull/26302

[jashook]

This queue change only applies to outerloop runs. We have seen failures in the innerloop CI.

See: https://dev.azure.com/dnceng/public/_build/results?buildId=319621&view=ms.vss-test-web.build-test-results-tab&runId=9348850&resultId=100000&paneView=debug

[jashook]

That being said we have seen successful runs which makes this strange...

[jashook]

Also hit on arm64 https://dev.azure.com/dnceng/public/_build/results?buildId=321130&view=ms.vss-test-web.build-test-results-tab&runId=9345192&paneView=attachments&resultId=100000

[jashook]

No idea why it is intermittent

[jashook]

I have drilled into more test results. This test has been failing intermittently over many runs over all architectures. I believe it should be disabled and investigated.

[BruceForstall]

For instance, I see these failures today:

2019-08-23T00:50:19.236Z    x64 OSX.1013.Amd64.Open Checked-no_tiered_compilation
2019-08-23T00:50:19.642Z    x64 OSX.1013.Amd64.Open Checked
2019-08-23T00:51:33.642Z    x64 OSX.1013.Amd64.Open Checked-no_tiered_compilation
2019-08-23T00:51:34.056Z    x64 OSX.1013.Amd64.Open Checked
2019-08-23T00:52:55.386Z    arm (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226    Checked
2019-08-23T00:52:57.822Z    arm (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226    Checked-no_tiered_compilation
2019-08-23T00:53:34.397Z    arm (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226    Checked-no_tiered_compilation
2019-08-23T00:53:35.273Z    arm (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226    Checked
2019-08-23T00:54:19.077Z    x64 OSX.1013.Amd64.Open Checked
2019-08-23T00:54:19.483Z    x64 OSX.1013.Amd64.Open Checked-no_tiered_compilation
2019-08-23T00:55:18.191Z    x64 OSX.1013.Amd64.Open Checked-no_tiered_compilation
2019-08-23T00:55:18.427Z    x64 OSX.1013.Amd64.Open Checked
2019-08-23T00:55:29.365Z    x64 OSX.1013.Amd64.Open Checked
2019-08-23T00:55:29.769Z    x64 OSX.1013.Amd64.Open Checked-no_tiered_compilation
2019-08-23T00:56:22.917Z    x64 OSX.1013.Amd64.Open Checked-no_tiered_compilation
2019-08-23T00:56:23.784Z    x64 OSX.1013.Amd64.Open Checked
2019-08-23T00:56:28.455Z    arm (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226    Checked
2019-08-23T00:56:29.014Z    arm (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226    Checked-no_tiered_compilation
2019-08-23T05:01:36.454Z    x64 OSX.1013.Amd64.Open Checked-no_tiered_compilation
2019-08-23T05:01:37.704Z    x64 OSX.1013.Amd64.Open Checked
2019-08-23T05:02:20.492Z    x64 OSX.1013.Amd64.Open Checked-no_tiered_compilation
2019-08-23T05:02:20.773Z    x64 OSX.1013.Amd64.Open Checked
2019-08-23T05:02:26.63Z x64 OSX.1013.Amd64.Open Checked
2019-08-23T05:02:27.208Z    x64 OSX.1013.Amd64.Open Checked-no_tiered_compilation
2019-08-23T05:04:47.154Z    arm (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226    Checked
2019-08-23T05:04:47.498Z    arm (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226    Checked-no_tiered_compilation
2019-08-23T05:05:55.534Z    x64 OSX.1013.Amd64.Open Checked
2019-08-23T05:05:57.52Z x64 OSX.1013.Amd64.Open Checked-no_tiered_compilation
2019-08-23T05:06:26.973Z    x64 OSX.1013.Amd64.Open Checked
2019-08-23T05:06:27.976Z    x64 OSX.1013.Amd64.Open Checked-no_tiered_compilation
2019-08-23T05:06:38.713Z    x64 OSX.1013.Amd64.Open Checked-no_tiered_compilation
2019-08-23T05:06:39.73Z x64 OSX.1013.Amd64.Open Checked
2019-08-23T05:08:50.457Z    arm (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226    Checked-no_tiered_compilation
2019-08-23T05:08:50.783Z    arm (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226    Checked
2019-08-23T05:11:49.385Z    arm (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226    Checked-no_tiered_compilation
2019-08-23T05:11:49.681Z    arm (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226    Checked
2019-08-23T08:48:21.946Z    x64 OSX.1013.Amd64.Open Checked-no_tiered_compilation
2019-08-23T08:48:22.571Z    x64 OSX.1013.Amd64.Open Checked
2019-08-23T08:50:41.151Z    x64 (Alpine.38.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.8-helix-09ca40b-20190508143246    Checked-no_tiered_compilation
2019-08-23T08:50:41.981Z    x64 (Alpine.38.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.8-helix-09ca40b-20190508143246    Checked
2019-08-23T08:51:08.444Z    arm (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226    Checked
2019-08-23T08:51:08.85Z arm (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226    Checked-no_tiered_compilation
2019-08-23T08:51:14.66Z x64 OSX.1013.Amd64.Open Checked
2019-08-23T08:51:15.38Z x64 OSX.1013.Amd64.Open Checked-no_tiered_compilation
2019-08-23T08:53:37.682Z    x64 Ubuntu.1804.Amd64.Open  Checked-no_tiered_compilation
2019-08-23T08:53:38.386Z    x64 Ubuntu.1804.Amd64.Open  Checked
2019-08-23T08:53:55.152Z    x64 OSX.1013.Amd64.Open Checked-no_tiered_compilation
2019-08-23T08:53:55.456Z    x64 OSX.1013.Amd64.Open Checked
2019-08-23T08:54:12.412Z    x64 (Alpine.38.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.8-helix-09ca40b-20190508143246    Release
2019-08-23T08:54:12.901Z    x64 (Alpine.38.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.8-helix-09ca40b-20190508143246    Release-no_tiered_compilation
2019-08-23T08:56:13.633Z    x64 OSX.1013.Amd64.Open Checked
2019-08-23T08:56:14.12Z x64 OSX.1013.Amd64.Open Checked-no_tiered_compilation
2019-08-23T08:58:44.897Z    arm (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226    Checked
2019-08-23T08:58:45.569Z    arm (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226    Checked-no_tiered_compilation
2019-08-23T08:59:06.469Z    x64 Ubuntu.1804.Amd64.Open  Checked-no_tiered_compilation
2019-08-23T08:59:07.101Z    x64 Ubuntu.1804.Amd64.Open  Checked
2019-08-23T09:03:50.871Z    x64 Windows.10.Amd64.Open   Checked-no_tiered_compilation
2019-08-23T09:03:51.09Z x64 Windows.10.Amd64.Open   Checked
2019-08-23T09:05:28.732Z    x64 Windows.10.Amd64.Open   Checked-no_tiered_compilation
2019-08-23T09:05:29.033Z    x64 Windows.10.Amd64.Open   Checked
2019-08-23T14:47:15.702Z    x64 OSX.1013.Amd64.Open Checked-no_tiered_compilation
2019-08-23T14:47:16.457Z    x64 OSX.1013.Amd64.Open Checked
2019-08-23T14:51:32.556Z    arm (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226    Checked
2019-08-23T14:51:32.939Z    arm (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226    Checked-no_tiered_compilation
2019-08-23T14:51:36.647Z    x64 OSX.1013.Amd64.Open Checked-no_tiered_compilation
2019-08-23T14:51:37.832Z    x64 OSX.1013.Amd64.Open Checked
2019-08-23T16:58:39.674Z    arm (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226    Checked-no_tiered_compilation
2019-08-23T16:58:40.205Z    arm (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226    Checked
2019-08-23T17:42:26.474Z    arm64   (Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-a45aeeb-20190620155855 Checked
2019-08-23T17:42:27.532Z    arm64   (Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-a45aeeb-20190620155855 Checked-no_tiered_compilation
2019-08-23T18:40:42.303Z    x64 OSX.1013.Amd64.Open Checked-no_tiered_compilation
2019-08-23T18:40:42.795Z    x64 OSX.1013.Amd64.Open Checked
2019-08-23T18:42:43.893Z    arm (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226    Checked
2019-08-23T18:42:44.753Z    arm (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226    Checked-no_tiered_compilation
2019-08-23T18:43:13.881Z    x64 OSX.1013.Amd64.Open Checked-no_tiered_compilation
2019-08-23T18:43:15.202Z    x64 OSX.1013.Amd64.Open Checked

[jashook]

Will modify the pr to disable the test

[mikedn]

Thanks, this is baffling. Any idea what to make of AzDO history like: https://dev.azure.com/dnceng/public/_build/results?buildId=321130&view=ms.vss-test-web.build-test-results-tab&runId=9345192&paneView=history&resultId=100000

Does it really show that it fails in all PR builds and succeeds in all master builds?!

[echesakovMSFT]

Could it be that the Build and Test jobs (in the failing PR jobs) are running on the different commits and the Build job doesn't have the fix addressed by dotnet/coreclr#20650 but the Test job already got GitHub_7147 test?

[mikedn]

Could it be that the Build and Test jobs (in the failing PR jobs) are running on the different commits and the Build job doesn't have the fix addressed by dotnet/coreclr#20650 but the Test job already got GitHub_7147 test?

I suppose that could explain the test failure but I have no idea if it's possible and why it would happen randomly.

Locally I haven't been able to get this test to fail on x86/x64 after multiple test runs. Probably the best I can do at this point is to open a PR that re-enables the test and attempt to run PR tests multiple times to see what happens. But for now I'll just wait for the CI to come to its sense, almost every PR fails either due to the ARM issue or some other random failure.

[jashook]

The arm issues should be accounted for. In relative terms, the CI seems to be working well today.

[BruceForstall]

at this point is to open a PR that re-enables the test and attempt to run PR tests multiple times

@jashook @mikedn Is it time to do this now?

[mikedn]

Yeah, the CI seems more stable these days. I'll get to it this week.