Runtime: Enable corefx-ci dnceng/public runs for both master and 3.0 with the full test matrix

[ViktorHofer]

Currently we don't run the public CI build for master in corefx. We should enable that to get analytics and have a full test matrix.

https://dev.azure.com/dnceng/public/_pipeline/analytics/outcome?buildDefinitionId=196

cc @safern @danmosemsft

[safern]

I don't think we should do that. We already run official build per commit and that is starting to fill the helix queues, hence slower test execution. If we enable this as well, that will be double the jobs we already send to helix. I think with official builds we can get the analytics we want.

[ViktorHofer]

Every repository is doing that, i.e. CoreClr, Roslyn. We should be consistent and have analytics that are publicly available for the master branch.

[ViktorHofer]

Additionally the numbers of PRs / commits that queue new builds is far higher than the number of commits that go into master.

[safern]

Hmm I don’t think we should spend that much resources on doing CI builds with the same matrix that we just did in the PR that was merged.

Cc: @mattgal

[MattGal]

I agree with Santi here; especially given that we're using Helix queues with physical machines that can't scale, the current amount of usage is untenable given our operating budget and available physical machines (i.e. ARM and Mac hardware).

Since the same machine queues would be used for both folks' PRs and these public master runs (which should be nearly identical to the internal ones anyways) this means we'd be making folks wait longer for their PRs. Given we continuously are doing full internal official runs every time someone commits to master in these repos, it feels like triplication.

I think if we get back to having 2-3 official full runs a day from the non-public machines this becomes slightly more feasible, but my sense is adding much more load to any of the major players in Helix is going to just exacerbate our inability to scale the OSX and ARM machines.

[ViktorHofer]

The number of public builds is dependent on the number of PRs that are submitted to corefx / commits that are pushed to PRs. Let's do some math.

On a strong day we have between 20-30 PRs (assumption).
Let's say 10 of these PRs contain multiple commits which are added over the day which queues another build everytime they are pushed.

30 PRs => 30 commits, 10PRs with ~3 commits => 20 additional commits => 50 commits / builds per day. If we now say that 1/3 of all the opened PRs go into master we would have 10 additional builds which means 20% more load.

I believe that analytics for failing build steps and test flakiness should be available in the public so that community members can track down issues more easily. Hiding them in our internal builds is not great. I understand the concern with resources, still I hope we can find a solution. This is something that came up in the CI council meeting with SteveC and it would be a shame if corefx is the only player that doesn't have these analytics publicly available.

[safern]

I believe that analytics for failing build steps and test flakiness should be available in the public so that community members can track down issues more easily. Hiding them in our internal builds is not great. I understand the concern with resources, still I hope we can find a solution. This is something that came up in the CI council meeting with SteveC and it would be a shame if corefx is the only player that doesn't have these analytics publicly available.

I agree on that. So a thing that I have on my plate is to lower the number of test runs from official builds. I.e, run tests from official builds just 2 times a day, but still have official builds per commit (this to accelerate dependency flow). Since we don't fail the build on test failures, it doesn't make sense to run tests for every official build we do. I would say just run tests on scheduled and manual builds.

WIth that, we would lower the payload to helix and we could enable this builds on master. I will do those 2 as part of 1 PR. I guess we still want batched builds for public-ci, correct?

[ViktorHofer]

That sounds reasonable. Maybe out-of-scope but we could consider running tests in official builds completely decoupled from the build process and use the cross-gend versions of our assemblies. We would run our tests the same way that coreclr and Mono do via the test asset payloads.

[safern]

That sounds reasonable. Maybe out-of-scope but we could consider running tests in official builds completely decoupled from the build process and use the cross-gend versions of our assemblies. We would run our tests the same way that coreclr and Mono do via the test asset payloads.

Yes we could do that as well, but I think that should probably be on a separate issue/PR. I think there is already an issue to run tests agains cross-gened binaries.

[ViktorHofer]

Right, that would be step 2 of the whole process here. Let's first do step 1 and continue with step 2 afterwards. We need the cross-gened bits testing mechanism anyway for 3.0.

[safern]

I'll take care of step 1 tomorrow in a spare time.

[ViktorHofer]

Thanks a lot!

[danmosemsft]

I would love for community to be able to see the full set of test results (configurations and outer loop) that we get on official builds. So if that's the suggestion, +1 to that.

[ViktorHofer]

@safern with your work here, can you please also check why tests in dnceng/public master are skipped?

[safern]

Sure.

[ViktorHofer]

Updated the title. Full matrix = everything from the official build except the uapaot tests (2k tests are currently failing there).

[ViktorHofer]

Done, closing.