Runtime: Nano CI builds completely broken

[stephentoub]

cc: @Priya91, @karelz, @mmitche
Our Windows Server Nano 2016 builds have been red for as long as we have history. Looking at the error message, it looks like the Jenkins configuration is broken:

08:56:50 Schedule job dotnet_corefx » master » outerloop_winnano16_debug_tst
08:56:50 ERROR: Failed to run DSL Script
08:56:50 com.cloudbees.plugins.flow.CouldNotScheduleJobException: Could not schedule job outerloop_winnano16_debug_tst, ensure it is not already queued with the same parameters or is not disabled
08:56:50    at sun.reflect.GeneratedConstructorAccessor1731.newInstance(Unknown Source)
08:56:50    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
08:56:50    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
08:56:50    at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83)
...

If we care about Nano (I assume we do!), we should fix this immediately.
If we don't, we should delete the jobs and remove the badges from the home page.

[benaadams]

I care... have no idea how Jenkins works though so can't fix it 😿

[mmitche]

@stephentoub The issue is that the nodes themselves don't seem to dynamically provision properly, and TP5 is really old now (which is what our old static image had). So we need to figure that out first

[karelz]

cc @danmosemsft

[danmosemsft]

Engineering work:
https://github.com/dotnet/core-eng/issues/358
https://github.com/dotnet/core-eng/issues/406
https://github.com/dotnet/core-eng/issues/135
https://github.com/dotnet/core-eng/issues/436

Other:
https://github.com/dotnet/corefx/issues/9975
https://github.com/dotnet/corefx/issues/7434
https://github.com/dotnet/corefx/issues/7433

[willdean]

The core-eng links don't work here - are they ms-private or has something been renamed?

[danmosemsft]

Oops, @willdean, I recall now they currently are as at present they also track internal engineering infra.

[steveharter]

Looping in @CIPop as he also would like to move off of TP5

[mmitche]

@smile21prc Is working on this.

[danmosemsft]

@smile21prc is working on this ETA this week apparently and the failure rate looks OK.

[smile21prc]

Re-implementing the old Nano CI isn't the correct way to fix the broken Nano CI builds because that approach is a bit of a hack.

@mmitche is working on the portable build + Helix in CI (dotnet/core-eng#643), which will fix Nano CI builds correctly once complete. As a result, this task doesn’t need to be tracked as a separate issue any more, as there won’t be additional Nano-specific work after his work is complete.

Closing this one as a dup of that ongoing work.