Arcade: Investigate Winforms internal builds failures

[JohnTortugo]

Winforms build is failing after I updated the branch with latest Arcade updates.

Builds from the internal/release/3.1 branch are passing the validation: https://dnceng.visualstudio.com/internal/_build/results?buildId=384191&view=results

However, builds for private branches based of on internal/release/3.1 AND updated with Arcade changes by doing darc update-dependencies --channel ".NET tools - latest" -- source-repo arcade are failing:

https://dnceng.visualstudio.com/internal/_build/results?buildId=384189&view=results
https://dnceng.visualstudio.com/internal/_build/results?buildId=382989&view=results

I think this is high priority as it's might block servicing builds.

@sunandabalu @AdamYoblick @mmitche @markwilkie @riarenas

[riarenas]

can't you just disable SDL for now for these branches while it's investigated further?

Meaning yes, we need to investigate, but I don't think it blocks servicing unless something further down the pipeline fails after SDL validation.

[riarenas]

darc update-dependencies --channel ".NET tools - latest" -- source-repo arcade

This looks wrong. These builds should be updating from the ".NET 3 tools" channel

[sunandabalu]

Its the same code that is running in the both branches for SDL, the error is unrelated to Arcade updates. This looks very similar to the issue called out here - https://github.com/dotnet/winforms/issues/1389

2019-10-09T19:33:30.5438687Z ##[command]"C:windowsSystem32WindowsPowerShellv1.0powershell.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'D:a_temp5eeaa1c5-1ef8-4c27-a53c-30eeb0944dd2.ps1'"
2019-10-09T19:33:31.4698544Z Processing D:a1seng..global.json
2019-10-09T19:33:32.8805375Z dotnet-api-docs_netcoreapp3.0 was not found in .
2019-10-09T19:33:32.8839454Z Microsoft.PowerShell.Commands.WriteErrorException: dotnet-api-docs_netcoreapp3.0 was not found in .

[JohnTortugo]

darc update-dependencies --channel ".NET tools - latest" -- source-repo arcade

This looks wrong. These builds should be updating from the ".NET 3 tools" channel

I'm using the command you suggested here: https://github.com/dotnet/arcade/issues/3868#issuecomment-533152364 should I change now to the ".NET 3 tools" channel?

[AdamYoblick]

WPF uses these native tool versions defined in their global.json (https://github.com/dotnet/wpf/blob/e5cfb5d12e5908facb15ecc398cc9a6035a81928/global.json#L21) as well, and they also use the post-build.yml and SDL validation.

https://github.com/dotnet/wpf/blob/e5cfb5d12e5908facb15ecc398cc9a6035a81928/azure-pipelines.yml#L75

Although I clicked on their internal build, and I don't see the stages/SDL steps there, so I believe it's not really running for them. I assume the same problem would occur for them if it was.

[AdamYoblick]

But I agree with Ricardo, if we need to disable the SDL runs while this gets sorted to unblock the build, I don't see any problem with that.

[JohnTortugo]

Cool. I'll disable the validation for now then.

[riarenas]

I'm using the command you suggested here: #3868 (comment) should I change now to the ".NET 3 tools" channel?

Yes, that issue and instructions were written up before we had snapped and created that channel. All 3.x branches are set up to pull from the new channel, so the servicing tests should do the same.

[JohnTortugo]

Got same error now in a different job: https://dnceng.visualstudio.com/internal/_build/results?buildId=384371&view=results

So, this is indeed not specific to SDL validation.

PS: I'll try running the darc update-dependencies command with the channel @riarenas suggested. That might be the reason of the problem.

[JohnTortugo]

Looks like indeed the issue was caused by updating against the wrong channel. Closing.

[AdamYoblick]

@riarenas @JohnTortugo We're seeing these failures in winforms master again. I checked the darc subscriptions and it looks like our master branch is consuming builds from the ".NET Tools - Latest" channel. Should we be changing this sub to consume from ".NET 3 Tools" instead?

winforms-designer is also consuming from ".NET Tools - Latest"

[mmitche]

@AdamYoblick .NET Tools - Latest is the correct channel for master.

[AdamYoblick]

Interesting...we're still seeing errors in our master branch then. Perhaps I need to update dependencies against the latest channel again?

[AdamYoblick]

Since our master build is still failing on internal builds, I'm going to re-open this while I troubleshoot.

[AdamYoblick]

@mmitche I just merged in a darc update from arcade -> winforms master (not manual, the PR was already made at https://github.com/dotnet/winforms/pull/2088), so if this doesn't fix the issue then we either need to force update dependencies locally or something else is wrong?

[mmitche]

No,. that one was built yesterday or today, so it should be plenty recent

[AdamYoblick]

ick, looks like we have other build issues. Give me a few :)

[AdamYoblick]

Ok nm, that issue was sporadic. Internal build after the darc update is running at https://dnceng.visualstudio.com/internal/_build/results?buildId=388477

[AdamYoblick]

Looks like the issue is persisting after a darc update into master: https://dev.azure.com/dnceng/internal/_build/results?buildId=388687

[AdamYoblick]

@sunandabalu I believe you were looking at this before but it seems to have cropped up again. Did the SDL fix for this ever make into the ".Net Tools - Latest" channel?

[sunandabalu]

@sunandabalu I believe you were looking at this before but it seems to have cropped up again. Did the SDL fix for this ever make into the ".Net Tools - Latest" channel?

There was no fix for SDL made, as there were no changes to SDL scripts/templates after 9/30.

Though something changes in tools.ps1 that the SDL scripts imports - https://github.com/dotnet/arcade/pull/3975/files#diff-58d6d2c2760628d9551cc8b40461ce53 . Do you know if this could break something in winforms?

[sunandabalu]

/cc @chcosta

[chcosta]

Looking

[chcosta]

Yes, this is failing because Winforms explicitly executes install-native-tools.ps1 in configure-toolset.ps1.

It would probably be useful to move the DisableNativeToolsetInstalls is "null" part of this check into tools.ps1 and then to update pieces of the post-build which don't depend on native toolset installations, to set the "DisableNativeToolsetInstalls" variable. A similar change should also be made to the Linux scripts.

@sunandabalu , do you want to work on the fix for this or do you want me to take this one?

[sunandabalu]

I'll take it

[AdamYoblick]

Ok thanks, I'm going to disable the SDL runs for winforms in the meantime just so we're unblocked. When a fix is in arcade, please let us know and we can update dependencies and re-enable :)

[AdamYoblick]

Disabled the SDL runs in the meantime, but the publishing steps are failing with the same error in the "Set Release Configs Vars" step:

##[command]"C:\windows\System32\WindowsPowerShell\v1.0\powershell.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'D:\a\_temp\4c8be151-82a0-480f-85b2-960e70f406ea.ps1'"
Processing D:\a\1\s\eng\..\global.json
dotnet-api-docs_netcoreapp3.0 was not found in .
Microsoft.PowerShell.Commands.WriteErrorException: dotnet-api-docs_netcoreapp3.0 was not found in .
##[error]PowerShell exited with code '1'.

https://dev.azure.com/dnceng/internal/_build/results?buildId=389041&view=logs

[sunandabalu]

@AdamYoblick The change is in PR , I have added the change to post-build-utils.ps1 as well which covers the "Set Release Configs Vars" step.

[AdamYoblick]

Thank you!

[AdamYoblick]

Hi everyone, any status on this? winforms internal master builds have been failing for quite a bit here. The latest one is at https://dnceng.visualstudio.com/internal/_build/results?buildId=395252

[chcosta]

There's a PR that is in progress - https://github.com/dotnet/arcade/pull/4133

[AdamYoblick]

Ahh great, thank you chris :)

[sunandabalu]

@AdamYoblick PR is merged to Arcade, it should flow into WinForms following your repo's regular cadence.

[AdamYoblick]

Thank you Sunanda! I will update dependencies into our master and verify :)

[AdamYoblick]

@sunandabalu Looks like we have a pending PR for update dependencies from arcade: https://github.com/dotnet/winforms/pull/2145. Does this include the change you mentioned?

Build: 20191021.4
Date Produced: 10/21/2019 9:44 PM
Commit: f104bcb28e4936b0c0ab7c0d8cb73892ab515e60
Branch: refs/heads/master

[AdamYoblick]

Nevermind, I ran darc trigger-subscriptions to get latest, just in case :)

[AdamYoblick]

Internal build with SDL re-enabled is running at https://dnceng.visualstudio.com/internal/_build/results?buildId=397556. I'll reply again when it finishes to verify this is fixed.

[AdamYoblick]

Looks good, closing this out :)