Arcade: Fix problem where build are looking for non-existent artifacts/SymStore/Release

[JohnTortugo]

Reported initially by @dougbu . Sample build here: https://dev.azure.com/dnceng/internal/_build/results?buildId=422550&view=results

I'll update the title / add more info when I know better what's going on.

[JohnTortugo]

This is a weird bug to me. The condition on line 212 here shouldn't be satisfied if the path doesn't exist: https://github.com/dotnet/arcade/blob/master/src/Microsoft.DotNet.Arcade.Sdk/tools/Publish.proj#L220

Furthermore, the .binlogs that I'm inspecting are HUGE and crashing the log viewer. @mmitche also took a peek on that.

/cc @alexperovich @markwilkie

[alexperovich]

Yea, the huge binlog is basically unusable. I can replay it with msbuild and try some textual searching, but that will be even bigger. This is a good reason for https://github.com/dotnet/arcade/pull/3859. Compacting all the build steps into one binlog makes the log unusable at all.

[tmat]

@alexperovich

3859 is not necessary to implement the workaround for reducing size of binlogs. See https://github.com/dotnet/arcade/pull/3859#issuecomment-548044973

[alexperovich]

No luck, the binlog is completely unusable without spending a bunch of time writing code to read it and filter out the interesting stuff. A textual log containing the interesting data is well over 2GB of text.

[alexperovich]

@tmat I don't have a strong opinion either way for #3859, but splitting needs to happen. Builtin to arcade is probably better so all repos benefit.

[JohnTortugo]

Thanks for looking into this Alex. I'll look for another repo/build that might be failing with same scenario and have a smaller binlog.

[alexperovich]

Hrm, I think this line in aspnetcore could be the culprit https://github.com/aspnet/AspNetCore/blob/9a1810c1dbe432fc7bc7e8bc68fa22ab787c0452/eng/Publishing.props#L14 It adds things to the @(FilesToPublishToSymbolServer) item group that aren't in $(ArtifactsSymStoreDirectory).

[alexperovich]

We should probably empty that item group before this line https://github.com/dotnet/arcade/blob/82c69e4d687077689fae4826e755ff5bf296c9c3/src/Microsoft.DotNet.Arcade.Sdk/tools/Publish.proj#L211 and error if there were things in it.

[dougbu]

We used the only workaround available to publish loose PDB files and an Arcade update broke us. How is that our fault?

[dougbu]

And

We should probably empty that item group before this line

Please do not do this. This will mean that we no longer have the (one) loose PDB file posted to the symbol servers.

[JohnTortugo]

It's an Arcade.SDK bug. I'm working to fix it. I'll keep the FilesToPublishToSymbolServer itemgroup. Using it's content as a condition for the existence of the SymStore directory is incorrect.

[dougbu]

Thanks @JohnTortugo Great to know I was overreacting

[JohnTortugo]

Update:

I've implemented a fix (copy the files to a temp location and upload that whole folder) and I'm in the process to test it. I think I'll be able to have a PR for it later today.

[JohnTortugo]

The fix for this didn't reach AspNetCore yet: https://github.com/aspnet/AspNetCore/pull/17061#pullrequestreview-316640551 . Keeping the issue in validate for now.

[dougbu]

@JohnTortugo aspnet/AspNetCore#17061 is now in (with many thanks to @Tratcher). The official build hasn't shown up yet and the CI build (which won't exercise this bug) hit provisioning problems i.e. is sitting at Queued.

Eventually, the build you want will show up in https://dev.azure.com/dnceng/internal/_build?definitionId=21&_a=summary&branchFilter=4872&repositoryFilter=120

[Tratcher]

https://dev.azure.com/dnceng/internal/_build/results?buildId=427987, and it looks like the Mac build is already past this error.

[dougbu]

That particular build eventually failed for unrelated reasons but our 'master' branch has had successful CI builds since e.g. https://dev.azure.com/dnceng/internal/_build/results?buildId=431338. Think this issue is resolved everywhere now.

[MattGal]

Per Doug's comment, and given Cesar has been on parental leave for a while, I'm closing this out; feel free to reactivate @dougbu .