Arcade: Publish symbols for full-framework amd64 binaries

[rainersigwald]

MSBuild produces MSBuild.exe and MSBuildTaskHost.exe as both 32-bit and 64-bit applications. Currently, Arcade does not publish symbols for the 64-bit copies.

I haven't fully dug into the problem here. The build produces valid .pdb files for both (portable for MSBuild.exe and legacy for MSBuildTaskHost.exe which targets net35), but they don't get published.

We manually published the symbols for 16.0 and can continue to do so, so it'd probably be ok if this is fixed only in the release-pipeline symbol publishing, but if we can fix it now that'd be great.

[markwilkie]

@rainersigwald - is this a ship block do you think?

cc/ @JohnTortugo and @jcagme

[rainersigwald]

Hasn't been so far, but I'm pretty annoyed at the manual steps for every insertion, which does slow our velocity down there. @livarcocc might have an opinion, too.

[jcagme]

Ack. I don't have a quick answer at this moment but I will investigate soon.

[rainersigwald]

Holler at me before you dive in too deeply @jcagme. We're doing some unusual things that are probably causing some of this.

[jcagme]

Will do. I just came back from vacations so I'm just draining my inbox. Once done I'll take a quick peek att things and let you know what were the findings.

[JohnTortugo]

I haven't fully dug into the problem here. The build produces valid .pdb files for both (portable for MSBuild.exe and legacy for MSBuildTaskHost.exe which targets net35), but they don't get published.

I think there is a confusing behavior in the "non-async" version of Arcade.SDK publishing that prevent symbols to be published during the same step as publishing to a package feed: https://github.com/dotnet/arcade/blob/a980f98a961cced039996eb49daabb76679c16a5/src/Microsoft.DotNet.Arcade.Sdk/tools/Publish.proj#L26 Maybe you're hitting that? It's weird that it only happens for the 64bit build, though. @tmat

[tmat]

What's confusing about it? It was never designed to be used at the same time as publishing to Azure feed. Either we publish to Azure feed or we publish directly. It doesn't make sense for the artifacts to be published to both, since the Azure feed would push them to the same places that the direct publishing would.

[rainersigwald]

I don't think that's my problem. I think things start to go wrong because we don't have a single ProjectReference from our VSIX project to the .exe projects (because we need each of them in two architectures).

[JohnTortugo]

What's confusing about it?

I just thought this could be the same issue as this one: https://github.com/dotnet/arcade/issues/1499 Looks like it's not.

[rainersigwald]

Did y'all do anything to fix this? @livarcocc pointed out that the last couple of insertions have started passing, with this from the internal symbolcheck job:

Checking 'E:\A\_work\2\vi\Pkgs\decomp\1\Contents\MSBuild\Current\Bin\amd64\MSBuild.exe'
Running: 'E:\A\_work\2\s\src\tools\VerifyInsertion\Scripts\..\Tools\symcheck\symchk.exe /v /s SRV*"E:\A\_work\2\a\pdb\2rwhbasu.l2r"*http://msdl.microsoft.com/download/symbols "E:\A\_work\2\vi\Pkgs\decomp\1\Contents\MSBuild\Current\Bin\amd64\MSBuild.exe"'
Exit code '1' != 0.
'E:\A\_work\2\vi\Pkgs\decomp\1\Contents\MSBuild\Current\Bin\amd64\MSBuild.exe' archived in Msdl?: 'False'
Running: 'E:\A\_work\2\s\src\tools\VerifyInsertion\Scripts\..\Tools\symcheck\symchk.exe /v /s SRV*"E:\A\_work\2\a\pdb\ya0dahqa.yki"*http://symweb "E:\A\_work\2\vi\Pkgs\decomp\1\Contents\MSBuild\Current\Bin\amd64\MSBuild.exe"'
Output contains success string?: 'True'
Pdb path correct?: 'True'. Extracted Pdb path: '\\redmond\symfiles\vsosymprxy01\cache\42\AF\8E561B72B7CE2C4514D917946236E10862E3504CE1B262CED358603D8E5D00\msbuild.pdb'
'E:\A\_work\2\vi\Pkgs\decomp\1\Contents\MSBuild\Current\Bin\amd64\MSBuild.exe' archived in symweb?: 'True'
...
Checking 'E:\A\_work\2\vi\Pkgs\decomp\1\Contents\MSBuild\Current\Bin\amd64\MSBuildTaskHost.exe'
Running: 'E:\A\_work\2\s\src\tools\VerifyInsertion\Scripts\..\Tools\symcheck\symchk.exe /v /s SRV*"E:\A\_work\2\a\pdb\dnawlbsr.fu1"*http://msdl.microsoft.com/download/symbols "E:\A\_work\2\vi\Pkgs\decomp\1\Contents\MSBuild\Current\Bin\amd64\MSBuildTaskHost.exe"'
Exit code '1' != 0.
'E:\A\_work\2\vi\Pkgs\decomp\1\Contents\MSBuild\Current\Bin\amd64\MSBuildTaskHost.exe' archived in Msdl?: 'False'
Running: 'E:\A\_work\2\s\src\tools\VerifyInsertion\Scripts\..\Tools\symcheck\symchk.exe /v /s SRV*"E:\A\_work\2\a\pdb\jxysk5bo.zor"*http://symweb "E:\A\_work\2\vi\Pkgs\decomp\1\Contents\MSBuild\Current\Bin\amd64\MSBuildTaskHost.exe"'
Output contains success string?: 'True'
Pdb path correct?: 'True'. Extracted Pdb path: '\\redmond\symfiles\vsosymprxy04\cache\66\C3\A590C2B607C80459304108FCCE46679534035F8EDBBCF1D5023F82222CEF00\msbuildtaskhost.pdb'
'E:\A\_work\2\vi\Pkgs\decomp\1\Contents\MSBuild\Current\Bin\amd64\MSBuildTaskHost.exe' archived in symweb?: 'True'

[markwilkie]

@JohnTortugo ?

[JohnTortugo]

I'm not aware of any change on that front. Is it still passing @rainersigwald / @livarcocc ?

[rainersigwald]

Test insertion from today's build: https://devdiv.visualstudio.com/DevDiv/_git/VS/pullrequest/185729. Symbol check is running: https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=2751454.

[rainersigwald]

(that passed. 🤷‍♂️)

[JohnTortugo]

That's good to know. We should understand what happened that make it work though. :-/

[riarenas]

@JohnTortugo next steps here? did we ever understand what changed that caused things to work?

[JohnTortugo]

@riarenas - I didn't investigate on that. @rainersigwald - Can you confirm this is still working?

[rainersigwald]

Seems to be; our insertions keep turning green.

[BenVillalobos]

We're running into this issue again here with our amd64 symbols. We've also noticed that the last successful insertion (build here) also didn't seem to be pushing them up from inside the build, but they were somehow available on the symbol server.

[JohnTortugo]

Updating here with things that we discussed offline on Teams.

In the successful insertion the amd64\msbuild.exe matched the symbols published in the net472\msbuild.pdb file. I asked @rainersigwald and he said that this was the expected behavior. In the insertion where the symbols where not found the net472\msbuild.pdb symbol file was published but the symbol checker didn't consider that as the correct file for the amd64\msbuild.exe.

@rainersigwald @BenVillalobos - It's not clear to me at this point if this is an issue with the infra for publishing of symbols or something else. Could you please share your findings here?

[rainersigwald]

I think I misunderstood your offline question. We have two different msbuild.exe/pdb outputs:

• artifacts\bin\MSBuild\Release\net472\MSBuild.exe
• artifacts\bin\MSBuild\x64\Release\net472\MSBuild.exe

they're both in net472 folders, but they aren't the same. I don't know the symbol-resolution rules well enough to know whether they're expected to match the same PDB--they should be _almost_ identical except for the 32-bit flag in the PE header.

I don't understand how Arcade detects symbols to publish. It's clearly not "just working" in our case, but I don't know why. The build on this project is a bit funky because it has the x86/amd64 flavors (in addition to .net core for AnyCPU).

[JohnTortugo]

...
I don't understand how Arcade detects symbols to publish. It's clearly not "just working" in our case, but I don't know why. The build on this project is a bit funky because it has the x86/amd64 flavors (in addition to .net core for AnyCPU).

@rainersigwald which of these should match the Amd64/MSBuild.exe symbol check? What I'm trying/will look here is if that file got published by Arcade SDK or not.

[rainersigwald]

artifacts\bin\MSBuild\x64\Release\net472\MSBuild.pdb should match the binary we place in bin\amd64\MSBuild.exe. It doesn't appear that it is being published--but it didn't look like that _based on a log from when it was working_ so Ben and I got pretty confused.

[JohnTortugo]

Moving again to backlog as this is a non blocking issue/priority right now.

[rainersigwald]

@JohnTortugo this was a blocking issue for MSBuild until we worked around it with microsoft/msbuild#5070. How did it get triaged down?

[JohnTortugo]

I moved it back to backlog because after our discussion on Teams I thought you had a temporary workaround for this. I understand that this is causing some unnecessary pain.

[JohnTortugo]

I think the problem here might be, at least in part, that some PDB files are getting overritten when Arcade SDK push them as Azdo Artifacts:

Upload 'E:\A\_work\368\s\artifacts\SymStore\Release\MSBuild\net472\amd64\MSBuild.pdb' to file container: '#/5204047/PdbArtifacts'
Associated artifact 3777451 with build 3445881
File upload succeed.
...
Upload 'E:\A\_work\368\s\artifacts\SymStore\Release\MSBuild\net472\MSBuild.pdb' to file container: '#/5204047/PdbArtifacts'
Associated artifact 3777451 with build 3445881
File upload succeed.
....
Upload 'E:\A\_work\368\s\artifacts\SymStore\Release\MSBuild\netcoreapp2.1\MSBuild.pdb' to file container: '#/5204047/PdbArtifacts'
Associated artifact 3777451 with build 3445881
File upload succeed.

MSBuild uses an old version of the SDK and I believe this problem is fixed in the newer versions of the SDK. I'm talking with @rainersigwald on Teams to see if it's possible to switch MSBuild to a newer version of the SDK or find another solution for this.

[Forgind]

@JohnTortugo, Can we reopen this? As I said in a comment in the insertion PR into MSBuild, the symbol check failed after I backed out #5070 (our workaround) even with your change.

[JohnTortugo]

@Forgind - Yeah, sure. I'm looking to see what happened.

[JohnTortugo]

I think the problem here was that publishing of symbols (in the post-build stages) didn't run at all because there was no Maestro channel configured for that repo+branch.

I configured a default channel for that repo+branch and queued a new build.

[JohnTortugo]

New finding: the builds produced from the exp\symbol-change-check branch aren't producing the amd64*.pdb files. @Forgind @rainersigwald do you know why?

[Forgind]

See #5070. It's our workaround that produces those files; I explicitly reverted that for the exp/symbol-change-check branch.

[JohnTortugo]

Talked offline with @rainersigwald and @Forgind . I'm investigating more into this.

[JohnTortugo]

Update on this. Turns out that my latest fix for the issue was _necessary but not sufficient_. There is more going on.

In talk with @rainersigwald he shared his theory of what's going on:

The current PDB capture happens based on Pack, which traverses to ProjectReferences. But [in MSBuild] we have the same project built twice--net472/x86 and net472/x64. That's not easily representable via ProjectReference, and we jump through some hoops to do it. But Arcade isn't seeing that

For the net35 binaries we don't have a choice but to do full PDB--portable wasn't an option in the hazy past

But the x86 net36 MSBuildTaskHost.exe works, so that's probably not the problem

consequently they had to implement the following workaround to copy the amd64 PDBs to the SymStore directory:

<Copy 
  SourceFiles="$(ArtifactsBinDir)MSBuild\x64\Release\net472\MSBuild.pdb"
  DestinationFolder="$(ArtifactsSymStoreDirectory)\MSBuild\net472\amd64" />

<Copy 
  SourceFiles="$(ArtifactsBinDir)MSBuildTaskHost\x64\Release\net35\MSBuildTaskHost.pdb"
  DestinationFolder="$(ArtifactsSymStoreDirectory)\MSBuildTaskHost\net35\amd64" />

It doesn't seem that the problem should be solved by adjusting the place the SDK looks for the symbols since the current failure seems to be due to a workaround (the hoops that @rainer mentions) for another issue (not being easy to represent multi arch in PackageReferences).

I'm not sure what's the right approach here. @tmat as an SDK expert can you please chime in here?

[tmat]

A few things to do:
1) Merge fix in Arcade to include PlatformName in the output path: https://github.com/dotnet/arcade/pull/4855
1) Update msbuild to the latest Arcade SDK (it's currently on a old one, is it not flowing?): https://github.com/microsoft/msbuild/blob/master/global.json#L15
3) This should be reverted: https://github.com/microsoft/msbuild/pull/5070

[Forgind]

Maestro made a pr into MSBuild to update the version of the arcade sdk. I blocked it temporarily while I ran the symbol check without microsoft/msbuild#5070, and that failed, so it hasn't been merged. I believe maestro should update msbuild#5122 after arcade#4855 goes in, so I can run the symbol check again at that point merge msbuild#5122 if it passes.

[JohnTortugo]

@Forgind - We'll need to port the change @tmat made to the Arcade servicing branch before MSBuild gets the fix. See comment in this PR: https://github.com/dotnet/arcade/pull/4855#pullrequestreview-359437511

[JohnTortugo]

Talked with @rainersigwald on Teams and he agreed with this plan:

• Keep MSBuild current workaround until they start development of .Net 5 and start consuming Arcade SDK 5.x which includes the proper fix.