Arcade: Pdbs are overwritten when publishing in multi-framework-targeted build

[AdamYoblick]

I’m in the process of trying to verify that the winforms-designer symbols have published correctly to symweb, and we have one dll that is failing while all the others are passing.
While investigating this, Dustin mentioned that some of our dlls are multi-targeted (they build against both .net framework 4.7.2 AND .net core 3.0).

The problem I’m seeing is the pdb’s are published into a single build artifact called “PdbArtifacts”, which is flattened.

https://dnceng.visualstudio.com/internal/_build/results?buildId=404832&view=results

So which WinformsSurface.Core.pdb is there?

Upload 'F:\workspace.1\_work\1\s\artifacts\SymStore\Release\WinFormsSurface.Core\net472\WinFormsSurface.Core.pdb' to file container: '#/2685471/PdbArtifacts'
Associated artifact 5225542 with build 404832
##[section]Async Command End: Upload Artifact
##[section]Async Command Start: Upload Artifact
Uploading 1 files
File upload succeed.
Upload 'F:\workspace.1\_work\1\s\artifacts\SymStore\Release\WinFormsSurface.Core\netcoreapp3.0\WinFormsSurface.Core.pdb' to file container: '#/2685471/PdbArtifacts'
Associated artifact 5225542 with build 404832
##[section]Async Command End: Upload Artifact
##[section]Async Command Start: Upload Artifact
Uploading 1 files
File upload succeed.

Maybe these uploads are async, and this is a race condition. Or maybe the second upload always wins.
In either case, we’re losing a pdb here, since one is stomping over the other. So when we try to upload symbols, we’re not sending everything, and symcheck fails when we try to verify.

So what probably needs to happen here is some sub dirs, one for each framework moniker, which is available in the output above. That would make sure stuff doesn’t get overwritten.

[tmat]

That's indeed incorrect. The converted PDBs that are to be published to symbol server are stored in a directory structure under artifacts/SymStore that includes the TFM. The problem is likely in the logic that copies these files to the artifacts.

@JohnTortugo

[dsplaisted]

@nguerrera @livarcocc FYI, looks like this means we won't get correct symbol publishing (for both target frameworks at least) even with https://github.com/dotnet/sdk/pull/3831

[nguerrera]

Is this being ported to 3.x servicing? We need it to flow to our 3.x releases

[AdamYoblick]

Regarding the dlls, there’s no way to specify a file pattern when pushing azdo artifacts? If not, copying to a temp dir with a file filter before pushing to azdo seems prudent. Otherwise this is going to inflate the size of the azdo artifacts quite a bit.

Just my 2 cents :)

[AdamYoblick]

I'm going to re-open this issue if you guys don't mind. The bug reported is technically fixed in the linked PR - all the pdbs are present. However, the dll's are also present, which is not correct. While this issue is no longer blocking, the solution is not ideal.

The normal size of the pdbartifacts artifact (from https://dnceng.visualstudio.com/internal/_build/results?buildId=416965&view=results for example) is 2.5 MB. The size of the new pdbartifacts artifact (from https://dnceng.visualstudio.com/internal/_build/results?buildId=416824&view=results) is 10.7 MB. That's over 4 times the size. And that's for winforms, which publishes a relatively small number of pdbs compared to some other projects. And pdbs are published for every master build, not just release, which is going to add up to a lot of space once this gets applied to all teams.

Cesar has mentioned that the azdo task that uploads to build artifacts doesn't let you specify a file pattern, it just does a single folder and all contents, recursive. That's why the dll's are now coming up too. If that's the case, there should be an intermediate step that copies only *.pdb, with directory structure preserved, into a staging directory. Then, upload to azdo artifacts from the staging directory. That would fix the bug and also avoid inflating the artifact size for no reason.

Thanks :)

[AdamYoblick]

And when I remove the dlls from the zip locally, it goes back down to 3MB. :)

[JohnTortugo]

Regarding the dlls, there’s no way to specify a file pattern when pushing azdo artifacts? If not, copying to a temp dir with a file filter before pushing to azdo seems prudent. Otherwise this is going to inflate the size of the azdo artifacts quite a bit. Just my 2 cents :)
…
________________________________ From: Nick Guerrera [e-mail] Sent: Tuesday, November 5, 2019 7:55:29 PM To: dotnet/arcade [e-mail] Cc: Adam Yoblick [e-mail]; Author [e-mail] Subject: Re: [dotnet/arcade] Pdbs are overwritten when publishing in multi-framework-targeted build (#4247) Is this being ported to 3.x servicing? We need it to flow to our 3.x releases — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Farcade%2Fissues%2F4247%3Femail_source%3Dnotifications%26email_token%3DAHASVDM3GEFIVLPUCNOLI5LQSIPZDA5CNFSM4JHAS7ZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDE7GLI%23issuecomment-550105901&data=02%7C01%7CAdam.Yoblick%40microsoft.com%7Ce399152ac42e40a289e508d7625c66f4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637086021321008094&sdata=m94r2NvNz3Wdy504k%2FvgG9bAzMqZWqP8fnIUJMZQlMs%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAHASVDMFBFYJXDCYDIAH5ETQSIPZDANCNFSM4JHAS7ZA&data=02%7C01%7CAdam.Yoblick%40microsoft.com%7Ce399152ac42e40a289e508d7625c66f4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637086021321008094&sdata=rEAgrQW3PMeiuTovf4UC3V%2FCuxshZpNvaJhr%2Bi%2B%2FVkw%3D&reserved=0.

AFAIK it's not possible to specify a pattern to upload the files if using the ##vso commands. I considered the option of copying the PDBs to a temporary folder but in my experience such copying can be problematic.. That being said, I expected this SymStore folder to only contain PDBs. Do you know why does it contains .dlls also?

[riarenas]

@JohnTortugo https://github.com/dotnet/arcade/pull/4284#issuecomment-550134655 seems to be the explanation for why the dlls are there.

[mmitche]

The normal size of the pdbartifacts artifact (from https://dnceng.visualstudio.com/internal/_build/results?buildId=416965&view=results for example) is 2.5 MB. The size of the new pdbartifacts artifact (from https://dnceng.visualstudio.com/internal/_build/results?buildId=416824&view=results) is 10.7 MB. That's over 4 times the size. And that's for winforms, which publishes a relatively small number of pdbs compared to some other projects. And pdbs are published for every master build, not just release, which is going to add up to a lot of space once this gets applied to all teams.

I understand the size concern here, but this is pretty small potatoes. core-sdk's normal day to day builds upload 3.3.GB of installers and zip files. We should probably fix this by just not copying the dlls to the directory, but the right way to do this is: https://github.com/dotnet/arcade/issues/4295.

Are you okay with closing this issue for now, now that you are unblocked?

[AdamYoblick]

If the arcade team doesn't mind, then I don't mind, since technically the bug is fixed and we're unblocked once this is available in arcade (and once the symbol publish step is changed to recurse as well, @JohnTortugo @mmitche I'm not sure if that's already been done?)

[JohnTortugo]

, @JohnTortugo @mmitche I'm not sure if that's already been done

Yeah, that part was this line: https://github.com/dotnet/arcade/pull/4284/files#diff-594b10f37a6a38c24f249a6d0481ddffR33