Arcade: Arcade's Publish.proj needs to provide a mechanism for uploading files other than PDBs

[MattGal]

• [ ] This issue is blocking --> not blocking but makes finding and loading matching symbols automatedly very hard
• [x] This issue is causing unreasonable pain --> Blocks debugging winforms designer crashes via automated symbol server loading of PDBs

See https://github.com/dotnet/arcade/blob/master/src/Microsoft.DotNet.Arcade.Sdk/tools/Publish.proj#L164-L166 . From discussing this with @mikem8361 , this task should be uploading the matching DLLs at the same time so they can be indexed and loaded.

It _might_ be as easy as adding a line to glob in DLLs:

    <ItemGroup>
      <FilesToPublishToSymbolServer Include="$(ArtifactsSymStoreDirectory)**\*.pdb"/>
      <FilesToPublishToSymbolServer Include="$(ArtifactsSymStoreDirectory)**\*.dll"/>
    </ItemGroup>

... in addition to making sure "the right" DLLs are copied under ArtifactsSymStoreDirectory

Details

• This doesn’t impact other repos like ASP or Runtime because:

  • Their interesting stuff is all published in nupkg & symbols.nupkg format
  • Their symbols.nupkgs include the DLLs they’re indexing
  • That said, if there are DLLs that aren’t also contained in a nupkg, it’s entirely possible there are non-indexed DLLs from runtime or other repos too.

• This does affect winforms-designer because:

  • At least in the builds I’ve looked at their only nupkg is just a reference assembly one that doesn’t contain PDBs
  • They publish their PDBs independently of nupkgs, so go down a slightly different path.

In order to fix this:

In Arcade, we will need to:

• Also glob binaries (.dll, .exe, possibly others?) when selecting files to publish to symbol servers (currently */.pdb only)
• Ensure that the Yaml templates provided for publishing work for this (looks like if the DLLs are included anywhere within “PDBArtifacts” artifact, it just works, otherwise we need to intentionally download one more artifact, call it something like “DLLsToIndex”
• [STRETCH] Potentially teach the symbols publishing task to know how to unpack vsix files (tricky because it includes DLLs we don’t publish)

[markwilkie]

@epananth - is this something you understand how to do?

cc/ @mmitche

[mmitche]

Not offhand. @MattGal when we feed symbol packages through the symbol publishing stage, do we get this behavior? I'm wondering why we haven't seen this for all the other repos.

[MattGal]

Not offhand. @MattGal when we feed symbol packages through the symbol publishing stage, do we get this behavior? I'm wondering why we haven't seen this for all the other repos.

That's context from the email that didn't make it here, I'll copy it over shortly. Most team's symbols.nupkgs also include the DLLs they are symbols for, which on the other side gets unpacked and indexed. When you directly submit PDBs, you need to directly submit (correctly matching) DLLs, and in my (quick) checks I didn't see anyone else doing what winforms-designer does.

There's a chance that submitting their VSIX directly (which DOES contain all the right DLLs to index) works too since it's also just a zip like a nupkg, but we should follow up with the symbols team before just assuming this.

[mikem8361]

If the .vsix packages are actually just normal .zip files, they should just work with the symbol uploader. It doesn’t care what the extension is.

[MattGal]

oh fantastic that makes things a bit simpler

[mikem8361]

There is some filtering of the directories/files within a zip file. It doesn’t publish anything under “ref/” which are normally reference assemblies in a nupkg.

    private bool ShouldIndex(string fullName, out bool convertThisPdb)
    {
        Debug.Assert(fullName != null);

        // There are some packages that have double forward slashes in the path i.e. "runtimes//lib".
        fullName = fullName.Replace("//", "/");

        // Skip the reference assemblies and their pdbs; they are not needed for debugging. Skip any
        // directory entries (end with /) and the special nupkg symbol package file.
        if (fullName.StartsWith("ref/") || fullName.EndsWith("/") || fullName.EndsWith("_.pdb"))
        {
            convertThisPdb = false;
            return false;
        }
        // If the file is on the exclusion list, don't index
        if (PackageExcludeFiles.Contains(fullName))
        {
            convertThisPdb = false;
            return false;
        }
        string extension = Path.GetExtension(fullName);
        Debug.Assert(extension != null);

        // Filter the windows pdb conversion to any directory not starting with "runtimes", runtimes/lib directories and any windows runtime platforms
        convertThisPdb = extension == ".pdb" && (!fullName.StartsWith("runtimes") || fullName.StartsWith("runtimes/lib") || fullName.StartsWith("runtimes/win"));
        return ValidExtensions.Contains(extension);
    }

[AdamYoblick]

Thank you everyone. 😄

Please let me know if you need anything from the winforms team, including testing whatever changes make it into arcade or if we need to fix anything on our end. We currently have no symbols in Watson, so we're happy to be a guinea pig for changes!

[AdamYoblick]

/cc: @JeremyKuhne @Shyam-Gupta

[MattGal]

Based off the code (I haven't inspected your VSIX but it's not likely structured like a nupkg inside!) we just need to get all the VSIX'es included with PackagesToPublish property of the task. We'll be sure to discuss this tomorrow in triage.

[AdamYoblick]

Vsix is indeed just a renamed zip file, doesn't look like the nupkg when you open it up. Our latest published vsix from our master branch is at https://artprodcus3.artifacts.visualstudio.com/Ab55de4ed-4b5a-4215-a8e4-0a0a5f71e7d8/7ea9116e-9fac-403d-b258-b31fcf1bb293/_apis/artifact/cGlwZWxpbmVhcnRpZmFjdDovL2RuY2VuZy9wcm9qZWN0SWQvN2VhOTExNmUtOWZhYy00MDNkLWIyNTgtYjMxZmNmMWJiMjkzL2J1aWxkSWQvODA4MTAyL2FydGlmYWN0TmFtZS9WU0lY0/content?format=file&subPath=%2FWinFormsDesigner.Setup.vsix

[mikem8361]

The symbol uploader should be able to handle these vsix’s. I don’t see anything in the layout that would be a problem.

[markwilkie]

So I'm hearing that uploading VSIXs will solve this which is great news. @MattGal - do you think this fits into FR then?

[MattGal]

So I'm hearing that uploading VSIXs will solve this which is great news. @MattGal - do you think this fits into FR then?

I think so; while it's been in this state a long time, it's probably reasonable in terms of scope, I had filed the issue with the intent of discussing today at triage though.

[markwilkie]

Sounds good. I can't make triage today as I have a conflict. The rationale for moving this to FR is the fact that it's blocking a customer scenario. However, as Matt points out, it's been this way for a while... I'm personally fine with any decision - so long as it presents a path forward.

[AdamYoblick]

It HAS been this way for a while. But the .net core winforms-designer wasn't really in a state to release to control vendors until very recently, and that's why we're starting to get watson dumps, which is what surfaced the issue. So not urgent, but would also prefer not to wait too long if possible 😄

[markwilkie]

Thanks for your understanding @AdamYoblick

[Shyam-Gupta]

@markwilkie Pardon me as I am not aware of 'FR'. When is this bug expected to get fixed ?

Although it has been in this state for a while, we do have a backlog of Watsons and PerfWatsons that we need to look into. With missing stack trace information its really hard to investigate them.
VS 16.8 Preview 3 is expected to release on 9/14. Can we do something to get symbol information for this release ?

/cc: @merriemcgaw

[MattGal]

@markwilkie Pardon me as I am not aware of 'FR'. When is this bug expected to get fixed ?

Although it has been in this state for a while, we do have a backlog of Watsons and PerfWatsons that we need to look into. With missing stack trace information its really hard to investigate them.
VS 16.8 Preview 3 is expected to release on 9/14. Can we do something to get symbol information for this release ?

/cc: @merriemcgaw

Think of it as an analog to the vintage CLR "quick response team" for dnceng. For releases that have already had all their assets pushed, the thing to do is if it's important is likely to work with some folks to manually push to this feed.

[garath]

Triage: @riarenas and @MattGal, please investigate to find the best way forward

[MattGal]

I believe this can be done with only this modification to arcade:
https://github.com/dotnet/arcade/pull/6151/files

... in conjunction I'm working with @AdamYoblick to get his DLLs he'd like indexed included in the PDB artifacts of his build (and indeed, since they're not published the typical ways, they are "PDB" artifacts) via including an `eng\publishing.props' file to do so.

[MattGal]

Adam has gotten the DLLs publishing with the PDBs, so now we wait for their next official build and see if indexing worked!

[AdamYoblick]

Yeah I didn't PR into our github repo yet, this is still internal testing. Basically I saw the dll's show up in the PdbArtifacts build artifact, but nothing published because I added the branch to the channel too late. I'm rerunning it to make sure that symbols publish. Once they do, I'll run symchk.exe against the dll's in the published vsix to make sure they have matching symbols. If they do, they we're good and I will PR to our repo 😄

[AdamYoblick]

Ok things are looking good 😄

Here's what I did to verify:

1. Get latest arcade sdk, modify winforms-designer build scripts to include the our VSIX dll's in the artifacts to publish, and run an internal build at https://dnceng.visualstudio.com/internal/_build/results?buildId=814695&view=logs&j=65907e9e-dc2e-508d-4ed4-c5ec62867dbe&t=64d9e60c-b051-596f-8314-f9b1945f4e19
2. Verified that dll's are published in the "PdbArtifacts" build artifact
3. Verified that the Symbol Publishing step passed
4. Downloaded the VSIX from the build artifacts and unzip it, and delete all the *.resources.dll files since those are just for localization (and they create noise when checking symbols)
   
   
   
   • del /a /s "<vsixUnzippedFolder>\*.resources.dll"
   
   
5. Download the Windows 10 SDK (https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk/) to get symchk
6. Run symchk against the dll's from the VSIX, like this: symchk.exe "<vsixUnzippedFolder>" /r /s SRV*http://symweb

When I ran that command, I see this:

SYMCHK: Nerdbank.Streams.dll FAILED  - Nerdbank.Streams.pdb mismatched or not found
SYMCHK: Newtonsoft.Json.dll  FAILED  - /_/Src/Newtonsoft.Json/obj/Release/netstandard2.0/Newtonsoft.Json.pdb mismatched or not found

SYMCHK: FAILED files = 2
SYMCHK: PASSED + IGNORED files = 63

The two failed files are not ours.

So I think it's safe to say that this is working properly. I'm going to PR into our github branch and verify again there, but I think we're good. 😄

/cc: @JeremyKuhne @dreddy-work @Shyam-Gupta @merriemcgaw

[AdamYoblick]

@MattGal , thank you for your help on this. Our latest build from master is at https://dnceng.visualstudio.com/internal/_build/results?buildId=816181&view=results. I've downloaded the vsix and run symchk against the dll's within, and I got no errors back.

So we should be good. Feel free to close out this issue. 😄

[MattGal]

Thanks Adam!