Roslyn: Add symbol files(.pdb/source) to NuGet packages

[DavidKarlas]

It would be nice to be able to step inside Roslyn code when using NuGet packages.

This is nice guide on how to do this:
http://blog.davidebbo.com/2011/04/easy-way-to-publish-nuget-packages-with.html

[jaredpar]

I agree it would be nice if we hooked into the source and symbol information into our packages.

@agocke do we have an issue tracking this yet?

[agocke]

Is this not the issue? ☺

From: Jared Parsons [mailto:[email protected]]
Sent: Wednesday, January 14, 2015 4:34 PM
To: dotnet/roslyn
Cc: Andy Gocke
Subject: Re: [roslyn] Add symbol files(.pdb/source) to NuGet packages (#3)

I agree it would be nice if we hooked into the source and symbol information into our packages.

@agockehttps://github.com/agocke do we have an issue tracking this yet?

—
Reply to this email directly or view it on GitHubhttps://github.com/dotnet/roslyn/issues/3#issuecomment-70020597.

[jaredpar]

@agocke i didn't know if we had an existing internal one tracking it :)

[agocke]

Nope, I asked David to file it here

[Pilchie]

@jasonmalinowski mentioned that it would be nice to have those PDBs stamped for source server info from github as well.

[davkean]

We should make sure that both corefx and Roslyn use the same mechanism here.

[DavidKarlas]

It would be also nice if you could make git tag for NuGet releases.

[jasonmalinowski]

Yep, we haven't migrated our historical branches from CodePlex yet. I think @jaredpar is working on that.

[johnkoerner]

Is there any update on this? I am seeing a crash and would like to step into Microsoft.CodeAnalysis.Workspaces to see what is causing it.

[jaredpar]

@johnkoerner this is something we are still looking into actively. It fell off the radar while we were pushing for 2015 RTM but started picking it up again last week.

[amcasey]

Are we still planning to do this for 1.1?

[ManishJayaswal]

yes we should do this for 1.1.

[agocke]

Actually, after internal discussion I don't think we should do this at all. We upload our symbols to the Microsoft symbol servers, so the symbols should be available via the public servers.

[davkean]

The symbols we publish point exactly to what source that external users have access to?

[davkean]

Usually symbols have a command to pull it from source control, say SD, TFS, or even GIT from VSO. But I've yet to see a symbol that MS has published that has pulled from Github, which would be needed for non-Microsoft employees.

[davkean]

We'd basically need to use something like: https://github.com/GitTools/GitLink

[ManishJayaswal]

@agocke do our pdbs have source server information? Maybe we should use GitLinks
https://github.com/GitTools/GitLink

[Pilchie]

I'm pretty sure we lost source server info when we switched our internal signed builds to git, so I agree we should get it back using GitLink. Also, we should verify that the symbols we publish on the external symbol server are full symbols, and not stripped symbols that don't have source/local information.

[paul1956]

Will there be any way to use the local Roslyn source on my machine or will an Internet connection be required? Really appreciate this in either case.

[amcasey]

@paul1956 Generally, if VS can't find the sources online it will offer to let you use local sources. Of course, at that point, you'd be on your own to sync to the right commit. If I find that GitLink has somehow solved that problem, I'll let you know - I'm still investigating.

[amcasey]

I've switched to investigating SourceLink since it seems to have more knobs (we have a fairly elaborate tree of solutions) and special handling for local line endings that don't match repository line endings.

[amcasey]

Over to @agocke to continue the investigation of SourceLink.

[tmat]

@agocke @amcasey I met the author of SourceLink at MVP summit. We are going collaborate on a source info solution for both native and portable PDBs.

[clairernovotny]

Just bringing the comment I added into the NuGet issue repo here as well as it's germane:

The current mechanism for symbol handling in NuGet, pushing symbol packages to symbol source, is no longer sufficient for some usages. The .NET Native tools require symbols to be available alongside the dll at build time, not on-demand at debug time as loaded from a symbol server.

Including symbols, especially source-indexed symbols, alongside the dll in the main package is less than ideal for a few reasons: they're large and vastly increase the package size for everyone, and it hinders "Just My Code" debugging by treating all packages as "my code."

I propose one of two options:

1. Enable publishing of .symbols. packages to NuGet.org and treat symbols packages like a satellite locale package -- upon install/restore, the contents are copied to the "parent" package in the right TFM directory. As a possible optimization to symbols package size, an additional naming convention can be used to split out on pivots -- say symbols.uap10.0-2.0.0.nupkg (the exact convention to be worked out...).
2. Start with 1, but don't copy symbols to the parent package. Instead, make the build pipeline "NuGet aware" so that the symbols are picked up in the right places. The same should be done with the locale satellites too, especially given the shared nature of the local package store now.

[ctaggart]

I'm the author of SourceLink. I met with @tmat last week. The portable pdb format looks awesome! I commented at https://github.com/aspnet/Universe/issues/131#issuecomment-154227813 last week, but this is a more appropriate place. The size of the portable pdb files is several times smaller. I recommend including them in the nupkgs, but let's use https://github.com/NuGet/Home/issues/1696 to get NuGet team's blessing. I'll work with @tmat to get the source indexes in the portable pdb files.

[tmat]

Note that although the Portable PDBs are the future it will take a while till all relevant tools (debuggers, symbol servers, source info, IL rewriters, etc.) will have support for Portable PDB. Meanwhile we need solution for the current PDBs.

[ctaggart]

My understanding is that Microsoft.DiaSymReader.PortablePdb allows existing Windows debuggers that use that COM API for debugging to work with the new format. Is that correct?

[tmat]

That is correct. However, it hasn't been released yet and currently it requires the app to be changed in order to use it. It might be possible, using some COM activation magic, to switch existing app from native diasymreader to the Microsoft.DiaSymReader.PortablePdb without changing it via a manifest file. But we haven't had time to figure it out yet.

[agocke]

This issue is no longer clear or clearly Roslyn-related. I'm going to close it unless someone has a clear ask for Roslyn right now.

[davkean]

I think the ask is still here - they want symbols to able to debug Rolsyn packages. Without us doing something, that's not going to happen.

[Pilchie]

I have a clear ask: For each published nuget package, there should be _some way_ to get full pdbs that include source information, and those PDBs should be stamped with source information to allow retrieval of the sources.

That won't deal with generated code, but will still improve the situation tremendously.

(I'm not specifying what way it is. It could be a zip file that you download separately as far as I'm concerned).

[ctaggart]

Yes, my ask is to please begin packaging the portable pdb files with the NuGet packages. Separately, I'l work with @tmat to get the source index into the portable pdb files. I will have time soon to work on that.

[agocke]

@davkean The symbols are available. They're uploaded to the Microsoft PDB server.

@Pilchie So the issue here is now source linking.

[davkean]

We upload public symbols?

[agocke]

@ctaggart Why portable PDBs?

[agocke]

@davkean I thought so.

[Pilchie]

Can we verify? I thought the symbols we published to the public symbol server were source stripped.

[agocke]

@Pilchie Yes, this should be verified.

[ctaggart]

@agocke Portable PDBs because
1) they are compact enough to put in the NuGet packages (several times smaller)
2) they work cross platform

[DavidKarlas]

About source code... Imho, just make git tags for every NuGet package version uploaded to NuGet.org... I think everyone debugging Roslyn code is capable of git clone roslyn && git checkout sha1 of tag that matches NuGet version....

[DavidKarlas]

Looks like git tags are already done for last few NuGet packages so I guess thats fine, just keep doing it ;)

[Pilchie]

@DavidKarlas we definitely plan to keep tagging source corresponding to NuGet packages, but you still can't debug if we don't publish the _full_ PDBs that include line number info instead of stripped symbols.

[clairernovotny]

I still argue that publishing to the Microsoft public symbol servers isn't good enough. It's too slow and they're not available at compile-time....this needs to go hand-in-hand with getting some better nuget support for symbols, either directly in the package or by putting the .symbols. packages on nuget.org and having rules around that.

It's important that the either live pdb's live alongside the dll's on disk at compile time or NuGet does the right thing to make sure they wind up in the output directory. This is to ensure that the pdb's are in the ILC toolchain early enough to be munged correctly to enable source debugging of nuget packages from .NET Native....which is where many errors tend to crop up.

[jaredpar]

@onovotny do you have an example of an OSS project that pushes .symbols packages that we could look at?

[clairernovotny]

@jaredpar Right now there .symbols packages don't work properly in the build/toolchain pipeline. They go to symbol source and that's not useful. That's the part that needs design/impl. @terrajobst asked me to open an issue to discuss it and thus we have https://github.com/NuGet/Home/issues/1696.

All of xUnit's NuGet's include source-indexed pdb files in the main NuGet package right now but we'd move to whatever the community decides is the right place to put them (aside from symbol servers).

[davkean]

We discussed this as a team, and came to the following conclusions:

This issue is not tracking the general problem of getting symbols for NuGet packages. If this is addressed by the community/NuGet/VS/.NET such in https://github.com/NuGet/Home/issues/1696, then we'll happily move to such a solution, but until then is solely attempting to solve the following scenarios:

1) As an owner of an analyzer, refactoring or code fix or a user of the Roslyn packages, I can step into Roslyn without cloning it to help me diagnose issues and behaviors with my extension/application.

2) As a contributor to Roslyn, I can attach to a released version of Visual Studio and debug and step into Roslyn to figure out bugs and issues without needing an connection to Microsoft's internal symbol server.

Here's the plan of action:

1) Figure out if we can just put symbols with external source links to Github on either the external Microsoft Symbol Server, or reuse the existing one that is used for .NET Framework stepping. Owner: davkean

2) Then, hook this up into our internal signed build pipeline with SourceLInk to point back to GitHub and publish that to the source server in 1). Owner: @dotnet/roslyn-infrastructure's backlog

[JoshVarty]

I'm interested in stepping through from my VS Extension to Roslyn source.

Is it possible to use PDBs built by compiling Roslyn locally? Does it matter that VS uses signed builds of Roslyn that we cannot generate ourselves? I've tried using Update 2 and the Update 2 RC tagged release.

I've been trying for a couple hours to get this to work, but I must be making a mistake.

[davkean]

You can't locally build PDBs and expect them to work against shipped binaries (not at least, until our determinism support is fully baked). Instead, the easiest way is to simply build Roslyn.sln, which will deploy all our binaries as extensions into the "RoslynDev" hive. Run VS from that hive (via devenv /RootSuffix RoslynDev, and you'll be able to run and debug against the local sources.

[paul1956]

@davkean could you give better directions on how to do this? Assume I know how to build Roslyn,
do I hit F5 on Roslyn.sln and then in the new instance of Visual Studio load my VSIX solution and hit F5 and in the 3rd instance load the solution I want to run my extension code on? Somehow I don't think this is what you mean. Somehow my VSIX needs to use the Roslyn libraries I just build from source and I am missing that connection.

[JoshVarty]

Yeah I tried this today. I can deploy my extension to the /RoslynDev hive, but then it just loads the official Roslyn DLLs from the Visual Studio folder in Program Files (x86).

I think the work around is:

1. Deploy my extension to the /RoslynDev hive.
2. Deploy Roslyn to the /RoslynDev hive. (My extension will now be loaded as well)
3. Place breakpoints in Roslyn source code that I want to step through.
4. Trigger those code paths from my extension.

[jflepp]

+1

[jasonmalinowski]

@tmat I think you get the honor of closing the oldest active issue in Roslyn. :smile:

[tmat]

@jasonmalinowski Cool. My pleasure. Fixed by https://github.com/dotnet/roslyn/pull/23038.

[DavidKarlas]

THANK YOU!!! When will this come to master?