Diagnostics: Feature request: dotnet-symbol should acquire RedHat symbols

[noahfalk]

dotnet-symbol by default only downloads symbols for microsoft binaries. We should explore options for customers that got their builds from RedHat and want to download those binaries.

@tmds - I haven't explored this at all yet. You may already have some knowledge about what is possible today and what could be possible with future work?

[hoyosjs]

It's interesting. I've seen GDB works with debuginfo-install on RHEL (and Fedora if I'm not mistaken). You can also use it directly if you know/figure out the package name. This means that RHEL has RPM packages for the native symbols. All managed symbols are actually included in the runtime packages.

I think I saw an article from @omajid, I skimmed but didn't quite get through all the details: https://omairmajid.com/posts/2020-08-05-dotnet-symbols-in-fedora/ The only issue is we don't play well with GDB. I tried looking inside one of the debug packages, and I see full source and the native .debug files. I see also source-build packages for fedora indeed have the PDBs: https://fedora.pkgs.org/32/fedora-updates-x86_64/dotnet-runtime-3.1-3.1.6-1.fc32.x86_64.rpm.html We could index them or make dotnet-symbol aware of them in the fedora/RH package servers. Each one has its own virtues and complications, and I don't know if there's any legal restrictions to that. The second one is admittedly a lot more work.

[tmds]

I missed this issue.

Like other software that comes with the distro, debug symbols are provided as packages. @omajid are we still missing something?

The nice thing of dotnet-symbol is that it allows you to acquire symbols automagically when you give it a coredump. That doesn't work for source-build .NET. You need to install the packages on that distro matching the coredump.

cc @dagood @crummel

[noahfalk]

Thanks all! In this case I got the feedback 2nd hand so I can't be certain, but the impression I got is that the customer did ultimately obtain the symbols elsewhere (such as an installed package) but they thought the other workflow was harder and wished that they could have used dotnet-symbol as they can for Microsoft provided builds.

... That doesn't work for source-build .NET. You need to install the packages on that distro matching the coredump.

Are there specific obstacles that prevent the open source build from being indexed so that it could work with dotnet-symbol? Is this type of workflow something RedHat would be interested in supporting? All I know so far is we've got at least one interested customer but there might be more if we started asking around : )

[dagood]

I doubt we would be allowed to redistribute RHEL binaries/symbols on Microsoft symbol servers. I'm inferring this from the fact that RHEL binaries can't be distributed in publicly available Docker images. It seems to me the root problem is that it lets people tag along with someone else's active RHEL subscription. (I believe CentOS rebuilds free packages from RHEL sources, but these are not the same binaries, and we can't rely on CentOS symbols matching up with RHEL dumps.)

Now looking broader--what if we index symbols for any distro that builds free packages:

• What's the cost of indexing a large number of various distros' builds?
• Do we have to be concerned about attacks?
  
  
  
  • Would it be a problem if someone finds a distro with vulnerabilities in its build processes and uses that as a vehicle to have us upload malicious symbols to the Microsoft symbol server?
  
  
• Can we only do this for a subset of distros? (For the reasons above?) How do we decide?

It seems more reasonable for each distro (including RHEL) to stand up its own symbol server. I don't know whether distro organizations tend to have services up like this. It seems a little unlikely to me for this to seem desirable for them, though, since distros already have a way to get symbols. We also would need a way for dotnet-symbol to know which symbol server(s) to search, without user input. (What's the point if the user has to look up the URL anyway? 😛)

@omajid's article linked above actually already addresses this from Fedora's point of view:

Fedora doesn’t have its own symbol server. Fedora also doesn’t want to push to an external symbol server.

---

I think getting dotnet-symbol working with the ordinary way people get symbols on Linux distros is the clear ideal.

There's more to it: I don't think we can reasonably add support to the Microsoft build of dotnet-symbol to handle this directly for all distros. Distros have varying package managers, symbol package formats, and even naming conventions for the packages themselves. Some sort of configuration system seems necessary for this, and/or each distro needs to build their own dotnet-symbol that works with how they have things set up.

This kind of approach came up recently when thinking about whether a .NET global tool could be used to discover/acquire updates to the SDK. Similar situation: pointing people at the package manager for source-built bits.

/cc @dleeapho

[omajid]

A couple of comments:

The only issue is we don't play well with GDB.

I am not sure what we here refers to: dotnet-symbol or debuginfo-install. gdb should be asking users to run debuginfo-install for packages already. For example:

$ gdb /usr/bin/dotnet
GNU gdb (GDB) Fedora 9.1-6.fc32
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/dotnet...
Reading symbols from .gnu_debugdata for /usr/lib64/dotnet/dotnet...
(No debugging symbols found in .gnu_debugdata for /usr/lib64/dotnet/dotnet)
Missing separate debuginfos, use: dnf debuginfo-install dotnet-host-5.0.0-0.2.preview4.fc32~bootstrap.x86_64
(gdb) 

See the last bit about use: dnf debuginfo-install dotnet-host-5.0.0-0.2.preview4.fc32~bootstrap.x86_64? It's a bit of an iterative process: gdb will suggest installing the debuginfo for each binary as gdb sees that binary and probably wont suggest installing everything up front. In this example, it didn't say anything about the runtime.

debug symbols are provided as packages. @omajid are we still missing something?

Not that I know of, at least on Fedora. RHEL should be the same but maybe lagging a little bit behind Fedora? They should both work like this: managed symbols are part of the runtime and installed by default; native/unmanaged symbols can be installed using debuginfo-install or dnf debuginfo-install.

It seems more reasonable for each distro (including RHEL) to stand up its own symbol server.

Arguably, they already do that in the form of the package manager repositories :smile:

There's a debuginfo server project in the works too. See slides here. It can handle unmanaged binaries. I am not sure if any distro is planning on adapting this yet.

I think getting dotnet-symbol working with the ordinary way people get symbols on Linux distros is the clear ideal.

For dnf/yum-based distros, maybe we could make it do the equivalent of: sudo dnf install $(rpm -qf /path/to/file). Or tell users to run this command directly?

[hoyosjs]

A couple of comments:

The only issue is we don't play well with GDB.

I am not sure what we here refers to: dotnet-symbol or debuginfo-install. gdb should be asking users to run debuginfo-install for packages already.

I mean that SOS and managed debugging largely works under LLDB, but not GDB. SOS (under LLDB and dontet-dump) and dotnet-symbol use the same symbol downloader logic to get symbols at a users request while debugging. There's a way to make GDB work to debug managed code, but it was never enabled by default (in fact, it's not even guarded by an environment variable) because it increases memory usage quite a bit and it doesn't have any integration with SOS nor the symbol server - it's basically stack traces using metadata if I remember correctly.

And thanks for the slides, it looks similar (at least in contract) to what we do and that'd be a big change for Linux. I'm definitely used to having to fetch the debug packages if I'm ever investigating native distro packages.

For dnf/yum-based distros, maybe we could make it do the equivalent of: sudo dnf install $(rpm -qf /path/to/file). Or tell users to run this command directly?

This works great for live debugging. As soon as you get a core from a production service, I don't know what the experience here looks like - and this is where dotnet-symbol often shines. Does opening the core on another machine still display the debuginfo-install with the package name suggestion? My guess is it uses the same rpm -qf logic to look for packages that provide it? I guess if you don't have the file you'd need to use something like repoquery in yum-utils?

[tmds]

I guess the main feature request is to be able to download symbols on a distro other than the one where the crashdump occurred?
I believe Microsoft is making it a feature to be able to debug Linux .NET coredumps on Windows?

There is work to be able to provide these symbols out-of-band. I think Red Hat won't put effort into it unless we have users asking for this feature. The default would still be to distribute these symbols via packages. The effort would be specifically to enable the Windows debug use-case for the RHEL .NET binaries.