Rust: Tracking issue for RFC 2603, "Rust Symbol Mangling (v0)"

[Centril]

This is a tracking issue for the RFC "Rust Symbol Mangling (v0)" (rust-lang/rfcs#2603).

Current status:

On nightly, you can control the mangling scheme with -Z symbol-mangling-version, which can be:

• legacy: the older mangling version, still the default currently
• v0: the new RFC mangling version, as implemented by #57967

To test the new mangling, set RUSTFLAGS=-Zsymbol-mangling-version=v0 (or change rustflags in .cargo/config.toml). Please note that only symbols from crates built with that flag will use the new mangling, and that tool support (e.g. debuggers) will be limited initially, until everything is upstreamed. However, RUST_BACKTRACE and rustfilt should work out of the box with either mangling version.

Steps:

• [x] Implement the RFC (https://github.com/rust-lang/rust/pull/57967 + https://github.com/alexcrichton/rustc-demangle/pull/23)
• [ ] Upstream C implementation of the demangler to:
  
  
  
  • [ ] binutils/gdb (GNU libiberty)
  
  
  • [x] [[PATCH] Move rust_{is_mangled,demangle_sym} to a private libiberty header.
    
    ](https://gcc.gnu.org/pipermail/gcc-patches/2019-June/523011.html) committed as https://github.com/gcc-mirror/gcc/commit/979526c9ce7bb79315f0f91fde0668a5ad8536df
  
  
  • [x] [[PATCH] Simplify and generalize rust-demangle's unescaping logic.
    
    ](https://gcc.gnu.org/pipermail/gcc-patches/2019-August/527835.html) committed as https://github.com/gcc-mirror/gcc/commit/42bf58bb137992b876be37f8b2e683c49bc2abed
  
  
  • [x] [[PATCH] Remove some restrictions from rust-demangle.
    
    ](https://gcc.gnu.org/pipermail/gcc-patches/2019-September/530445.html) committed as https://github.com/gcc-mirror/gcc/commit/e1cb00db670e4eb277f8315ecc1da65a5477298d
  
  
  • [x] [[PATCH] Refactor rust-demangle to be independent of C++ demangling.
    
    ](https://gcc.gnu.org/pipermail/gcc-patches/2019-November/533719.html) (original submission) committed as https://github.com/gcc-mirror/gcc/commit/32fc3719e06899d43e2298ad6d0028efe5ec3024
  
  
  • [x] [[PATCH] Support the new ("v0") mangling scheme in rust-demangle.
    
    ](https://gcc.gnu.org/pipermail/gcc-patches/2020-November/558905.html) (original submission) committed as https://github.com/gcc-mirror/gcc/commit/84096498a7bd788599d4a7ca63543fc7c297645e
  
  
  • [ ] lldb
  
  
  • [ ] Linux perf
  
  
  • [ ] valgrind
  
  
• [ ] Adjust documentation (see instructions on rustc-guide)
• [ ] Stabilization PR (see instructions on rustc-guide)

Unresolved questions:

• [x] [Punycode vs UTF-8](https://github.com/rust-lang/rfcs/blob/master/text/2603-symbol-name-mangling-v2.md#punycode-vs-utf-8), some prior discussion in https://github.com/rust-lang/rust/issues/7539
• [x] [Encoding parameter types for function symbols](https://github.com/rust-lang/rfcs/blob/master/text/2603-symbol-name-mangling-v2.md#encoding-parameter-types-for-function-symbols)

[eddyb]

cc @rust-lang/compiler

I'd add this as an unresolved question: https://github.com/rust-lang/rfcs/pull/2603#issuecomment-489649554

I'd rather change the grammar now, if ever, before merging https://github.com/rust-lang/rust/pull/57967.

[michaelwoerister]

I think the Punycode vs UTF-8 and Encoding parameters types for function symbols questions are actually resolved. The RFC text contains a recommendation on what the resolution should look like (since before the motion to merge) and nobody objected.

[eddyb]

I tried the "type/value namespace" shorthands, and this is what I got, stats-wise:
(apologies for having two tables, but it was hard to combine them further)

986807 symbols: sizes and compression ratios (▶)

v0 without shorthands (types use Nt and values use Nv)

| | legacy | mw | ▶ | mw + compress | v0 | ▶ | v0 + compress |
| ---: | --- | --- | --- | --- | --- | --- | --- |
| min | 30 | 25 | 0.97x | 25 | 25 | 0.96x | 25 |
| avg | 81 | 313 | 1.46x | 193 | 318 | 1.47x | 194 |
| max | 688 | 13409 | 14.83x | 1115 | 13759 | 15.24x | 1105 |
| total | 76.6MiB | 295.4MiB | 1.62x | 182.0MiB | 300.2MiB | 1.64x | 183.5MiB |

v0 with shorthands (types use Z and values use V)

| | legacy | mw | ▶ | mw + compress | v0 | ▶ | v0 + compress |
| ---: | --- | --- | --- | --- | --- | --- | --- |
| min | 30 | 25 | 0.97x | 25 | 24 | 0.96x | 24 |
| avg | 81 | 313 | 1.46x | 193 | 303 | 1.48x | 184 |
| max | 688 | 13409 | 14.83x | 1115 | 12925 | 14.99x | 1047 |
| total | 76.6MiB | 295.4MiB | 1.62x | 182.0MiB | 285.5MiB | 1.64x | 173.9MiB |

So that's around 5% reduction, kinda nice, but not that much.

[robinmoussu]

I just opened a bug in compiler-explorer because I noticed collision in the demangled name of different monomorphisation of the same function. As far as I understand, the current mangling scheme of rust still use the v1 witch doesn't contains the required information, while this proposed v2 would solved it? Did I correctly understood what the current situation is?

[eddyb]

@robinmoussu As per #57967, you can set RUSTFLAGS=-Zsymbol-mangling-version=v0 on nightly.
EDIT: maybe we should update this tracking issue with that information?

[crlf0710]

@eddyb @michaelwoerister
I wonder if it's possible/feasible to put some "mark" into the mangled symbol name to indicate that this stack frame is preferred not to display in backtraces, #68336.

[eddyb]

Could just rely on a _ prefix or something similar, but either way I would prefer not to add something that marginally related, to the symbol mangling syntax.

[eddyb]

Updating the RFC's "v2" to "v0" as per https://github.com/rust-lang/rust/pull/57967#issuecomment-463596992 (should've done it back then).

[eddyb]

I've just submitted "[PATCH] Support the new ("v0") mangling scheme in rust-demangle.", to the gcc-patches mailing list (yes, the demangler used by binutils and gdb is in libiberty, the main copy of which is in GCC - would've been easier to contribute to it if it were separate, oh well).

If everything goes well, once that's merged we'll be able to also upstream the same implementation to lldb, Linux perf, valgrind, etc. - that's been the main blocker for rolling out the v0 mangling.

[eddyb]

Potential change we might want to make to const value mangling: https://github.com/rust-lang/rust/issues/61486#issuecomment-599085848.

The tricky aspect is whether we change the way _: usize is encoded today, from jp to p, or we leave it as-is so that demanglers can keep working unchanged.

EDIT: AFAIK, that situation can only be reached with #![feature(const_generics)], so we should be able to defer it to whenever we get around to #61486.

EDIT2: just checked and my 1 million symbols dataset doesn't contain a single instance of jp for an array length, so I think we can remove support for it.

[eddyb]

Quick update: I haven't received a response on the final patch in 2.5 months.

This means I can probably resubmit with the placeholder constant syntax removed (see my previous comment, i.e. https://github.com/rust-lang/rust/issues/60705#issuecomment-599087150), and/or maybe we can even implement some form of ADT mangling in the interim, since it doesn't seem to be that difficult (see https://github.com/rust-lang/rust/pull/71232 for something similar).

But this is disappointing and another delay in getting the new mangling scheme to be supported in native tooling, which would've ideally been all dealt with last year.

[mark-i-m]

Perhaps an MCP could be used to get it on the compiler team radar and push it through?

[eddyb]

@mark-i-m What do you mean? A Rust MCP won't make GCC devs suddenly respond to my emails.

[mark-i-m]

Oh wait it won't 🤯

Lol, sorry, I misunderstood.

[bstrie]

So support for this in rustc is completely implemented at this point, and now we're just waiting for tooling vendors to accept patches? If so, rather than waiting passively does anyone have contacts in the GCC project who we can reach out to to shepherd this along?

[benesch]

I've asked Ian Taylor to look at it, since he's reviewed many of the last patches to the demangler: https://gcc.gnu.org/pipermail/gcc-patches/2020-October/557390.html If Ian is busy, perhaps we can ask Jeff Law or Jakub Jelinek, who have committed the prior patches.

[eddyb]

@benesch's updated version of the patch just landed :tada: (as https://github.com/gcc-mirror/gcc/commit/84096498a7bd788599d4a7ca63543fc7c297645e)

It includes ports of @varkor's 3 PRs related to const generics:

• https://github.com/alexcrichton/rustc-demangle/pull/39 "Add demangling for bool, signed integers, and char"
• https://github.com/alexcrichton/rustc-demangle/pull/40 "Elide the type when the const value is a placeholder p"
• https://github.com/alexcrichton/rustc-demangle/pull/41 "Print char using {:?}"

Thanks to @benesch for updating the patch, pushing it through, and for @jwakely for committing it!

[benesch]

And thank _you_, @eddyb, for doing all the heavy lifting here! I'll be very excited when we can stabilize the new mangling scheme.

[bstrie]

eddyb previously wrote:

If everything goes well, once that's merged we'll be able to also upstream the same implementation to lldb, Linux perf, valgrind, etc. - that's been the main blocker for rolling out the v0 mangling.

So is that the next step? I'm unsure if by "upstream" that means that the same code will need to be landed in all those projects manually, or if it's implying that those are all using libiberty and we'll just need to wait for them to upgrade their dependency.