Rust: Stabilize --pretty=expanded

[joshtriplett]

--pretty=expanded is an incredibly useful debugging tool for macros. The change in rustc 1.19 to stop supporting unstable command-line options on the stable/beta compilers makes it all the more important to stabilize unstable options that people use, and this seems like it easily qualifies.

Much like --emit mir, this should not make any commitments about the output, only that the option should continue to exist. In particular, I'd even suggest making it explicitly clear that there's no commitment for the output to compile.

[shepmaster]

For usage in the playground, it would be lovely if it was actually --emit something, perhaps --emit macro-expanded?

[nrc]

I strongly don't want to do this. The generated code is not guaranteed to compile (because hygienic names are not reflected in the generated source text), it means a big chunk of compiler code (pretty printing) is now exposed to stable users, and I worry about bugs, etc. in that code.

I worry that this will encourage tools to try and use that output code, which is absolutely the wrong thing to do.

For debugging macros, we should be able to use expansion info in spans, plus some output saved from macro expansion to give a much better tool - look at every stage of expansion, not just the start and end, have def/ref info for each stage, etc.

[joshtriplett]

@nrc I'd love to see an even more capable mechanism, absolutely, and the idea of stepping through macro expansion step-by-step sounds incredible. But in the interim, --pretty=expanded is what we have for macro debugging, which means stable users have approximately nothing.

If you want to discourage tools from using it, stick an intentional line at the top explaining that the output format isn't stable and prevent it from compiling. But I don't think it's reasonable to take away the only mechanism stable users have for macro debugging, offer no recourse, and not yet have another solution available.

Now, that said, that does mean this will still break cbindgen and other tools which apparently process this code. And that seems quite unfortunate as well, but perhaps more justifiable. But right now, I'm a lot more worried about what to tell stable users who ask "how do I debug my macros?".

[kennytm]

@nrc Every-stage-expansion is useful, but just like we have both -Zdump-mir and --emit mir, I don't think that precludes the introduction of --emit expanded/--print expanded.

[nrc]

But right now, I'm a lot more worried about what to tell stable users who ask "how do I debug my macros?".

We tell them to use nightly. Switching compiler with rustup is so easy, that I don't think a stable requirement is a reason to stabilise an otherwise unstable feature.

[nrc]

I feel like expanded AST is different from mir, because no-one parses the MIR code and expects it to compile (at least as Rust, it would compile as its own language).

[kennytm]

@nrc using nightly doesn't always work, as the involving macro may change between rust versions (consider the difference of thread_local! between stable and nightly.)

[ehuss]

I was wondering if the output isn't compilable, could it at least be parseable? Related to #47287 I came across an issue where a macro expansion results in unparseable output. In particular, I was trying to use cargo expand on the cargo source tree, and rustfmt chokes on this idiom. Not a big deal, one can work around it by not using rustfmt, but unformatted macro expansion can be difficult to read.

// Works
fn f1() -> bool {
    macro_rules! try(( $ e : expr ) => (
        match $ e {
            Some ( e ) => e ,
            None => return false ,
        }
    ));

    try!(Some(42)) == 42
}

// Expanded version doesn't work
fn f2() -> bool {
    match Some(42) {
        Some ( e ) => e ,
        None => return false ,
    } == 42
}

// Could be OK with parenthesis (at least in this case)
fn f3() -> bool {
    (match Some(42) {
        Some ( e ) => e ,
        None => return false ,
    }) == 42
}

[fedochet]

For debugging macros, we should be able to use expansion info in spans, plus some output saved from macro expansion to give a much better tool - look at every stage of expansion, not just the start and end, have def/ref info for each stage, etc.

@nrc can you please provide more details about how we can retrieve such information (if this is possible)?

[kvark]

Note that the situation needs some kind of solution at the moment. --pretty=expanded is used by cbindgen, which helps gluing together WebRender with Gecko as well as powering other crates, like wgpu-bindings.
If stabilizing the pretty output is not an option, what would be the right way to proceed with making cbindgen robust? cc @eqrion

[eqrion]

Assuming stabilizing the output is not the right path forward, I believe something like my rust-ffi prototype is a good path forward.

I've been too busy to work on getting it to a MVP though.

[nrc]

There's also cargo-expand. We'll be looking at adding some support to the RLS which might be a better path to stabilisation that --pretty=expanded

[petrochenkov]

I'd like to revive this issue.
cbindgen is an important tool that must work on stable, if it requires pretty-printing, then so be it.

I think providing --emit expanded working on best effort basis should be acceptable, if we document it.
The pretty-printing output should at least parse successfully, pretty-printer can insert parentheses to maintain parsing priorities for code expanded from macros, if there are cases when it doesn't, then we should treat them as bugs.

[est31]

There is no need to do a full metadata build of non-build and non-proc-macro dependencies of a crate. Yes, expansion is needed, but expensive stuff like type checking or MIR building isn't. rustc still assumes this though.

In terms of real world tools, this has the effect that cargo expand is as slow as cargo check, even though it only needs expansion of most crates in the DAG and thus could be way faster.

Regarding stabilization this opens the question: would stabilization expose any property of the current code (beyond slowness) that would prevent rustc to be refactored in the future to support faster --emit expanded?

[petrochenkov]

@est31
You've seen the update that I've just posted at https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/Stabilizing.20.60--pretty.3Dexpanded.60/near/212910473?
(I don't think metadata was mentioned previously in this thread.)

I'm pretty sure that we can produce that .rmeta earlier than regular .rmeta, perhaps at cost of some code duplication in rustc and inability to import it as a regular crate.