Rust: Tracking issue for `literal` fragment specifier (RFC 1576)

[nikomatsakis]

Tracking issue for rust-lang/rfcs#1576.

[durka]

Is anyone working on implementation already? If not I can take a crack at it.

[LeoTestard]

I'll probably do it.

[LeoTestard]

Huh oh, we have an issue already: do we mean to include potentially minus-prefixed literals? If yes, we have a problem because 1. it means we have to return an Expr instead of a Lit, which means I'm not sure one will be able to use the bound fragment in a non-expr context (for example in a static array type?) and 2. it's no longer always a single-TT, so the benefits in terms of future-proofing are a bit less good. If we don't... well, we loose expressivity. I have no idea what is more intuitive though... Would you, as a Rust user, expect $l:lit to parse -5? I certainly can think of use cases for this...

[nikomatsakis]

@LeoTestard good point. I think I would have expected it, yeah...but I also agree with the complications.

[durka]

Can't we upgrade Lit to support negative literals?

[joshtriplett]

@LeoTestard I'd expect "literal" (or a future integer-literal specifier) to include -5 (negative 5), but not - 5 (the negation of positive 5).

Perhaps that expectation doesn't match reality, though. For instance, based on parsing rules for non-literals, I'd expect -5.function() to act like -(5.function()).

So, if handling both -5 and - 5 as literals makes this easier, that works. But "literal" definitely needs to include negative numbers, or we'd have a weird user trap to explain.

[petrochenkov]

@joshtriplett

I'd expect "literal" (or a future integer-literal specifier) to include -5 (negative 5), but not - 5 (the negation of positive 5).

Unary operators, including minus, are processed purely by the parser at the moment, no lexer involved. So their treatment is whitespace agnostic.
(I personally think this is a good thing.)

[LeoTestard]

I agree with @petrochenkov. -5 is not syntactically different from - 5 and has no reason to be.
But I also agree with the fact that the user will probably expect negative numbers to work as literals (even it – syntactically – they're not).

[petrochenkov]

- literal is also supported in literal patterns while arbitrary expressions are not.

fn main() {
    match 10 {
        -1 => {}
        -3 ... -2 => {}
        _ => {}
    }
}

It'd be reasonable to support it in fragments as well.

[jorendorff]

Another thing that won't match is a macro call, like concat!("hello", "world"), or any of the other macros that I think of as expanding to a literal (include_str!, env!...).

I dunno, maybe this is not such a good idea after all. Re-reading the RFC in a more skeptical light, I felt like the "Motivation" section could use some more concrete use cases. Maybe this is just failure of imagination on my part, though!

I think the likelihood of this feature attracting people who really want constant expressions should be acknowledged as a drawback, too.

[RReverser]

Another thing that won't match is a macro call, like concat!("hello", "world"), or any of the other macros that I think of as expanding to a literal (include_str!, env!...).

That's fine IMO. Personally I'd like it to match only explicit literals in code to distinguish, for example, between literal and identifier and produce different code for them. Right now this is not possible because all of tt, expr, ident are too broad and conflict with each other.

[da-x]

Hi All,

I have implemented this feature in my fork of Rust over 1.25.0, under this branch:

https://github.com/da-x/rust/tree/literal-fragment

Following a quick review by a rustc mentor I may have the free time to provide a full pull request for this.

[da-x]

BTW B-unstable label has the tooltip "implemented in nightly and unstable" but I did not see it there, i.e. commits & pull requests referencing this. Did I miss it?

[da-x]

@nikomatsakis

The implementation that I worked on was merged, yay!

[clarfonthey]

Is there anything stopping an FCP for this?

[joshtriplett]

@clarcharr It's currently merged in unstable; do you mean an FCP to declare it stable?

[clarfonthey]

Yes, an FCP to stabilise. Should have clarified.

[jendrikw]

Currently, it is not possible to pass a literal from a macro to another macro. link to playground. I think this is a bug.

[dtolnay]

Interesting! That is a bug. I opened #52169 to track it.

[RReverser]

@jendrikw Agreed, this should be allowed just like passing any other node types.

[0e4ef622]

Is there anything else preventing this from being stabilized? The issue above has been fixed.

[Centril]

This has baked for some time now and I believe it should be ready for stabilization.
So let's start the review.

@rfcbot merge

The feature gate test is here:

• https://github.com/rust-lang/rust/blob/master/src/test/ui/feature-gates/feature-gate-macro-literal-matcher.rs

Unstable book:

• https://github.com/rust-lang/rust/blob/master/src/doc/unstable-book/src/language-features/macro-literal-matcher.md

Tests defined here:

• https://github.com/rust-lang/rust/blob/master/src/test/ui/run-pass/macros/macro-literal.rs

[rfcbot]

Team member @Centril has proposed to merge this. The next step is review by the rest of the tagged teams:

• [x] @Centril
• [x] @aturon
• [x] @cramertj
• [x] @eddyb
• [x] @joshtriplett
• [ ] @nikomatsakis
• [x] @nrc
• [x] @pnkfelix
• [x] @scottmcm
• [ ] @withoutboats

No concerns currently listed.

Once a majority of reviewers approve (and none object), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[scottmcm]

Reading through the tests, I see that -2 is a literal.

While that makes sense to me, I remember hearing that it wasn't, but was instead a negation operator and a literal. Am I misremembering?

[eddyb]

cc @alexcrichton @dtolnay @petrochenkov on https://github.com/rust-lang/rust/issues/35625#issuecomment-421616280

[Centril]

@scottmcm

While that makes sense to me, I remember hearing that it wasn't, [...]

I agree that it makes sense.

I would say that even if -2 is interpreted as negation + a literal, I would probably still want the literal macro matcher to accept -2 since that behavior seems more intuitive. In other words, I would see -2 being -(2) more as an implementation detail than what a literal is in the mental model.

[clarfonthey]

I think that the nuance with -2 causes a bit of a problem, because I don't think it should be treated as a literal, but I also don't think that the current implementation of literal would allow detecting it either. You could naïvely allow $(-)?$lit:literalbut that would allow -"hello" which is clearly wrong.

[da-x]

@clarcharr there are tests currently implemented that verify that -2 is detected as a literal.

[clarfonthey]

I worded that wrong; I meant that only having a literal matcher and not being able to distinguish strings and numbers means that you can't easily make -2 not a literal.

[alexcrichton]

@scottmcm for procedural macros the input as lexed by the compiler is - 2 (two tokens) but procedural macros can create Literal tokens that are negatives (like -2). In that sense matching -2 for :literal makes sense to me.

[Jezza]

Are literals generated by the compiler counted as true literals?
A quick example would be something like:

macro_rules! bind {
    ($type:ty) => {
        bind!($type, concat!(stringify!($type), ".html"));
    };
    ($type:ty, $path:literal) => {
        println!("Page: {:?}", stringify!($type));
        println!("Path: {}", $path);
    };
}

This fails to match, as the expansion, I guess, happens after the selection.

I just naively assumed that would have worked.

[dtolnay]

concat!(stringify!($type), ".html") is not a literal generated by the compiler, it is an identifier followed by an exclamation mark followed by a set of parentheses containing some tokens. See for example:

macro_rules! m {
    ($first:tt $second:tt $third:tt) => {
        println!(stringify!($third));
        println!(stringify!($second));
        println!(stringify!($first));
    };
}

fn main() {
    m! {
        concat!(stringify!($type), ".html")
    }
}

If you were to use it in expression position, it would be interpreted as an invocation of concat! which expands to a string literal. In this case and in your bind macro though, it is never in a syntactic position where it would be treated as an expression.

[Jezza]

I can understand semantically why it doesn't make sense, I just thought it would have, but that being said, I already changed it to $path:expr, so there's no issue.

It just made more sense to me as a literal, because according to the compiler, it _could_ generate a literal from it.
I guess it was more a question than an actual issue.

[petrochenkov]

concat!(stringify!($type), ".html") is not a literal generated by the compiler, it is an identifier followed by an exclamation mark followed by a set of parentheses containing some tokens.

This is true, but there's more to it!
A few built-in macros can expand their arguments eagerly through \~m a g i c~, so they accept concat!(...) and likes even if they expect string literals:

fn main() {
    println!("{}", env!("PATH")); // OK
    // println!("{}", env!(10)); // ERROR expected string literal
    println!("{}", env!(concat!("PATH"))); // OK
}

Fortunately (or not), this currently cannot be done in user-defined macros.

[Jezza]

I actually just realised that I can't change it to $path:expr, as it's used in include_str.
So, I've just had to manually unroll the macro.

It doesn't look pretty. :(

Ninja edit:
I just tested what actually happens if you try to pass in a non-literal with expr.
It rejects it the exact same way.
Does the include_str already make use of this feature?
If not, the include_str seems to propagate the required token.
Oh, unless the compiler steps in... That's actually probably the case...

I've reverted it back to the :expr type.

[rfcbot]

:bell: This is now entering its final comment period, as per the review above. :bell:

[rfcbot]

The final comment period, with a disposition to merge, as per the review above, is now complete.

[Centril]

@da-x Since you implemented this, would you be willing to write up the stabilization PR?

[da-x]

@Centril sure

[jethrogb]

This appears to be missing from the Rust reference