Rust: Tracking issue for plugin stabilization (`plugin`, `plugin_registrar` features)

[aturon]

Tracks stabilization for the plugin and plugin_registrar feature gates.

[aturon]

Current status: awaiting a complete revamp of the plugin system.

cc @nrc

[jimmycuadra]

Is anyone actively working on this? Curious what the status is since using Serde without the need for code generation with syntex on stable Rust is very high on my Rust wishlist. :smiley:

[steveklabnik]

@nrc is

[aturon]

In particular, check out the discussion here, which points to a bunch of blog posts by @nrc amongst other things. He plans to finish the blog series and post an RFC soon.

[jimmycuadra]

Thanks Aaron—I had read a couple of the posts about the macro system but it didn't click that this was intended to replace compiler plugins for the purpose of syntax extensions. It makes sense now, and looks great. I can't wait. :}

[knight42]

Any progress now?

[SimonSapin]

Yes! The "Macros 1.1" RFC at https://github.com/rust-lang/rfcs/pull/1681 was accepted. It’s implementation is tracked at https://github.com/rust-lang/rust/issues/35900.

[SimonSapin]

The one remaining plugin in Servo is a register_late_lint_pass that analyses use of GC-managed types to make sure they’re "rooted" correctly. It traverses expressions HIR expression inside functions and methods, and requires access to the resolved type of these expressions as well as attributes on the definitions of these types. (Looking for custom attributes that indicate which types must be rooted. These attributes also need to be allowed by the compiler.)

Could this kind of analysis be built on top of RLS? Could that be a way forward for custom lints without relying on unstable rustc internals?

[eisterman]

Any progress now?

[0rvar]

Are there any plans for stabilizing plugins in the foreseeable future?

[oli-obk]

The plan is to remove the plugin API entirely.

[vityafx]

@oli-obk Could you go into details please?

[oli-obk]

Even if we stabilized the plugin interface, it would be useless, because there is no stable API it offers. Instead things like proc macros 2.0, custom derive, ... are stabilized by offering an interface that does not require the language to guarantee a plugin interface to the compiler. All of these non-plugin-requiring APIs can be implemented with various other schemes, even if they are currently implemented via plugins in the background.

Note: this issue is not about allowing writing bin crates which can be extended via plugins, it is about plugins that can extend the compiler.

[vityafx]

Is not rocket a plugin to a compiler? If so, that means we will bury it alive?

[oli-obk]

Rocket is a plugin to the compiler because proc macros 2.0, and most notably their attribute support is not implemented yet. Of course there is no desire to eliminate any uses of the plugin API without a viable alternative. But there is also no desire to stabilize it while there are much better alternatives on the horizon. Especially if stabilizing has a high cost for the compiler, while the alternatives do not have the same cost associated with them.

[Centril]

@SimonSapin What's the status on Servo's use of this?

[SimonSapin]

Unchanged. https://github.com/servo/servo/blob/c7cd1b83a11/components/script_plugins/lib.rs uses #[plugin_registrar], register_late_lint_pass, and register_attribute.

[oli-obk]

Is there a tracking issue for moving to a Pin based GC in servo?

[SimonSapin]

Is there a tracking issue for moving to a Pin based GC in servo?

The current effort around GC safety is at https://github.com/asajeffrey/josephine/, but as far as I know it does not involve std::pin::Pin. @asajeffrey can say more.

[montao]

The plan is to remove the plugin API entirely.

I am just learning it. Should I not? I think I got a working example

[oli-obk]

I am just learning it. Should I not?

I personally would suggest not to use the API, especially not to add new uses of it.

What is your use case? Maybe we can find an alternative way for you to reach your goal without using the plugin API

[montao]

I am just learning it. Should I not?

I personally would suggest not to use the API, especially not to add _new_ uses of it.

What is your use case? Maybe we can find an alternative way for you to reach your goal without using the plugin API

I have a metaprogramming project with Basile in Paris who is a research engineer, he mostly works in C and generates many plugins using dlopen in C. We are looking for alternatives to C. We looked at Go but Go includes the runtime in the plugin which creates too large files if we generate thousands of plugins.

So far I think that Rust is the only alternative to C in doing this. We really tried hard to achieve it with Go which was our first choice but right now it looks like the changes and fixes we would have to make to the Go build system are too many and too complicated. Therefore I was glad when I got this to work with Rust. I could create a project with several plugins and I can also make the code generate custom plugins at runtime, and it seems that these will be dynamically linked using Rust which is suitable for our case.

My Rust code with the plugins https://gitlab.com/montao/rust-multiplugins

My Go code with the plugins, which includes the runtime and therefore duplicates code https://gitlab.com/montao/forbasile

[oli-obk]

Looking at the Rust examples, you can achieve the same thing with procedural macros. They are documented here and above that in more detail

I believe these do cover your use cases. Please let me know if that assessment is wrong.

[osor-io]

First of all, my apologies if I'm asking for features that are possible to do some other way or if I'm missing some key parts of the plugin/macro systems 😓

I've been investigating this side of Rust for not that long but I'm very excited to try to do some things in it 😀

I've yet to investigate it further but it does seem like the case that the plugin features offer everything that macros do right? Can you do something like annotate a file once and then wrap every procedure with a profiler related tag or something?

I'd like to do something like the following

#[profile_every_procedure]

fn do_this () {
    . . . some code . . .
}

fn do_that () {
    . . . some code . . .
}

fn do_some_other_thing (argument : u32) {
    . . . some code . . .
}

And get my plugin/macro/whatever will replace plugins to give me this:

fn do_this() {
    start_profiling_tag("procedure: do_this");
    . . . some code . . .
    end_profiling_tag();
}

fn do_that() {
    start_profiling_tag("procedure: do_that");
    . . . some code . . .
    end_profiling_tag();
}

fn do_some_other_thing (argument : u32) {
    start_profiling_tag("procedure: do_some_other_thing, inputs: [argument : u32]");
    . . . some code . . .
    end_profiling_tag();
}

In general I'd like to now have to do excessive tagging, just to enable a plugin to do something. Another feature that would help me immensely would be the possibility to have some state in my compiler plugin/macro so when my callback happens I can store something in there that I can read in future calls. For example, let's say I'm generating Rust code based on something and I generate a new struct that I insert into the AST called struct Input and later I happen to want to generate some more rust code that also needs to generate a struct Input. I'd love to be able to know that I already generated that struct, for some arbitrary reasons that I stored in my own structure, and then decide if I want to not generate the new struct, do it with some other name, etc...

Would all of these things be possible with the Macro system or with something that replaced plugins? Or are they possible to do without the need for nightly in a not very painful way (or at all?)?

Sorry if I'm missing important bits here or if I don't understand some fundamental facts about Macros and plugins. I just got scared at the plugin features going away since I believe they could add so much to the language and I was very excited to use them only to discover that they are only nightly, their use is discouraged and they might just dissapear 😢

Cheers! 😄

[oli-obk]

Yes I do believe everything you just described is possible with procedural macros.

You are specifically looking for procedural attribute macros. Though you'd use #![profile_every_procedure] (note the !) to make sure it applies to the entire module and not just the next function.

Though I think the way you are describing it, you'd want this in the crate root and process the entire crate at once. This way you have a global view on the entire crate and can do a global analysis (wrt generating structs only once and such).

Sorry if I'm missing important bits here or if I don't understand some fundamental facts about Macros and plugins. I just got scared at the plugin features going away since I believe they could add so much to the language and I was very excited to use them only to discover that they are only nightly, their use is discouraged and they might just dissapear

the plugin system was a stopgap solution until we got proper procedural macros. No features are disappearing, they are just getting implemented in a sane way that can move to stabilization at some point. The plugins, by their very definition are inherently unstable.

[osor-io]

It looks like I was looking too much at plugins while ignoring the better procedural macros indeed 😄

I get that there's ways to keep state between calls to procedural macros? I'd like to have more than one macro that reads from some shared state to be able to do some code generation that requires me to get some context on what some other macros might have done already 😓 I hope this isn't pushing it too much.

Oh and thanks for the quick response!

Cheers! 😃

[oli-obk]

Well.. procedural macros can do whatever they want. They can even write to files or access the network. Whether that is a good idea is a different question. They are not meant to share state (neither were plugins meant to do that). Shared state during compilation interferes with incremental compilation and may result in nondeterministic builds.

If, on the other hand, you create a single crate-wide attribute, that attribute can see the entire crate at once and generate code inside it. So you don't need shared state between instances, but can just do a global analysis.

[Centril]

They are not meant to share state (neither were plugins meant to do that). Shared state during compilation interferes with incremental compilation and may result in nondeterministic builds.

Moreover, we do not guarantee a specific order of execution if you write something like:

#[my_macro_a]
fn foo() { ... }

#[my_macro_b]
fn bar() { ... }

so there is no guarantee that if you save some state in #[my_macro_a]'s execution, you'll be able to access it in #[my_macro_b]. If you save things in statics, we also don't guarantee that two consecutive macro invocations will see the same results.

[osor-io]

Oh, I see 😄 It really would be great to be able to save some state and establish dependencies between macros, or some other system to get a guaranteed order of executions about the macros you care about. I know that a crate wide macro that looks at everything is possible but keeps you from using other macros as helpers for the main one.

[crlf0710]

@osor-io Just in case you haven't found it out, the macro wizard @dtolnay have brought us this crate:
https://github.com/dtolnay/inventory

[syntacticsugarglider]

What's going to replace the support for custom lints provided by lint plugins? Some sort of alternative seems like a good idea.

[SimonSapin]

At this point there is no replacement. See https://github.com/rust-lang/rust/issues/62868 for some discussion of this.