Rust: proc_macro support for reading files/strings to spanned TokenStream

[mystor]

Currently, as far as I am aware, there is no way to get spans for tokens parsed from a source other than the macro's callsite, and no good way to tell rustc & cargo a procedural macro depends on another file. It would be nice for macros which want to load other files to enable this.

It'd be nice to support some mechanism for loading a file as a TokenStream. Ideally this would take the form of a method in the crate directly parsing a file / string to a TokenStream. For example:

1. A function to load a file as a TokenStream with accurate span info etc, e.g.

   fn lex_file(path: &Path) -> io::Result<TokenStream>;
   

2. A function to parse a string with a given filename. This might need an additional mechanism to register the source file as a dependency, e.g.

   fn lex_str(src: &str, filename: &Path) -> Result<TokenStream, LexErr>;
   fn add_path_dependency(path: &Path) -> ...;
   

To make it generally useful & able to implement something like include!() we'd probably also want/need something like #54725 to get the file to load relative to.

[mystor]

CC @alexcrichton @dtolnay

[alexcrichton]

Agreed this would be a good idea to have! I don't know quite the shape that it should take, but it's something we'll want eventually

[TommasoBianchi]

I think that implementing a proc_macro_include system would fit very well with the procedural macro system we have right now, and it will allow to implement macros like include!() without resorting to compiler built-in code (plus any other similar utilities). I think that the signature might look something like:

#[proc_macro_include]
fn include_file(path: &Path) -> TokenStream

Erros (for example, file not existing) would be handled by panicking in the same way as they are handled for standard proc_macro.

I'm fairly new to this repo, but as far as I have understood there should be nothing significant holding this back.

[oddcoder]

Is there any progress done on this one?
This is issue is currently roadblocking some work I am doing where I needed exactly that, So if there is anyway I can contribute to make this issue resolved I am happy to help

[TommasoBianchi]

Hi @oddcoder, sorry for the late answer but lately my work kept me fairly busy and pretty far from Rust, unfortunately. Feel fre to pick up this issue if you want to, I did never manage to reach anything useful.

[oddcoder]

@rustbot claim

[Alexendoo]

Triage: Hi, are you still working on this issue @oddcoder?

[oddcoder]

ops sorry, I got stuck at some point, then got carried away with my still ongoing exams.

[Alexendoo]

No worries, good luck with the exams. Please claim the issue again if you want to work on it in the future

@rustbot release-assignment