Crystal: RFC: Ability to expand macro (recursively)

[waterlink]

Currently the only way (I know) to debug a macro is to put debug() at the end of it. It outputs expanded macro. It does not do expansion recursively, so if the macro in question, had more macro calls in its body, they will not be expanded.

Example:

macro a
  "a"
end

macro b
  "b"
end

macro c
  a
  b
  {% debug() %}
end

c

Outputs:

  b

This is very good already for debugging. And sometimes it would be useful to have a and b expanded (and all macro they call too) recursively:

  "a"
  "b"

I have currently couple of ideas:

• change behavior of {% debug() %}
• add optional argument for debug(): debug(recursively = false)
• different debugging macro: {% debug_expand_all() %}
• crystal tool expand src/your/source/file.cr, which will apply all the compilation steps until full macro expansion and will output a stringified AST tree (so that it will look kind of like a source code, but without any macro calls inside)

If that is something that you think compiler might not need (as it is, possibly, only needed in quite rare cases), then I would like to build a shard for that, but I am afraid, I will need some clues on how I can implement such macro, that acts as debug(), but does recursive expansion, or how I can control compiler execution to the point, that I can execute only certain steps of the compiling pipeline.

WDYT?

[waterlink]

The best I could do is to do a full puts Compiler.new.compile(sources, tempfile).original_node. It outputs exactly what I want, but is there a way to only expand macro (without actual type inference and codegen)?

[waterlink]

I have built this shard for this: https://github.com/waterlink/expand.cr

[asterite]

It's kind of hard. The way the compiler works is:

• A macro is expanded. This generates a string.
• The string is parsed into an AST and inserted into the current program (in reality, the current macro call has an expanded property and whenever the semantic stage needs to solve it, it first check if it has an expansion and does semantic check to that instead).
• If the AST is a def, it's just inserted into the program. Later if the def is used, a copy of it is made and then it's typed. When that happens, when a call points to a macro, only there it's expanded.
• If the AST is not a def, well, if there's another macro call there, it will be expanded when the semantic stage reaches it.

So, it's not like macro expansion happens all at once. We might change this, of course, but the current way might make saving some macro expansions if those are inside methods that are never invoked. We could maybe add another {{debug_all()}} macro method that would turns that into a string, then analyze the result by always expanding things right there, recursively, and finally printing the result...

However, the way I see it is, if you reach the point where you need this funcionality it probably means you are using too many macros (though that doesn't mean that I'll maybe try to implement debug_all in the future :-P)

[waterlink]

Of course. I am really using a lot of macros. I would call them higher-order macros: macros that generate another macro for a user to use in some sort of block, where they are effective. Enables to do very nice DSLs. As you might expect, it is hard to debug when something goes rawry on the logic side. So everything compiles, DSL looks correct, but behavior is incorrect, that is the moment, where you want to see, what actually this DSL represents. With debug() macro, I suppose, you would have to pass around some sort of recursive_debug = true flag for all your macros, and make them call all other macros with the same argument, and conditionally do debug() at the end of their bodies. And then have some hard time understanding the output (it turns out pretty confusing, since it will output parts of macro in exact order, they are expanded, which happens lazily and it will hard to understand, where which part of expansion belongs to).

The tool I have built works really great. At least for my current needs it shows the exact code, that is being compiled after all macro expansions and transformations. Though I have to scroll through all the require expansions..

[waterlink]

Another "higher-order macro" characteristic is to store compilation-time state in some ASTs, that are not used in the actual code. That, of course, might make macro even harder to understand, but it makes them, probably, turing complete, allowing to accomplish really-really a lot of things.

[MakeNowJust]

crystal tool expand is now implemented, so I think this can be closed.

[mverzilli]

Good catch! Always on-point @MakeNowJust 🙏

[gerrywastaken]

@MakeNowJust @mverzilli are you sure?

cat test.cr | crystal tool expand
crystal tool expand test.cr
cat test.cr | crystal tool expand test

All just lead to the help text being output:

crystal tool expand [options] [programfile]

I haven't been able to get that command to do anything other than output help text.

[Blacksmoke16]

It's a bit of a pain to use but assuming a file of app.cr:

macro foo
  "foooo"
end

foo

The command would be crystal tool expand -c ./app.cr:5:1 ./app.cr

1 expansion found
expansion 1:
   foo

# expand macro 'foo' (/path/to/app.cr:5:1)
~> "foooo"

[gerrywastaken]

Ah! Thank you @Blacksmoke16! That should also help anybody else ending up here via Google.

[straight-shoota]

So the usage instructions need to be improved.