Crystal: Remove macro methods from the language

[asterite]

According to the docs:

Macro defs allow you to define a method for a class hierarchy and have that method be evaluated at the end of the type-inference phase, as a macro, for that type and subtypes.

This is how Reference#inspect(io) is implemented (simplified a bit to show the essence):

class Reference
  macro def inspect(io : IO) : Nil
    io << "#<{{@type.name.id}}:0x"
    object_id.to_s(16, io)
    {% for ivar, i in @type.instance_vars %}
      {% if i > 0 %}
        io << ","
      {% end %}
      io << " @{{ivar.id}}="
      @{{ivar.id}}.inspect io
    {% end %}
    io << ">"
    nil
  end
end

Why does it need to be a macro def? Because:

1. We want this method to be "copied" in every subclass, where @type.instance_vars can be different.
2. At least before 0.16.0, the final type of an instance var was only known at the end of the type inference phase.

But 2 no longer applies, because now the type of instance variables is known even before the "main" type inference begins.

For point 1, we could simply have the method be marked as a "macro" method, at parse time, if a @type is found inside macro code. That is, if a method uses @type inside a macro it's probably because the method will want type information for the most specific type in the hierarchy.

I just checked all macro def instances in Crystal's standard library and they all use @type.

We can greatly simplify the language with this change. For example, the above could be re-written like this:

class Reference
  def inspect(io : IO)
    io << "#<{{@type.name.id}}:0x"
    object_id.to_s(16, io)
    {% for ivar, i in @type.instance_vars %}
      {% if i > 0 %}
        io << ","
      {% end %}
      io << " @{{ivar.id}}="
      @{{ivar.id}}.inspect io
    {% end %}
    io << ">"
    nil
  end
end

Benefits:

• No new concept (macro def) has to be learned. The above is just a method, it works, and the compiler is smart and knows that the method needs to be instantiated differently for each subclass.
• No need to specify a return type, because now the compiler will be able to infer this (all needed types are now known)

I'm planning to move forward with this change, but I'd like to know if any of you is using macro defs in your projects, and if so, are they all using @type inside a macro? I checked mocks.cr, spec2.cr and artanis, which heavily use macros, and didn't find macro defs.

Hmm... I also checked frost and here I see some macro defs that don't mention @type ( /cc @ysbaddaden ). But I think in this case we can let the methods be defined in the included macro hook. As an alternative, we could introduce a @[MacroMethod] annotation to let the compiler know that the method needs to be resolved differently for each subtype (I don't like the macro def syntax).

[ysbaddaden]

I like being capable to say that a method is dependent to the class, and not inherited. It's not used often, but can be powerful sometimes.

I prefer this to be explicit thought, and not implicit by detecting that the method includes a macro that references @type. I think it's more readable.

I don't care about the syntax; having a decorator would be better, indeed. It says "please treat this method as special" and it differentiates from macros. I'm just not sure MacroMethod is very explicit, but I'm at a loss to imagine something :-/

[ozra]

I vote for explicit way, having different mechanisms based on code in body feels a bit surprising, and these constructs aren't that common.
Perhaps attribute mirroring included macro hook? @[ProcessWhenIncluded] or such, (but better / more specifically worded).

[asterite]

Thank you for the feedback!

One more thing: did you know that the body of a macro def is a macro? I mean, it's parsed as a macro. I'd like to change that, it makes things unnecessary complex, we can always use {% begin %} ... {% end }% inside the method definition. For example this compiles right now:

class Foo
  # This is a valid macro, it will of course fail to parse once we invoke it,
  # but it's better if we give a syntax error before that moment
  macro def foo : Int32
    a +
  end
end

So, this can be changed without a problem. Once we do that, the name macro def stops having sense, because it's just a regular def that is "instantiated" for every subtype. We could drop this feature and use macro inherited, except that this won't work because you might have types that already inherit, say, Reference, but you add the inherited hook later.

Another thing is that a macro def is analyzed at the end of the type inference phase... I don't think this is needed anymore, because when methods are analyzed all the type hierarchy and types of instance vars are already defined, so doing it at the end or at the beginning should have the same effect.

Maybe we should find an attribute name. Another possibility is to stick with the idea of having the compiler recognize this if it finds @type inside macro code inside a method. If this functionality is desired, but we don't need to use @type, we can always do:

class Foo
  def magic
    {% @type %} # just so the compiler treats this method in a different way, has no effect otherwise
    # code...
  end
end

But, yeah, this last bit feels more hacky. My thought was that this behaviour is usually (always) needed when you access @type. At least on macros in the std behave this way, and 99% of the others in other shards.

In conclusion, for now I'll change two things:

1. A macro def's body will be parsed like a regular method instead of a macro
2. The macro def's return type won't be mandatory

After that we can continue discussing the other bits :-)

[ozra]

I would prefer the mentioned explicit annotation, _and_, that if a def contains a reference to @type in macro-code, it errors until the annotation is added. This makes sure the code will work as intended. And it makes sure it's obvious to the reader.

[asterite]

So, points 1 are 2 are done now. This is a breaking change because I found some macro defs that used the feature that their body was parsed as a macro (I found a couple of them in minitest, @ysbaddaden). The fix is really simple, though: just put the method's body between {% begin %} ... {% end %}.

With this change we can now turn any object into a hash... this was impossible before because we needed a precise return type:

class Reference
  macro def to_h # previously we needed a return type here...
    {% if @type.instance_vars.empty? %}
      {} of Symbol => Nil
    {% else %}
      {
        {% for ivar in @type.instance_vars %}
          {{ivar.id}}: @{{ivar.id}},
        {% end %}
      }
    {% end %}
  end
end

class Foo
  def initialize
    @x = 1
    @y = "hello"
  end
end

hash = Foo.new.to_h
p hash # => {:x => 1, :y => "hello"}
p typeof(hash) # => Hash(Symbol, Int32 | String)

Not sure what's that useful for, though. But once we introduce named tuples, we'll be able to convert any object to them, and there it makes more sense (you can basically treat an object as a "hash" of properties, but a very fast and efficient hash). This can be useful to automatically dump a type to json without defining an explicit mapping.

Another nice thing is that now macro defs work with yield, which previously didn't, so now we can generically yield a type's instance vars, types and values.

All of this, of course, wouldn't have been possible without "explicit" type annotations in instance vars... so now we can start compensating that small annoyance :-)

[ysbaddaden]

Wouldn't that help for marshaling Crystal objects, too?

[jhass]

I prefer this to be explicit thought, and not implicit by detecting that the method includes a macro that references @type. I think it's more readable.

I think the example in #2647 shows that doing it implicitly feels more natural and intuitive. Therefore I vote for removing explicit macro defs from the language.

[asterite]

I made it so that {{@type}} inside a method turns it into a macro def for the compiler.

The syntax macro def is still available if you want to make this explicit, though if we don't find cases of "macro def"s where {{@type}} isn't used I'd say we remove that syntax. For example, some previous changed made macro defs resolve types relative to the place where they are defined (#2616). So if one needs to resolve a type relative to the more specific type one would have to do {{@type}}::SubType, so there the compiler too will mark the method as "macro def". With that, I'm not sure other cases where {{@type}} isn't used exist.

[asterite]

@ysbaddaden Yes, I believe that now that macro defs no longer require a return type they can be very poweful. For example one can turn an object into a named tuple and then access it almost as an (immutable) hash: https://play.crystal-lang.org/#/r/znn :-)

(don't know if that's useful, though, but it's definitely possible)