Elixir: Missing documentation of AST metadata entries

[mguimas]

Environment

$ elixir --version
Erlang/OTP 21 [erts-10.2.3] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]

Elixir 1.8.1 (compiled with Erlang/OTP 21)

Current behavior

The function Macro.update_meta/2 is a public API, and documented as

Applies the given function to the node metadata if it contains one.

This is often useful when used with Macro.prewalk/2 to remove information like lines and hygienic counters from the expression for either storage or comparison.

however this function cannot be used properly if the developer does not know what are the possible metadata entries in use by Elixir.

Expected behavior

The documentation should be enriched with information about all the metadata entries in use by Elixir, what are their purpose, and when they are inserted by Elixir in the AST (e.g., :line is not always inserted).

[josevalim]

The documentation should be enriched with information about all the metadata entries in use by Elixir

One thing to note is that the metadata is a general storage, so you can have all kinds of information here. Some of it may be completely intrinsic to the compiler and change in future versions, which we should also document.

Also, :line should always be available, otherwise it is a bug.

[mljrg]

I tested the following code in a file (with Elixir 1.8.1),

q = quote do
def foo, do: 1/0
def bar, do: 2/0
end
Module.create(Y, q, Map.put(__ENV__, :file, "y.ex"))

then when calling Y.foo() and Y.bar() the lines in the stracktraces were the same.

Interactive Elixir (1.8.1) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Y.bar()
** (ArithmeticError) bad argument in arithmetic expression
    (things) y.ex:5: Y.bar/0
iex(1)> Y.foo()
** (ArithmeticError) bad argument in arithmetic expression
    (things) y.ex:5: Y.foo/0
iex(1)>

I think this is because quote does not add metadata line: number. Only the quotes passed to macros get the line: number metadata.

I think quote should then add line numbers always.

Is this a bug in quote?

[josevalim]

Oh, I see what you mean by missing line then. So quote does not add line because many (most?) times you want to keep the line number of where the code is injected and not from where the code is generated. If you want to keep it in your case, then you need to pass the line: :keep option. But then, this is not really documentation about AST, it is documentation about how quote behaves.

[mljrg]

Well you mean location: keep, in which case we get for the above example

quote: {:__block__, [keep: {"lib/x.ex", 0}],
 [
   {:def, [keep: {"lib/x.ex", 4}, context: Elixir, import: Kernel],
    [
      {:foo, [keep: {"lib/x.ex", 4}, context: Elixir], Elixir},
      [
        do: {:/, [keep: {"lib/x.ex", 4}, context: Elixir, import: Kernel],
         [1, 0]}
      ]
    ]},
   {:def, [keep: {"lib/x.ex", 5}, context: Elixir, import: Kernel],
    [
      {:bar, [keep: {"lib/x.ex", 5}, context: Elixir], Elixir},
      [
        do: {:/, [keep: {"lib/x.ex", 5}, context: Elixir, import: Kernel],
         [2, 0]}
      ]
    ]}
 ]}

in which case the lines in the stacktraces of Y.foo() and Y.bar() are correct now.

Yet, the lines come from :keep entries, not from :line entries. Is this behaviour expected or a bug as per what you said

Also, :line should always be available, otherwise it is a bug.

?

In addition, the quotes macros receive have line numbers given by :line, and perhaps it would be better for consistency to have line numbers in both situations given by :line?

Obviously, this :line case is orthogonal to this open issue: :line is just one of several entries in the AST metadata, for which this issue was opened to request all those Elixir metadata entries to be documented.

Note that if the : line case is a bug, then perhaps another issue should be opened.

[josevalim]

Also, :line should always be available, otherwise it is a bug.

When I said this, I was thinking about the AST generated by the parser. But since anything can generate the AST by hand, then the rules may not apply to it. For instance, I can simply do this: {:foo, [], nil}. And quote is another example that emits a slightly different metadata.

In addition, the quotes macros receive have line numbers given by :line, and perhaps it would be better for consistency to have line numbers in both situations given by :line?

Right, the reason why we need to have both is because the macro system needs to know what comes from the parser and what comes from quote to be able to merge them both accordingly when macros are expanded.

So there isn't another issue here but this discussion has been helpful to the issue. :+1:

[mguimas]

This is the first time I have seen the :import_fa metadata key:

iex(2)> quote do: {"string", &inspect/1}
{"string",
 {:&, [import_fa: {Kernel, Elixir}],
  [{:/, [context: Elixir, import: Kernel], [{:inspect, [], Elixir}, 1]}]}}

Could this key have a more clear name?

Anyway its purpose should be documented along with the other metadata keys injected by Elixir.

[josevalim]

As I mentioned, some Metadata are private to the compiler behavior, so we
don’t document them because we don’t want users to rely on it in case we
have to change them. Although we will add a note that there is private

metadata.

José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

[maennchen]

Is there an existing place to put documentation for AST or would this be documented next to Macro.update_meta/2?

[josevalim]

I am not sure about the best place to put this either. Options?

[josevalim]

Oops, premature send. Macro.update_meta/2 is definitely an option. The issue is that different places use different metadata. So we could also scatter it around each place, like quote and Macro, but that can be confusing. Perhaps unifying everything in the Macro module doc is best?

[mguimas]

I think all documentation for the AST metadata should be centralized somewhere for easier reference, and links to this documentation should be added here and there as convenient.

A possibility is to have an entire page dedicated to documenting the AST, such as a new page under "PAGES". This page could have this documentation, plus perhaps other documentation that is under quote/2. In fact, quote/2 seems to have too much documentation for a function, and when this is the case, perhaps it is time to add a new topic under "PAGES".

Then there could be links to this new page from quote/2, from update_meta/2, and perhaps from other places as needed for easy access.

[josevalim]

So let's document it at the Macro module body for now. I don't think the amount of content here is quite short. We can always upgrade to a page later.

[maennchen]

I tried to find all currently known metadata options (I don't know if this list is complete):

• ambiguous_op
• alias
• alignment
• context
• counter
• defined
• file
• generated
• import
• import_fa
• keep
• line
• origin
• required
• super
• var

I guess the idea would be to describe the general structure of the AST (3-element-tuples containing an identifier, metadata, and contents) and describe these metadata options as well. Is that correct?

Edit: Found that we already document the structure of expressions etc. here.

[OvermindDL1]

Well (... -> any()) is an interesting option to see appear in the literal type.. I've never seen that in the AST before, wonder how it works and why it can return any() instead of more t() or so...

EDIT: Hmmm, I think the typespec is lying...

iex(6)> {{:., [], [{:__aliases__, [alias: false], [:IO]}, :inspect]}, [], [fn -> :vwoop end]}                     
{{:., [], [{:__aliases__, [alias: false], [:IO]}, :inspect]}, [],
 [#Function<20.128620087/0 in :erl_eval.expr/5>]}
iex(7)> {{:., [], [{:__aliases__, [alias: false], [:IO]}, :inspect]}, [], [fn -> :vwoop end]} |> Macro.to_string()
"IO.inspect(#Function<20.128620087/0 in :erl_eval.expr/5>)"
iex(8)> {{:., [], [{:__aliases__, [alias: false], [:IO]}, :inspect]}, [], [fn -> :vwoop end]} |> Code.eval_quoted()
** (CompileError) nofile: invalid quoted expression: #Function<20.128620087/0 in :erl_eval.expr/5>

Please make sure your quoted expressions are made of valid AST nodes. If you would like to introduce a value into the AST, such as a four-element tuple or a map, make sure to call Macro.escape/1 before
    (elixir) lib/code.ex:590: Code.eval_quoted/3
iex(8)> {{:., [], [{:__aliases__, [alias: false], [:IO]}, :inspect]}, [], [fn x -> x end]} |> Code.eval_quoted()     
** (CompileError) nofile: invalid quoted expression: #Function<6.128620087/1 in :erl_eval.expr/5>

Please make sure your quoted expressions are made of valid AST nodes. If you would like to introduce a value into the AST, such as a four-element tuple or a map, make sure to call Macro.escape/1 before
    (elixir) lib/code.ex:590: Code.eval_quoted/3
iex(8)> {{:., [], [{:__aliases__, [alias: false], [:IO]}, :inspect]}, [], [fn x, y -> {x, y} end]} |> Code.eval_quoted()
** (CompileError) nofile: invalid quoted expression: #Function<12.128620087/2 in :erl_eval.expr/5>

Please make sure your quoted expressions are made of valid AST nodes. If you would like to introduce a value into the AST, such as a four-element tuple or a map, make sure to call Macro.escape/1 before
    (elixir) lib/code.ex:590: Code.eval_quoted/3
iex(8)> {{:., [], [{:__aliases__, [alias: false], [:IO]}, :inspect]}, [], [fn x, y, z -> {x, y, z} end]} |> Code.eval_quoted()
** (CompileError) nofile: invalid quoted expression: #Function<18.128620087/3 in :erl_eval.expr/5>

Please make sure your quoted expressions are made of valid AST nodes. If you would like to introduce a value into the AST, such as a four-element tuple or a map, make sure to call Macro.escape/1 before
    (elixir) lib/code.ex:590: Code.eval_quoted/3

[michalmuskala]

You can have &Module.function/arity funs as literals in the AST, you can't have closures - unfortunately types in dialyzer are too weak to capture that difference.

iex(2)> Code.eval_quoted(&String.split/2)
{&String.split/2, []}

[OvermindDL1]

You can have &Module.function/arity funs as literals in the AST, you can't have closures - unfortunately types in dialyzer are too weak to capture that difference.

Ahh, that makes sense. Really need documentation on those types.

[josevalim]

Right, there is a distinction though: you will never receive &Remote.fun/arity in a macro or when quoted, but if you have a macro that returns it, then Elixir knows how to "serialize" it.

[josevalim]

@maennchen thanks for the fantastic work on lifting those up. I have pushed a Macro.metadata type to the source, I believe we can document the keys in there. Here is my proposal:

@typedoc """
A keyword list of AST metadata.

The metadata in Elixir AST is a keyword list of values. Any key can be used
and different parts of the compiler may use different keys. For example,
the AST received by a macro will always include the `:line` annotation,
while the AST emitted by a quote never has the `:line` annotation.

The following metadata is public:

  * `:column` - The column number of the AST node. It is included only
    in AST generated by `Code.string_to_quoted/2` with the `columns: true`
    option
  * `:context` - Defines the context in which the AST was generated.
    For example, `quote/2` will include the module calling `quote/2`
    as the context. This often used to distinguish code from code generated
    by a macro/quote
  * `:counter` - The variable counter used for variable hygiene. In terms of
    AST, each variable is identified by a `{name, metadata[:counter] || context}`
    tuple
  * `:generated` - If the code should be considered as generated by
    the compiler. This means the compiler and tools like dialyzer may not
    emit certain warnings
  * `:keep` - Used by `quote/2` with the `location: :keep` option to annotate
    the file and line of the quote source
  * `:line` - The line number of the AST node

The following metadata is private. Do not rely on them as they may change
or be fully removed in future versions. They are often used by `quote` and
the compiler to provide features like hygiene, better error messages, etc:

  * `:alias` - Used for alias hygiene
  * `:ambiguous_op` - Used for improved error messages in the compiler
  * `:import` - Used for import hygiene
  * `:var` - Used for improved error messages on undefined vars

"""

Feel free to send a PR with the above with any change you may find relevant.

I did not include the following AST because they are used by the compiler and it never surfaces to users:

• :alignment - keeps the alignment in bitstrings
• :defined - used by defmodule to change the macro env
• :origin - to annotate implicit try for error messages
• :import_fa - has been removed
• :super - used to annotate super calls between Elixir passes

[maennchen]

@josevalim When we write „any key can be used“, we should mention that the compiler internal keys should not be used to prevent conflicts. Do you agree?

[maennchen]

PS: I‘m going to create a PR tomorrow if no one else is already on it.

[josevalim]

When we write „any key can be used“, we should mention that the compiler
internal keys should not be used to prevent conflicts. Do you agree?

Agreed.

José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

[maennchen]

There's still one point unclear to me: If we introduce new keys in the compiler in the future, how do we prevent a breaking change since the user could've used the same keys and now runs into conflicts?

[josevalim]

Yes, that’s always a possibility but I would say it is extremely uncommon
for you to add new keys because there just isn’t a use case. So maybe we
should either say it is extremely uncommon to add new keys or just say they

are read only?

José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

[josevalim]

We can also say: if you plan to add your own keys, make sure they are
properly namespaces with the name of your library or project, such as:

:my_lib_meta.

José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

[josevalim]

Closing in favor of PR.