Elixir: Provide an API to access documentation metadata at compile time

[hauleth]

Environment

• Elixir & Erlang/OTP versions (elixir --version): 1.7.2
• Operating system: macOS

Current behavior

I wanted to create macro that would allow me to remove boilerplate over generating structures for events in my application. To document such structures I wanted to use @doc to simulate feeling that these are normal parts of the external module instead of being separate modules, ex.:

defmodule MyApp.Event do
  import Events

  @moduledoc false

  @doc """
  Creation of new submission
  """
  @doc deprecated: "Foo"
  defevent SubmissionCreated,
    id: any(),
    name: String.t(),
    author: [map()]
end

While basic implementation is dumb easy documentation is quite challenging. What I have achieved is:

defmodule Events do
  @moduledoc false

  defmacro defevent(module, fields \\ []) do
    keys = Keyword.keys(fields)

    quote do
      docs = Module.get_attribute(__MODULE__, :doc)
      deprecated = case Module.get_attribute(__MODULE__, :deprecated) do
        nil -> []
        value -> [deprecated: value]
      end

      {set, _} = :elixir_module.data_tables(__MODULE__)
      metadata = case :ets.lookup(set, {:doc, :meta}) do
        [{{:doc, :meta}, metadata, _}] -> deprecated ++ Keyword.new(metadata)
        [] -> deprecated
      end

      defmodule unquote(module) do
        if docs, do: Module.put_attribute(__MODULE__, :moduledoc, docs)
        if metadata != [], do: @moduledoc metadata

        @type t :: %__MODULE__{unquote_splicing(fields)}

        defstruct unquote(keys)
      end

      :elixir_module.delete_definition_attributes(__ENV__, nil, nil, nil, nil, nil)
    end
  end
end

Which is fairly ok, except of the part where I need manually get data from ETS for documentation metadata. It cannot be mitigated by using Module.get_attribute/2 as it explicitly requires atom as a second argument while metadata are stored under {:doc, :meta}.

Expected behavior

Somehow allow fetching documentation metadata in macros. This would allow macro writers to utilise that metadata in some way like example above.

[fertapric]

I would add "at compile time" to the title, because once compiled I think you can access the documentation using Code.fetch_docs/1 :)

[josevalim]

The documentation metadata and related fields is stored as private fields on the table, that's why you can't access it and you shouldn't rely on the compiler internals for this. :)

The other thing is that, even if we provide a way to access metadata, then you would also need a way to remove the documentation because there is no actual function definition. It feels like the best way to go here is to use your own attribute, such as @eventdoc, especially because you won't pretend that you are defining a function when it is actually a module.

[josevalim]

The proper solution here would be to make @doc and friends an accumulated attribute, so accessing @doc would return all previously set values:

@doc "foo"
@doc bar: 1
@doc #=> [{2, bar: 1}, {1, "foo"}]

This would also allow us to get rid of special cases in both Module and Kernel. The trouble with doing this now is that it will break code that expects @doc to always be a binary (or similar). So we can only do it on Elixir 2.0.

[hauleth]

@josevalim I agree that in this particular situation it makes sense to use custom attribute instead, but imagine something like that:

defmacro defrecord(name, tag \\ nil, kv) do
  quote do
    def unquote(name)(), do: …
    def unquote(name)(values), do: …
    def unquote(name)(record, values), do: …
  end
end

And while you maybe do not want to share most docs between them then currently supported tags makes sense for them (like :since or :deprecated).

[mguimas]

@josevalim Instead of this

@doc "foo"
@doc bar: 1
@doc #=> [{2, bar: 1}, {1, "foo"}]

could you consider this

@doc "foo"
@doc bar: 1
@doc #=> [doc: "foo", bar: 1]

?

[josevalim]

@mguimas unfortunately a metadata can be named :doc, so that would be ambiguous, we will probably use a tuple-based format, such as {"doc", meta: "data"}, or a map: %{doc: "...", metadata: [...]}.

[hauleth]

@josevalim I would vote for the tuple + keyword list so:

@doc "foo:
@doc bar: 1
@doc bar: 2

Would result with:

{"foo", [bar: 1, bar: 2]}

[mguimas]

@josevalim I believe that a good data structure is one that makes it is easy to use Enum to traverse the information associated with the @doc entries.

So in the example

@doc :foo
@doc bar: 1
@doc bar: 2
@doc :bar

perhaps a good solution is

[:foo, {:bar, 1}, {:bar, 2}, :bar]

which respects not only the contents, but also the order of the doc entries.