Gleam: Generate erlang typespecs for use with dialyzer

[jamesmacaulay]

I'd like to propose a modification to the compiler to generate Erlang typespecs for any types defined within Gleam. I'm using Gleam from within an Elixir project and I've come to really appreciate having Dialyzer integration within my editor for my Elixir code. Having generated typespecs would allow that experience to extend automatically to code that calls Gleam functions, and would also allow re-exporting types from Elixir or Erlang modules that "wrap" Gleam modules.

For example, the following Gleam code:

pub type Tree(value) {
  Leaf(value)
  Node(Tree(value), Tree(value))
};

pub fn any(tree: Tree(a), check: fn(a) -> Bool) -> Bool {
  case tree {
    Leaf(i) -> check(i)
    Node(left, right) -> any(left, check) || any(right, check)
  }
}

pub fn has_even_leaf(tree: Tree(Int)) -> Bool {
  any(tree, fn(i) {
    i % 2 == 0
  })
}

...could generate the following types and function specifications along with the function implementations (unless I've got these wrong—let me know):

-type tree(Value) :: {leaf, Value} | {node, tree(Value), tree(Value)}.

-spec any(Tree :: tree(A), Check :: fun((A) -> boolean())) -> boolean().

-spec has_even_leaf(Tree :: tree(integer()) -> boolean().

[lpil]

A great suggestion! Let's definitely do this one

[michaeljones]

I'd be happy to try to take a look at this. @lpil would you mind giving an overview or sharing your thoughts on what bits of the code base are involved and anything else you might normally share in such a situation :) Not rush though. I've got a busy week.

[lpil]

I would possibly make a new module called src/erl/typespec.rs and start with a function that can turn a Type into a pretty::Document representing the typespec in a similar way to how the erl module does.

One thing to keep in mind is that once this is added to the Erlang code generation all of the integration tests (that's most of them) will break because the format of the generated Erlang will have changed. Perhaps there is an argument in favour of making typespec generation optional in order to make testing easier.

[michaeljones]

I've been playing around with this but I lack the familiarity with Erlang & typespecs to be confident with it. I would appreciate a bit of help with a few things:

• Gleam types are capitalised but it seems that typespec types shouldn't be? If there a standard helper for converting? Or do I just lower case? Or maybe snake case the type? Or might that end up clashing somehow with other contents in the file?
• Do records need to be extract and given their own -record entry and then that is used in the -type spec?
• I assume custom types can otherwise have their entries combined with |?
• And the entries are generally represented as {name_as_atom, SomeKindOfValue}? So Option would be:
  -type option(Value) :: {some, Value} | none

No rush. I'm not sure how much time I'll have for it this weekend. I am enjoying exploring it a bit though.

[lpil]

When rendering Erlang we go from CamelCase to snake_case, there's a helper function from a library that we use in the erl module.

Do records need to be extract and given their own -record entry and then that is used in the -type spec?
I assume custom types can otherwise have their entries combined with |?
And the entries are generally represented as {name_as_atom, SomeKindOfValue}? So Option would be:

Yes that's it. Something like this:

-export_type([option/1]).
-type option(Value) :: {some, Value} | none.

Though we do also generate record headers, so we'd need to check if they are compatible when using with this typespec. We can't use a record definition in place of the type definition because records cannot have type parameters, unless I am mistaken.

I don't know much about Dialyzer as I've never found it to bring enough value to put any additional time into.

[michaeljones]

A first attempt is here #881 - It is incomplete though and might be a dead end. Needs assessment.

[lpil]

For anyone looking to pick up this work, Michael's PR (https://github.com/gleam-lang/gleam/pull/881) would make a good foundation.

[lpil]

I've just run the new Gleam typespec generation on the stdlib and then run rebar3 dialyzer and it had surfaced some errors.

I really dislike dialyzer's error messages.

Error 1

gen/src/[email protected]
  31: The pattern {'error', 'nil'} can never match the type {'ok',binary()}

The code for this one:

-spec string(gleam@dynamic:dynamic()) -> {ok, unicode:unicode_binary()} |
    {error, unicode:unicode_binary()}.
string(From) ->
    gleam@result:then(
        gleam_stdlib:decode_bit_string(From),
        fun(Raw) -> case gleam@bit_string:to_string(Raw) of
                {ok, String} ->
                    {ok, String};

                {error, nil} -> % this is the line of the error
                    {error, <<"Expected a string, got a bit_string"/utf8>>}
            end end
    ).

I don't understand this error at all. gleam@bit_string:to_string returns {ok, unicode:unicode_binary()} | {error, nil()}

Error 2

gen/src/[email protected]
  11: Invalid type specification for function gleam@io:print/1. The success typing is (atom() | binary() | string()) -> 'nil'

-spec print(unicode:unicode_binary()) -> nil().
print(String) ->
    io:fwrite(String, []),
    nil.



md5-d73e50708c3f64ac6f3c249580e644a7



gen/src/[email protected]
  16: Invalid type specification for function gleam@io:println/1. The success typing is (_) -> 'nil'



md5-b095d31a8011630c3cb2953da15fabda



Same as above. I don't quite understand the `_` argument. What does that mean? `any()`?

# Error 4



md5-abc8760a2bb26e2f1e027c7d32f0afcb



```erlang
-spec each(list(AGH), fun((AGH) -> any())) -> nil().
each(List, F) ->
    case List of
        [] ->
            nil;

        [X | Xs] ->
            F(X),
            each(Xs, F)
    end.



md5-eca82283a71f10485fe563cdbccfa6eb



gen/src/[email protected]
  26: Invalid type specification for function gleam@os:insert_env/2. The success typing is (binary(),binary()) -> 'nil'



md5-f5d97302ffeb2ff76efb09eccdeb1610



Can OS environment variables really have keys and values of random binary data without any encoding?
Either way how do we fix this? Wrap the function in an Erlang function of our design and try and restrict it to unicode data in some way I guess. Not sure how exactly.

# Error 6



md5-ceedc692c6a8ddc1b2e5e3f8c5fbdb44



```erlang
-spec delete_env(unicode:unicode_binary()) -> nil().
delete_env(Key) ->
    os:unsetenv(erlang:binary_to_list(Key)),
    nil.



md5-00efec05a9103b7cd53373e88bee0c20



gen/src/[email protected]
 148: Invalid type specification for function gleam@string:pad_left/3. The success typing is (binary() | maybe_improper_list(binary() | maybe_improper_list(any(),binary() | []) | char(),binary() | []),integer(),string() | char()) -> binary() | {'error',binary(),binary() | maybe_improper_list(binary() | maybe_improper_list(any(),binary() | []) | char(),binary() | [])} | {'incomplete',binary(),binary()}



md5-e6d4bea9b7cf5a36416965fceac39921



A quality dialyzer error message.
I'm not sure what this is saying but I guess it's the same thing of our type definition being stricter than the Erlang would accept.

# Error 8



md5-3570f4b89e02e6968c0f0ee601112bd6



```erlang
-spec pad_right(unicode:unicode_binary(), integer(), unicode:unicode_binary()) -> unicode:unicode_binary().
pad_right(String, Length, Pad_string) ->
    gleam_stdlib:string_pad(String, Length, trailing, Pad_string).

Same as above

Thoughts

I'm frustrated that we have these nonsense errors. I'm worried that in adding this we may have made it much harder to use the Erlang interop by merging this. Or do these errors not matter? How big a problem is it if your deps have dialyzer errors? I am not a dialyzer user.

Pinging @itsgreggreg @QuinnWilton @CrowdHailer and any other peeps who might know things about Erlang/Elixir types

[itsgreggreg]

Yeah I may have been fundamentally misunderstanding something about the dialyzer here, which is no surprise considering every time i try to _use_ the dialyzer I leave with the same feeling. My 2¢ is the only reason to add specs would be to get the them working with the dialyzer.

Some low hanging fruit that I got wrong:
nil is the atom nil
nil() is []

that sholud fix a couple of these.

You can take the parens off here https://github.com/gleam-lang/gleam/blob/main/src/erl.rs#L1742 and re-run the dialyzer and see how much that helps.

[lpil]

Well spotted there Greg. I will try those!

It looks like Elixir uses binary() instead of unicode:unicode_binary(). This seems wrong to me but perhaps it helps. I will try that.

https://github.com/elixir-lang/elixir/blob/ae5188d8996f9ab44358be3d7acd4447a2de0677/lib/elixir/lib/string.ex#L271-L278

Quinn has also found some cofig we could tweak: https://github.com/erlang/rebar3/blob/master/rebar.config.sample#L102

[lpil]

Excellent work @itsgreggreg ! With those two changes it's just down to those last two ugly ones. I will investigate further.

[itsgreggreg]

Here's me trying to format the error for Error 8

       ( binary()                                  % Argument 1
          | maybe_improper_list( binary() 
                                 | maybe_improper_list(any(), binary()
                                                                | []
                                                      ) 
                                 | char()
                             , binary() 
                                 | []
                             )
        , integer()                               % Argument 2
        , string()                                % Argument 3
            | char()
        ) 

        -> 
        binary()                                   % Return
          | {'error'
            , binary()
            , binary() 
                 | maybe_improper_list(binary() 
                 | maybe_improper_list(any(),binary() | []) 
                 | char()
            , binary() | [])
            } 
          | {'incomplete', binary(), binary()}

Which _looks like_ where gleam is saying binary, erlang is saying something like iodata, or something like that. I'm not at a good place to dig deeper at the moment, but that is my hunch.

[lpil]

I found the problem. Dialyzer seems to think that the string to use for padding _cannot_ be a binary. This is strange as we are passing one already. Seems there's a small type bug somewhere in OTP here.

I've have the Erlang convert the binary into a list. It's slightly wasteful but it type checks now!

Phew, ok, that wasn't as bad as I feared. I've bad memories of being stuck with dialyzer errors for days

[lpil]

https://github.com/gleam-lang/stdlib/commit/fb54aac688dbad5650562224d9a505d5d24770a8

Here's the patch 🎉

[itsgreggreg]

radical, great work!

[lpil]

You did all the hard stuff! I cannot take any credit.