Reason: Proposal: es modules-style top-level `import` and `export` statements

[jaredly]

Background

One of the things I really like about the javascript ecosystem is the clarity & ease of module management. Looking at a file, you can tell exactly which functions form the public API, and which things are imported from elsewhere. Additionally, the use of paths (in contrast to, say, python) remove much ambiguity and make it relatively simple to find where a module is that is being referenced.

Proposal

Add import and export declarations to reason syntax, making both inter-file dependencies and the file's public API explicit. .rei files would be unnecessary, as we would then generate .mli files from the export declarations. (Optional: export declarations would be required to be fully type-annotated)

Example

With this proposal, a .re file might look like:

import MyUtils from "./utils.re";

export let my_public_value: Int = 10;

etc.
This removes the need for .rei files: .mli files can be automatically generated from export declarations in the .re files.
It might be useful to require export declarations to be fully type-annotated, in the name of readability & helping developers to avoid making breaking changes to the public API of a module (although it's technically unnecessary, as we could just infer the type and use that in the .mli declaration).

The reason docs say that .rei files help with incremental compilation -- would this change hurt incremental compilation in some way? I expect that parsing a file is fast, and if we require type declarations on exports then we wouldn't have to do any inference either, meaning performance shouldn't be hit too bad.

Specifics

import {util_fn} from "./utils.re" => let util_fn = Utils.util_fn
import MyUtils from "./utils.re" => module MyUtils = Utils

We'd also have to block Ocaml's default behavior & raise an error when accessing a module that you haven't imported.

Unanswered questions:

Ocaml's default is import *, which makes me sad. I'd love to not support it.
Are there actually good arguments for using it? (other than "typing less is more important than readability")

[yunxing]

Thanks for the neat proposal. I think making import and export more explicit is the right direction to go.

One thing I don't quite like about the relative import path is sometimes people may abuse it and write "../../../../../../a/b/c/d/e.re", which is super hard to maintain. An alternative here is Go's import style, where everything is relative to the project's root folder or source folder.

import * also makes me sad I'm all for removing support for it. But in the meanwhile we still need to provide backward compatibility to existing ocaml codebase so that they can be converted to Reason.

[jaredly]

I agree about ../../../../../a/b/c/d/e.re, but I'd rather not lose relative path references.
npm has the concept of local paths, such that you define in package.json:

{
  "dependencies": {
    ...
    "mymod": "./path/to/mymod",
    ...
  }
}

And then you can require("mymod") (or require("mymod/subfile")) from anywhere in the project without messing with relative path munging.

I like that this enables wider sharing but encourages locality (as compared to haste, which doesn't encourage locality and requires heavy editor support to know where any module is located)

[SanderSpies]

@jaredly Do you want to take a go at this?

[bobzhang]

this actually makes the build system easier, but it is quite invasive change

[texastoland]

but it is quite invasive change

Invasive in regards to changing OCaml semantics?

[bobzhang]

it's still syntax changes, converting ocaml back to reason is not easy, since you have nested modules

[IwanKaramazow]

Ever since programming in Reason, I found js style import & export more & more annoying. 😄

making both inter-file dependencies and the file's public API explicit

Every file is a module, doesn't restricting a file with an interface solve this problem?

[jaredly]

Dependencies: There's no way to look at a file and see "what other modules does this depend on?" (except for the ever-popular include SomeMod which makes matters 100x worse). You just have to hunt through and see if you see any capitalized words that aren't defined elsewhere in the file. Additionally, local dependencies (defined within this project, but within another file) and nonlocal dependencies (defined in another project) are treated identically. I think explicit imports reduce confusion considerably.
Public API: looking at a file, I have no idea what methods/etc are exported -- I have to jump over to the interface file for that. I'd love to have them consolidated.

[chenglou]

We can have sugar for this import:

let (a, b) = MyModule.((a, b));

(cc @yunxing, extra parens needed around that, is this expected?)

this is a much less invasive change, because you can sugar this to some import syntax without changing anything else. Good first step at least.

Being a JSer myself, I've often thought the Facebook style require('foo') where foo could be anywhere is nice. No need for tooling to refactor paths when we move files around. Also, since our upcoming build system dictates a flat src/ (at least in the meantime), it actually makes more sense not to have "./Foo" and just have Foo everywhere (also notice that the former is stringly typed).

I also hate open and open!. Makes things so much harder for static analyzer and build systems (now you need to provide much more info to indicate where a certain variable comes from, because that open might or might not contain it). I'm for fewer opens, or even none of it, in conjunction with the above import sugar (but there might also be some situations where open is mandatory in the language? Not sure).

[bobzhang]

@chenglou I fully agree, open just makes build system harder

[yunxing]

@chenglou

let (a, b) = MyModule.((a, b)); 

is expected. We can add a sugar for it.

[jordwalke]

@jaredly

This removes the need for .rei files: .mli files can be automatically generated from export declarations in the .re files.

Have you considered how bindings inside of a module (file) can be type constrained, and then the interface can chose to constrain that type in a different way (or allow that type to be abstract)? How would you export all the various forms?

Ocaml's default is import *, which makes me sad. I'd love to not support it.
Are there actually good arguments for using it? (other than "typing less is more important than readability")

There's utility in import *. There's probably a set of libraries that you consider so fundamental, access to its contents should be immediate. There's a module somewhere called Pervasives that has the integer + function defined. (You don't want to have to import (+) from Pervasives at the top of the file, so it's opened once.) If building a web app, you might want to open ReactDom at the top of the file and get access to all the <tags>. Aside from these use cases, I don't tend to do open X.

import {util_fn} from "./utils.re" => let util_fn = Utils.util_fn
import MyUtils from "./utils.re" => module MyUtils = Utils

@chenglou proposed something like:

let (utilFunc1, utilFunc2) = MyUtils.()

Which would be really simple to implement, but I'm not sure I like the syntax. But the thing I like about it, is that there needn't be a separate concept of "importing", when standard let/destructuring will suffice (unless I don't understand everything that import is supposed to do).

Regarding relative path resolution: That seems independent of whether or not .rei are autogenerated via a .re file + some export keyword. Is that perhaps another discussion topic we could open up in another thread or is it related to the .rei/export/import proposal?

Additionally, local dependencies (defined within this project, but within another file) and nonlocal dependencies (defined in another project) are treated identically.

Can you explain? Have you seen the CommonML approach to namespacing? (which @chenglou's jengaboot project reimplements). CommonML's approach is a contender for the canonical namespacing strategy which we'd provide built in support with Reason. How does it compare or overlap with an import proposal? The approach there is a compromise between "haste" style flat namespaces, and commonJS require. So inside your package, you can see every other module (based on their file name), regardless of path. Outside of your package, people have to access your modules via the MyPackage namespace.

So while my package can do MyUtils.someVal (regardless of where myUtils.re is inside my package src folder), another package that depends on my package must do MyPackage.MyUtils.someVal (again, regardless of where myUtils.re is inside my package's src folder). In a way, that would be like my JS package being able to do require('myUtils').someVal, but another package that depends on me having to do require('MyPackage/myUtils').someVal (except in the CommonML approach, it doesn't matter where myUtils.js would be located inside the MyPackage source directory)

So it's kind of like a hybrid of Haste (FB's module resolution that uses a completely flat namespace globally for each file), and commonJS which has package level scope + relative paths. It allows forming many small flat namespaces scoped by their containing package, instead of one giant one.

[yunxing]

@chenglou @jordwalke Now I like Cheng's syntax suggestion with punning more.

[jaredly]

@chenglou I'm rather strongly opposed to Haste-for-everything dependency resolution :D for reasons stated above.
Also Foo is stringly typed too, inasmuch as everything is a string. compile-time string resolution is actually not at all a problem; "stringly typed" as a criticism only makes sense if you're doing runtime dispatch off of strings that aren't compile-time checked.
And a flat src/? I would hope we don't bake that into the language / module resolution semantics, even if the build system has that limitation at the moment. Spacial organization of files into folders is super useful to me. Being able to move files around without touching the files that depend on them is cool, but imo too much of a cost -- again, looking at a file, you don't know where the dependency is likely to be.

@jordwalke I hadn't considered the need to constrain things differently internally vs externally. Is that something you've come across? And would the exported type ever more _less_ constrained than the internally used type? At any rate, I don't imagine it would be too difficult to come up with a way to specify "this is the exported type" vs "this is the internal type annotation". tbh lots of the ocaml code I've looked at (which admittedly isn't a ton) don't use many type annotations b/c everything is inferred, so I assumed exporting would be the only place annotations would be used.

That said, all of these problems (knowing where files are located, the annoyance of an extra interface file) could be solved by a SufficientlyHelpfulIDE. Is that the lens through which we'll always be looking at these files, or will we still sometimes look at them on github?
[on the other hand, the spacial reasoning point /folder organization point still stands imo]

[sgrove]

Perhaps it's too late and might add too much noise, but I'm curious as to the disadvantages of e.g. Clojure/Script's globally-qualified module names, which _require_ that the file/folder structure match the namespace structure.

For example, I have a module:

sgrove.http.client, then internally the structure of the package will look like src/http/client.clj

It's completely unambiguous, makes basic tooling very easy, and makes tooling like Codeq trivial as well (which again requires globally-qualified module names), and allows for @jaredly to layout code appropriately.

I'm confused about the advantage of relative imports, which seem similar to open or open! in that I (or my tools) have to now have lots of context about my project to understand a given module. For using local modules, that's a matter for the tooling at compile time.

Sorry again if this is too noisy, I'm genuinely interested in understanding this different way of doing things!

[jaredly]

Haha I'm actually also working on a tool for clojurescript to give me back relative imports :D
the basic idea being: locality is useful. Not only is spacial reasoning useful for organizing the whole project, but when I'm in a certain file, I am _there_; there is a location associated with it, and some files are closer than others. If a file references a lot of files that are "far" away from it, that is a code smell; similar to when one OO object is reaching deep into some.obj.attr.theThingIWant.

[bsansouci]

@sgrove The argument I've heard against using globally qualifier module names that match the folder structure is that if one library modifies their folder structure, they break everyone that depends on them. That coupling between folder structure and the code depending on the _API_ of the library is maybe unnecessary.

[sgrove]

@bsansouci That's an interesting argument - I haven't run into it personally (changing the folder structure likely means that the API will be affected as well, and hence require updates), but I can see it bothering people.

@jaredly I don't quite understand the argument of locality, especially re: code smell. But sounds like a discussion that's better had outside of this specific thread, so I'll ping you separately to understand more.

[jordwalke]

And would the exported type ever more less constrained than the internally used type?

I don't think it is possible to have an external type be _less_ constrained than the internal type. I'm asking if you've thought of all the possibilities because if borrowing some syntax or patterns from other languages, I'd like to know if those languages we're taking inspiration from have ways to represent all the things that the ML module system can. If so, we'd probably want to match for consistency.

Regarding separate compilation: I don't think an export keyword will ruin the ability to make compilation more incremental. It just puts more complexity into the build system. Right now, you can use a simple Make based system which uses mtimes to determine if an interface has been touched, its implementation has been touched, or both. If an interface has not been touched, yet its implementation has, you may not need to recompile things that depend only on the interface. This can massively speed up recompilation times. When there are not two separate files, this simple mtime based approach isn't sufficient, and we need to perform diffs on the AST directly. This is exactly the kind of thing our out-of-the-box build system could handle. (There's also many other good reasons for wanting to do AST change detection instead of mtime (changing comments or adjusting formatting shouldn't trigger a recompilation!)).

Also, I think someone should open a new discussion for relative vs. flat vs. hybrid namespacing. We don't need to be locked into one or the other, even if our initial implementation of the build system chooses one. I'd like to hear arguments in favor/against, but I'd like to focus on what it means to import/export first. (am I correct in saying that it is largely independent of the import/export discussion)?

[jaredly]

👍 good call, relative/absolute is a separate issue

[jaredly]

@jordwalke

But the thing I like about it, is that there needn't be a separate concept of "importing", when standard let/destructuring will suffice (unless I don't understand everything that import is supposed to do).

One of the biggest benefits I see from having a special syntax for imports is readability -- making it obvious what the external dependencies are that a given file has. This is my main problem with Ocaml's "everything is just automatically visible" system.

[jordwalke]

So import draws a distinction between external dependencies and internal ones? What about local modules?

Why would we want to be able to import something externally but not internally like in this example:

let module M = {
   let x = ...
};

import x from M;

I agree we'd benefit from a better syntax for copying a whitelist of identifiers into scope, from another module. But I'm just not clear on what the goals of an import feature are and how, in your experience, your developer experience has been improved as a result of import. Yes, import * isn't very helpful and I don't suggest using Reason's open for that purpose in most cases. So if we have a way to only open a whitelist of identifiers such as:

import (x, y) from SomeModule

then it seems your primary concern about open * is addressed. What else is solved by this, and what is not solved by this? Do the answers to those questions change if the syntax for "importing" uses the keyword `let?

let (x, y) = SomeModule;

Any concerns about import * should still be addressed by that too. How is the developer experience improved by having to admit that SomeModule is not defined in the same file? What's so special about a file?

(Also, you'd always be able to access SomeModule.x directly without importing it)

[jberdine]

One point that is not clear to me from this discussion is whether the import statements are something that could be inferred from the source code and shown to developers for readability purposes, or something else? Compare to inferred types, which can be shown to the developer for code understanding purposes, but do not need to be maintained manually since they are inferred. Is the same situation applicable for imports? Or is the goal more to make sure that every module is protected from new names being added to its dependencies, possibly causing clashes? In the second case, the established practice is to use open _extremely_ sparingly (e.g. one common preamble per project, or opened in a small scope no larger than the body of one function). I generally consider that any open needs to be explicitly justified during code review, and almost never is. An exception is that it is usually fine to open a module that only contains other modules, so that the open does not pollute the global namespace. But maybe I'm just an ogre. Another solution is to define/open a local copy of a module that is restricted to the signature you want, thereby protecting from newly added names.

Also note that I'm not sure how much help manually maintained import statements would be for the build system, since it would still need to track transitive dependencies and additionally ensure incremental builds work correctly in the presence of e.g. cross-module inlining. For this last point, see e.g. https://sympa.inria.fr/sympa/arc/caml-list/2016-07/msg00021.html .

Personally, I have found the distinction between interface and implementation files very helpful. Even for internal submodules, I sometimes define separate interfaces and implementations (module M : sig ... end = struct ... end) and generally write the interface first. The interface is/should usually be much shorter than the implementation, and the benefit of isolating the exposed api from the implementation details greatly helps readability of the exposed api for people who want to write clients. Also, the activity of writing the interface is one of the main opportunities for the author to realize that the exposed api is not as simple or clear as it could be.

I do agree that when reading the implementation, it would be nice if it was easier to answer the question of whether a given type or value forms part of the external interface. But I'm not convinced that this is the same issue as the desire to remove the separate interface itself.

I actually think that it would be more valuable to have a good mechanism for using uniform short module names across a project, while still being able to give longer more descriptive names for external clients.

[jaredly]

@jordwalke

What's so special about a file?

Files are units of code, they are a tool we have for organization :D I can imagine a variety of post-file worlds where they don't enter into the equation, but for now they're what we have.
More especially, files form a natural limit of the amount of things I have to hold in my head. If inter-file dependencies are not explicit, then I have to hold the whole project in my head

(Also, you'd always be able to access SomeModule.x directly without importing it)

I would vote to disallow that - require inter-file dependencies to be explicitly defined

[chenglou]

Man I just wanna organize my stuff using sharable OS X tags... this relegates the file organization into something local (akin to what reason is doing for syntax): personal, intuitive and opinionated (see: nuclide working sets). My way of organizing file might differ from yours and that's ok. For once we have a clean start for this.

Btw flat src/ is nice because of perf reasons (simpler globs, no need to validate that there's no duplicate module name, etc.).

I also like the natural tendency people have to name things a bit more distinctively with a flat dir. Instead of Messenger/DialogueComponent you have MessengerDialogueComponent (feel free to alias it to a shorter name at call site). Forces you to carry around more info.

[texastoland]

Where's the 😭 reaction 😝

[jaredly]

@jberdine

whether the import statements are something that could be inferred from the source code and shown to developers for readability purposes, or something else?

Having it encoded in text helps readibility in github / etc, which I think it still something we care about (otherwise why not just store the AST as json?)

I also think the argument you use about open (that it's useful to call out so you can think critically about it) also applies to inter-file dependencies. In the same way that it's a red flag for two OO classes to have lots of dependencies on each other (maybe you split things up wrong), the same might apply to two files.

The interface is/should usually be much shorter than the implementation, and the benefit of isolating the exposed api from the implementation details greatly helps readability of the exposed api for people who want to write clients. Also, the activity of writing the interface is one of the main opportunities for the author to realize that the exposed api is not as simple or clear as it could be.

I do agree that when reading the implementation, it would be nice if it was easier to answer the question of whether a given type or value forms part of the external interface. But I'm not convinced that this is the same issue as the desire to remove the separate interface itself.

Fantastic points! It's possible for them to be separate & still have the "what's exported" be obvious within the file.

[jaredly]

guess I oughta throw in the caveat now as later -- I have some strong opinions, and I like to engage these questions critically. I am however very fallible and am excited by cases where my strongly held opinions turn out to be inferior to new information. I really enjoy this discussion! 💃

[jordwalke]

Fantastic points! It's possible for them to be separate & still have the "what's exported" be obvious within the file.

The only problem with that is that now you have redundancy (to keep in sync etc). I think we could support .rei files for those who like them, but also come up with a way for .re files to play the role of both .re and .rei via the export keyword. My intuition is that the export keyword would only be encouraged when you don't have a separate .rei file.

[jaredly]

yeah, that sounds worth exploring

[jaredly]

(conversations from gitter)

[cullophid]

I used to hate it when language had imports like open Foo. It made it very hard to figure out where the functions was comming from, and how the code was organised. I saw it all the time in Java where people would have 20 lines of import foo.*. making it almost impossible to read the code without ide support.
For me reason is quite different though. Since the open keyword respects scope, you can open a module inside a function. I have round this amazingly useful especially with react.
when i write a render function I usually open Ui; on the first line. I now have access to all my ui components in the render function, and only in the render function.
Because of this I can keep it out of the larger module scope. In this case open actually makes it easier to find out where things are defined rather than harder.
If I was to use js style import instead two things would happen.

1. I would have to be explicit about what Importet (slight hassle, but generally an improvement)
2. I would move my import out of the render function.
   I don't think I would want to write `import {Column, Row, Button, Title, Card, CardHeader, CardContent, CardFooter, ...} from more than once per file.

I do like the idea of export though. I am not a big fan of the rei files (I admit that i dont understand the technical implications of them).
Also would it not be possible to allow rei style type annotations in re files ?