Compiler: Request: reexport values from dependencies

[evancz]

Issues #693 and #161 both request the ability to import values and then reexport them directly. For example:

module Foo (map) where

import List exposing (map)

Issue #227 requests whole module reexport, like this:

module Foo (module List) where

import List

Both cases can cause problems for automatic enforcement of semantic versioning. If someone does this from a module in a dependency, breaking changes in the dependency are suddenly breaking changes in this module.

[Apanatshka]

Both cases can cause problems for automatic enforcement of semantic versioning. If someone does this from a module in a dependency, breaking changes in the dependency are suddenly breaking changes in this module.

Isn't that always the case when you re-export things from a different package? I think you can already do this today by using:

module Foo (map) where

import List

map = List.map

I see how that might potentially break semantic versioning when you have a wide upper bound on core. I guess that should be statically forbidden then. But completely forbidding a package to be able to depend on different MAJOR versions of other packages may be annoyingly restrictive. I'm not sure if it would be a big problem though. So being conservative there would fix this entire issue of semantic versioning breakage.

[maxsnew]

I don't see how this would interact poorly with automatic semantic versioning. Could someone explain it more fully?

I consider this feature essential for separating a library's external interface with its internal module structure.

[texastoland]

One of you said this is an issue in an existing library? Re-exporting definitely strikes me as a code smell in any platform I've used. I'd be curious to see if there's a way to shuffle dependencies around.

[maxsnew]

I just bit the bullet and combined 2 files in my IO library into one module: https://github.com/maxsnew/IO/commits/master

[maxsnew]

@dnalot re-exporting is a code smell? I want it so I can have one module that is the interface to a library, while the actual implementation of the library has arbitrary module structure. What's wrong with that?

[texastoland]

Would you mind putting a gist together you've been trying to make work? With all the requests coming in, we need concrete use cases. Sometimes, there may be a more subtle change available (perhaps re-openable modules). Other times, there are patterns (maybe dependency inversion) that simply hasn't filtered from one member's codebase into the community.

[maxsnew]

How concrete do you need me to be? I've run into this in every library I've ever written but I don't think it would help to actually show the code.

Here's the essence. You have internal modules that have tons of code in them to implement whatever types and operations you have. You split them into multiple modules to maintain your sanity.

Here Internal1 defines a datatype and it exports its constructors and some operations on it so other internal modules in the library can use it.

module Internal1 (DataStructure(..), op1, internalOp1, internalOp2) where

type DataStructure a = ...
...
... tons of code
...

module Internal2 (op2) where

import Internal1 (DataStructure(..), internalOp1, internalOp2)

...
... tons more code
...

But now I want to publish this library and use it in my other code so I want to have one entrypoint and I want to leave the constructors to DataStructure opaque so I can maintain invariants and such:

module External (DataStructure, op1, op2) where

import Internal1 (DataStructure, op1)
import Internal2 (op2)

Right now I can't do this in elm. Since I can't reexport types, if I want the type to be exported opaquely in the library interface then I need to put all code that needs to see its internal structure in the same module.

Here's the problem in IO in detail:
Before I had one module that constructed the IO data type and operations like print and write to a file (which is supposed to be opaque to users) and another where I did a lot of stuff to serialize it to javascript and communicate over ports. The serialization code needed to see the underlying representation so it could serialize it. But then I had to have users import 2 different modules, one for getting the IO datatype and everything and another for getting the function that would actually make it run over a port.

Instead I shoved all of that code into one module so that users would only have to import one thing. To fix that I had to rename a bunch of things because the serialization module was using the same names as the internal module and now they were no longer namespaced.

If you really want to see the code then look at IO.elm and Runner.elm here and then the new version with everything in one file is here.

[texastoland]

Sorry @maxsnew I overlooked your previous link to IO. Thanks checking it out!

[maxsnew]

For an example that's not from me, the diagrams package exports all of the constructors to its datatype and includes a note that says "please don't use them": http://package.elm-lang.org/packages/vilterp/elm-diagrams/5.1.0/Diagrams-Core#Diagram .

I'm speculating but I think he's doing that for the same reason, the other modules in the package need to see the constructors, so he has to expose them externally or put the whole thing in one module.

[mgold]

There's a solution to this, and I use it in my turtle library. I use Turtle.Core to define the union type (among other functions) which is exported, including tags. However, "Turtle.Core" is not an exposed module. The result is that the Turtle and Turtle.Advanced modules can use core, and expose it how they like - sometimes directly, sometimes through a thin but important proxy like forward = Core.Forward. Because Core isn't documented with {-| ... -} style docs, I don't feel like I'm reexporting much. I think the Diagrams library would be greatly improved by this structure.

As for _import_, if you offer everything in nice modules, you shouldn't need to reexport them in one big module as well. Instead, the client code should do something like this:

import Turtle exposing (Step)
import Turtle.Advanced as Turtle

Yes, this will work and place all the functions into the Turtle namespace. The only trick is not to have any exposed duplicate names in the union of all exported modules. In practice, this isn't hard to do.

[maxsnew]

So this works because you can type alias the datatype and export the alias right? That works for my main uses (thanks!)

This would still be a problem if you DID want to expose the constructors of your data type, though, right? Like if I wanted users to be able to pattern match on my type.

Minimal example:

module Internal where
type Foo a = Foo a | Bar a

module Other(foo, Foo(..)) where

import Internal exposing (Foo(..))

foo = 3

module Top where

import Other exposing (Foo(..))


main = case Foo 3 of
         _ -> let loop x = loop x in loop ()

When I compile I get this error:

-- NAMING ERROR -------------------------------------------------- ././Other.elm

Could not export `Foo` which is not defined in this module.

1| module Other(foo, Foo(..)) where

[mgold]

Yeah, you'd have to play around and see. However, even adding tags/constructors will be a major release, because previously exhaustive patterns may no longer be such.

The other thing is I'm fairly certain the docs will read (in my case) type alias Step = Core.Step. That is, even though it's opaque since it comes from an imported opaque type, the docs still expose the existence of the core module. I released Turtle right before the "you must document everything" rules went into effect and I'm not sure what I'd do to update if I wanted to now.

[texastoland]

I finally got home to look through this. Let me know if I'm off the rails. Both IO and Turtle's root modules contain mostly boilerplate you want to eliminate by re-exporting. Both IO.Run and Turtle.Core intermingle public API together with implementation details. Fundamentally it'd be nicer to merge API into a single module from multiple files e.g.:

• module IO where (IO, etc.) (instead of IO.core)
• module IO where (run) (instead of IO.Runner)

[mgold]

But that means you couldn't import only some of the modules. One of the key features of the Turtle library is you can do import Turtle exposing (..) and then tell middle schoolers to have at it.

In your proposal, would functions from one file that are not exported be visible in other files in the same module? What keeps client code from reopening a module and messing around? Ruby manages to make "open classes" work, but it's a highly dynamic language where swapping out method definitions makes sense. In Elm, we'd have to statically forbid any duplication. If I add IO.foo and then (the other) Max adds a method of the same name officially in a minor release, that can break.

[texastoland]

Not a proposal just trying to understand. It's actually how Swift works which is very static. Stuff not exported is only visible within the file. In the case of Turtle I didn't understand the encapsulation. What would be different if the names exported from Turtle had been defined in Turtle instead of Core? Are you saying you have one API "for dummies" (Turtle) and another for power users (Core) without having to import both? Is there a use case besides teaching? Open modules is based on the notion of wanting to split API across files but not have overlapping API.

[mgold]

Technically the power user module is called Advanced, and it relies on Core underneath. The Advanced module is also pretty useless without the main Turtle module, which is not "for dummies" as it is "for kids". But anyway, yes it's overlapping APIs split across files, with the implementation in a third file. Some of the tags from core are exposed (through normal functions) in Turtle, others in Turtle.Advanced, and others not at all. Core exposed everything, but it's not exported.

[texastoland]

Okay I think I get it. You have three different classes of API user: typical, power, and kid. Regardless what class you want them to find everything they'd need to "get stuff done" in a single import. Fundamentally they could import all three but they'd have to know more than necessary. A secondary issue is your DSL may even depend on functions from some other package e.g. data structures. This distinction is important because it negates restrictions on only re-exporting "submodules" like Turtle.Advanced. A major concern I see is cyclic dependencies. This requires complex resolution algorithms in the Swift world.

[texastoland]

I.e. when I import both Core and Advanced, because they each expose functions that are mutually exclusive, I suppose you can ignore the re-exported version since they're guaranteed to export the same thing. The motivation makes sense but now when all I want is a couple functions from the kids module, I have the choice of any of the three. IDE tooling would ask which of the three you want to import without knowing how or whether they differ. Moreover your hypothetical exposed data structure function can't be guaranteed to interoperate with the same package of a different version I'm depending on. Right?

[mgold]

No, sorry, that's not right. There are only two classes of user - kids and adults. There are only two _exported_ modules, "main" (which is just Turtle) and Advanced. Kids just use main. Adults use main and Advanced, using the import Turtle.Advanced as Turtle trick from above to make it seem like there's only one module. The adults need the methods in the kids' module, and then some more. And for kids you'd import Turtle exposing (..) to make it seem like there's _no_ module.

Core is just an implementation detail - both modules use it, it's where all the crazy logic is, but clients don't need to know about it. And they won't, since it's not exported. Except when I'm forced to document type alias Step = Core.Step (sad face).

_Exposed_ is a property a value: is it visible outside the module, or not? _Exported_ is a property of the _module_: is it shown to the outside world when you publish?

[texastoland]

The concerns above about re-exporting are still relevant? Sent you a Hangout if you have time to connect the dots for me!

[mgold]

Oh, they totally might be. I was just trying to describe how I got Turtle to work.

[texastoland]

It sounds like Turtle presents an uncommon use case of hiding specific tags within a discriminated union and exporting others. @maxsnew Okay to close?

[maxsnew]

@dnalot Not ok to close since re-exporting a type with its constructors is still a perfectly legitimate use-case.

[mgold]

If we're just focusing on types, I can get behind that. Like I was saying about Turtle,

module Turtle.Core where
type Step = ...

--- another file
module Turtle where
import Turtle.Core as Core
type alias Step = Core.Step

Step, as seen from Turtle, is an opaque type, but will still be documented referring to Core.Step, which I don't want.

[texastoland]

@mgold I wondered when I woke up whether this could be implemented an an Either of two types, one exported (Step) and one internal (Step')? I could mock it up if helpful.

[mgold]

No, I'm not sure if that would work. But you got me thinking and I think _this_ would work:

type Step = S Core.Step

And then I just have to stick a bunch of Ss all over the place, but I think it would work.

No wait, I'd have no way to get the S tag in Advanced. So back to the drawing board.

[texastoland]

I'm going to fork it too.

[mgold]

Be my guest, let me know what you come up with.

[sporto]

I started making a lib and needed to export some functions and my actions to the user. At first everything was in one file and all was good. But as it grew I started splitting it.

At some point I had an internal module that wanted to use the actions in the root file. That created a circular dependency, so the solution was to move the actions to its own file.

That works great for the library internals but now I don't know how to export those actions cleanly to the client application.

Is there a solution for this? Putting everything back into one file seems like a poor approach.

[matiasfha]

I know this is kind of old but I'm have the same issue trying to export some types from an "inner" module to the caller (i'm writing a "library").

Example

{-- file src/Layout/Types.elm --}
module Layout.Types exposing (Direction(..))
type Direction
    = Row
    | Column


{-- main file of the library located in src/Layout.elm --}
module Layout exposing (..)
import Layout.Types as Types exposing (..)
type alias Alignment =
    Types.Alignment

{-- Main caller of the library (testing purposes) src/Main.elm --}
import Layout exposing (..)
type alias Model =
    { direction : Direction }

state : Model
state =
    Model Column 

The last file fails to find "Column" .

How can this be solved?

[evancz]

I do not want to track feature requests in issues for a couple reasons:

1. Rather than focusing on things that could be better, I am prioritizing things that cannot be done. In this case, the workaround is creating new definitions and type aliases. So this can be done, but could be better.
2. Feedback online is not representative of Elm users. The fraction of people using Elm who will never give feedback through a GitHub issue is certainly higher than 90%. So like in normal products, doing user studies is a better way to gather feedback.
3. If a feature request is "alright" it can sit around for a long time, cluttering up all the other issues that are related to bugs. Having it open or closed does not change whether we can look back at it though.

So for those reasons, I am closing this. We can still refer back to it at an appropriate time.

[maxsnew]

Looking forward to the user study!