Julia: Re-exporting one module's symbols from another

[dcjones]

It would be convenient if there were a way to do this.

module A
    export lots, of, stuff
    # ...
end

module B
    using A
    # magically export everything that was just imported from A
end

Doing this explicitly (copy the long list of exports from A into B) is a bit ugly and error prone. I imagine any alternative would involve a new keyword though.

Concretely, Gadfly is useless without symbols from Compose (and probably DataFrames), but a bunch of compulsory using statements isn't great.

[dcjones]

Actually, I would propose altering export's semantics to make this work:

module B
    using A
    export names(A)
end

[johnmyleswhite]

This is a very good point. I've taken to doing using DataFrames at the start of any new package code that has DataFrames as a requirement.

[StefanKarpinski]

This could be done with a macro, but maybe we should provide some syntax for it.

[stevengj]

My suggestion would be exportall A, which would re-export the exported names of A (but not the unexported names!).

[vtjnash]

A potential downside is that it may make discovery of exported names a bit harder. Also, I'm disinclined to support reserving another name exportall. So, alternatively, something like: using A as B with reexport perhaps? Or export A. (a module name followed by a period)?

[stevengj]

using A as B with reexport seems weird to me (and if you are willing to add a reexport keyword, why not reexport A?), but export A. seems fine to me if a bit subtle.

I don't understand your concern about discovery of exported names; could you clarify?

[vtjnash]

Currently, it easy to parse any julia file, search for export at the start of an expression, and interpret the comma-separated list that follows to quickly identify the public interface for a module.

using ... as ... with reexport adds special syntax to using, not a new keyword

[stevengj]

I see, you are worried about manually parsing for exports. It's not too bad with reexport, though—you just need to do the same thing recursively for each reexport.

If you are wedded to using, I would tend to just support using A reexported.

[vtjnash]

I'm just tossing around other proposals for consideration. Very likely, none of them are inherently better. I didn't mean to imply the as ... part was required. using A rexported or using A reexported as B or using A as B reexported would all be valid. Whether this is a single word or a phrase depends on whether we want to be closer to a english sentence or a keyword. I could accept either.

[staticfloat]

I personally kind of like exportall. It's got a nice symmetry with importall.

[simonster]

I made a macro that mostly works for this pupose. I say mostly because relative module syntax does not work. The parser parses @reexport .MyModule as reexport.@MyModule and throws a syntax error on @reexport .MyModule1, .MyModule2. But @reexport MyMod.MySubMod should work.

[jiahao]

If we ever get a resolution of this issue, we would be able to remove hundreds of lines' worth of redundant-seeming exports from Base.LinAlg and then again from Base.

[kmsquire]

I'm with @staticfloat, in that I find exportall most appealing.

What's the downside to adding that as a keyword?

[simonster]

What does exportall do if you haven't imported/used anything from A, e.g.:

module A
    exported = true
    export exported
end

module B
    exportall A
end

Exporting bindings that are not accessible in B is not really an option, so does it throw, export only the A module and none of its exported symbols, or implicitly call using or importall? Unless it does something implicitly, it is more verbose than the alternatives, since you need two lines of code instead of one and you need to repeat the module name(s) twice. And if it does something implicitly, then we are adding a fourth keyword that imports bindings.

[kmsquire]

What does exportall do if you haven't imported/used anything from A

Throw an error. Even though it's more verbose, I find it clearer and more natural than most of the proposed alternatives.

[simonster]

If exportall is going to throw if the module isn't already imported/used, it may as well be a macro? Since all imported/used modules will be in the module's scope by virtue of using, it's not clear to me it needs to support relative module paths, which is the only thing a macro couldn't do.

[tknopp]

Whats the advantage of being a macro? It would be totally inconsistent if we have import, importall, export, @exportall

[simonster]

Minimalism? It would be one fewer reserved identifier, one fewer Expr head, and in Julia instead of C. It is already a little awkward if it joins end, elseif, and else as (I think) the fourth keyword that throws an error if another keyword has not been used before it, and in this case it's a runtime error instead of a parse error.

I kind of see the symmetry, but it doesn't seem quite right to me: You can add bindings from another module to the current module and you can export bindings in the current module from the current module, but you cannot export bindings in another module from the current module unless those bindings are already also present in the current module.

Personally, I would still prefer #5608 or #5626, which combine the loading and exporting of the bindings into one step. It seems to me that:

using .Windows, .Periodogram, .FFTFilt, .FilterDesign, .Util
exportall Windows, Periodogram, FFTFilt, FilterDesign, Util

is less elegant and easier to screw up than:

@reexport using .Windows, .Periodogram, .FFTFilt, .FilterDesign, .Util

But if everyone else wants exportall and is averse to the @ symbol, I can try to make that happen.

[tknopp]

Well, minimalism is a point that is always a concern when thinking about adding a keyword.

I think the more general question is whether the regular Julia user should be confronted with the usage of macros. I would say no. This is an advanced thing and should not be part of the regular workflow.

exportall is from my perspective something that will be needed in the "regular use" Its quite common to structure modules (namspaces, ...) into two levels.

[lindahua]

@tknopp: Yes, _writing_ a macro might be an advanced thing that ordinary users should not need to worry about. However, I don't think _using_ a macro is that advanced -- they don't even need to understand how a macro works behind the scene in order to use it. All what we need is to clearly document the macro's usage.

For instance, @inbounds is a good example which everyone can easily understand and use.

Also, a code author should be quite tech savvy when he is worrying about something like re-exporting names.

[tknopp]

@lindahua: macros are great. But still they do "magical" code transformations that I think most users should not be confronted with. @inbounds is a good example. This is a thing for advanced usage not for regular usage.

I think reexporting names will be not that uncommon in practice. The export mechanism is Julias way for information hiding. And having submodules in modules is common and should also be encouraged.

[pao]

A better example for "macros all users should be comfortable with" is @printf.

[kmsquire]

A better example for "macros all users should be comfortable with" is @printf

Which I think should be replaced with a pure Julia implementation when we can. (I have one partially implemented.)

[simonster]

@printf is pure Julia. It's just a macro. While a function could be useful for rare cases where the format string could vary at runtime, I am not sure how a function could be as performant as the macro, since the macro generates code based on the format string at load time.

We also have @__FILE__, which is a keyword in most other languages.

[tknopp]

Probably @time and @assert are the best examples of macros for which my statement that macros are only for advanced users does not hold.

[StefanKarpinski]

Another approach to this is not to import and then re-export, but rather to make these modules somehow "inherit" from the other modules. I.e. something like this:

module A
    export lots, of, stuff
    # ...
end

module B
    export lots, of, stuff
    # ...
end

module C <: A, B
    # has all of A and B's bindings and exports
    # but adding bindings doesn't affect A or B
end

In particular, one could consider a "bare module" – currently created with the baremodule keyword – to be a module which inherits from no modules. However, you don't want this to be the default since it's handy to import all operators and have all of Base's exports available via using. This default behavior could be expressed as this parentless module:

module Default <: ()
    importall Base.Operators
    using Base
end

Then the default module behavior would be this:

module Mod <: Default
    # as if we'd already done importall Base.Operators and using Base
end

Thus, writing module Default <: () would be equivalent to the current baremodule and writing module Mod by itself would just be shorthand for module Mod <: Default.

[simonster]

@StefanKarpinski That proposal seems elegant, but I'm a little worried about the nested module case. I guess you'd have something like:

module DSP <: .Windows, .Periodogram, .FFTFilt, .FilterDesign, .Util
module Windows
    (...)
end
module Periodogram
    (...)
end
(...)
end

To make this work, we'd have to first load all nested modules, then import their bindings, and finally load the main module. I'm not sure how big of a change this would be.

I'm also not sure one always wants to inherit unexported bindings from other modules. In the nested module case, there's no reason we'd need the unexported bindings in the main module, and some could conflict.

[StefanKarpinski]

I didn't really mean for unexported bindings to be inherited. But yes, that is what I said.

[StefanKarpinski]

Needs some more thought, but I think there's something clean here.

[tknopp]

I like Stefans proposal if it plays nice with with submodules.

[simonster]

@StefanKarpinski Any further thoughts? Thinking about this more, I am a little worried about creating another way to import bindings that is subtly different from using, import, and importall. This might confuse more than it clarifies.

[thusly]

I recently read a Julia dev paraphrasing someone else, in saying that Julia takes a "we're all consenting adults here" attitude. I'd very much like to be able to consent to exporting all symbols (or having a module inherit the exports of a parent, following @StefanKarpinski's implementation idea).

If I'm writing library A, I'd like to be able to divide its concerns up into submodules of X and Y. In this way someone who only needed, say, the "Network" portion could employ using A.Y, but most users would be opting for using A. There isn't any more implicit namespace pollution involved here than in the using call itself; the module inheritence would still only be acting on explicitly defined exports from modules X and Y, but I would no longer need to export every new function added to either submodule twice. (And with further nesting comes further code duplication.)

I love the module implementation and its decoupling from the files themselves, it makes structuring a project (and later refactoring) that much easier. But without being able to shuttle around the symbols I'm exporting from submodule to parent, maintaining that clean organization becomes a constant thorn in the side, as you micro-manage your exports from one level to the next.

It would be of further use in reducing code duplication when you have a group of conceptually related packages or modules that are always imported together. For a concrete example, there's the OpenGL library. You have the actual API versioned symbols you require, as well as the GLU library, and your own wrappers and convenience functions. For every submodule you write, you're importing three separate modules that will always be imported together in your code, instead of being able to do something like:

module GL <: OpenGL, GLU
# Convenience functions here
end

Sorry for the lack of brevity, but I wanted to share some use cases. I'm a new Julia user coming mostly from python, and I've been bitten by this problem multiple times over the last few days.

[papamarkou]

Has there been any further progress, decision or discussion on this matter? Is @simonster 's Reexport package the agreed standard way to re-export the symbols of another package, i.e. should I use it?

[johnmyleswhite]

Since I haven't seen any other options emerge, I think Reexport is the de facto standard.

[papamarkou]

Thanks @johnmyleswhite :) I will be relying on it and will keep an eye on this thread.

[tknopp]

It would be really great if we could have this functionality in Base. It would be great to have metapackages that simply group several packages together in order to provide a larger set of functionality.

[vtjnash]

Needs some design thoughts to figure out how we want this to work and interact with using / import / importall / etc.

[timholy]

Crossref #14472

[vtjnash]

I put together a macro implementation of something like this for one of my PRs as a demo: https://github.com/JuliaLang/julia/pull/22147/files#diff-4fc3422b2208cb02c3e17c7cc0a56446R44

[Keno]

Seems feature-y. There might be syntax we could want, but we if that does happen we can live with a macro until 2.0.

[ip1981]

Re-export is evil.

[epatters]

I disagree that Reexport is evil. In writing a package with many levels of submodules, I find Reexport very convenient for managing namespaces across levels without lots of boilerplate code and code duplication. I echo the sentiment expressed above that this would be great functionality to have in Base.