Julia: Some sort of policy to regarding unexported functions in Base?

[IainNZ]

There have been a couple of times during 0.4 development where unexported functions have been removed or changed. Every time its come up we seem to go in circles a bit about what to do about it. Given the resources available, I feel we should just keep it simple:

No support or deprecation warnings, just change/remove them. Only exported functions get deprecation warnings or mitigation regarding changing arguments.

But we need consensus around this to make it stick.

[timholy]

Even though I've paid the price for this myself on occasion, I can't imagine any other sensible policy. :+1:

[johnmyleswhite]

+1

[rsrock]

+1

[ScottPJones]

Unexported things are definitely supported in Julia, and are deprecated all the time.
Things like to_index and compilecache are not exported, for example.
compilecache had been called compile. If that had made it into 0.4, instead of being changed before release, it would have needed to be deprecated as well.
The real issue is that Julia doesn't have any way currently of declaring that a function/method/type should not be accessible outside of it's module, which would have prevented these cases of internal functions being used outside base, and there are other valid reasons for not exporting a name (to avoid name pollution).

[timholy]

In general user-code should not call to_index or compilecache. __precompile__ is an exported function.

[kmsquire]

A very minor example of a function that would be affected by this policy is Base.permute!!(a, p). It exists so that Base.permute!(a, p) can permute a in place, but leave the permutation p untouched. But if a user knows that p is okay to clobber, she may wish to call the double-bang version, to avoid copying p.

It's unexported and undocumented, and therefore not widely used outside of Base (only DataFrames that I'm aware of, probably because I added it there). It's also the only example I'm aware of in base with a '!!', which is probably why there was resistance to export it in the first place (ref: #2201).

I don't think this is anything that should prevent the policy proposed here from being put in place (these things should probably just be handled on a case-by-case basis as needed), but I thought I would mention it.

[aviks]

I am agnostic about this being a _policy_ in Base . However, I think we should be careful in claiming this to be an overall Julia design guideline. (Not that anybody is suggesting that at the moment, but people tend to follow practices from base, and so I thought I should articulate this, if only for the record)

Given Julia's module features, it is sometimes good design, and at other times necessary, to not export names from some module. Every package can, and needs to, make this decision on its own.

[ivarne]

I think @aviks is right, and that there should be room for documenting (and deprecating) unexported names (especially in packages, but it also makes some kind of sense in Base).

Maybe it would make sense to tie deprecation/midigation policy to documentation status, instead of only exportation status?

[IainNZ]

Updated title to keep this focussed on Base only, I don't want to get into packages (they can do whatever the feel is appropriate).

[mauro3]

One counterexample to this proposal in Base is the Profile module which only exports @profile. But quite a few methods belong to its public interface, for instance Profile.print.

I think a way to specify which unexported functions belong to the public interface would be good. Which is similar @ScottPJones's comment above about making methods inaccessible (although I wouldn't go that far).

[toivoh]

I agree.

[tknopp]

Why aren't these functions in the Profile module exported?

My mental model was that public==exported which seem to hold in 99% of the cases except where submodules are used for introducing namespaces

[timholy]

Profile does introduce a namespace. Notice the functions are called print, init, clear, and retrieve; odds are quite high that you don't want Profile.print instead of Base.print each time you say print. Given that we have modules, there's no reason to call them profile_print, profile_init, and profile_retrieve. Pkg behaves by similar rules.

[alyst]

Just a thought from a user perspective.
I think in general it would be nice to let power users/package developers to take advantage from some "advanced" Base functions (e.g. permute!!()) and commit to tracking their compatibility between the releases.
Such functions are not meant to be called from the REPL, so they could be safely left unexported. However, they still need to be a part of Julia documentation (i.e. have a @doc block) to be properly used. That could be their distinctive feature.
If somebody wants to use something Julia developers have not documented, she/he should be really on her/his own.

[aviks]

Given that we have modules, there's no reason to call them profile_print, profile_init, and profile_retrieve. Pkg behaves by similar rules.

This is the kind of use I was referring to. I was thingking of JSON.write() but there clearly are lots of such uses in Base.

My mental model was that public==exported ...

I don't think that can or should be true in Julia.

But let me say, while I am debating the larger design problem, as far as the specifics of this issue goes, I am all for nuking code aggressively. Julia is still a 0.x language. If it looks like some code is un/under-used, delete it, with a good faith effort at notifying packages. At this stage in our lifecycle, that is perfectly acceptable, nay, necessary. Also (to prevent some counterarguments), I say this as someone who uses Julia in real life :).

[tkelman]

I don't like making this a policy, as phrased it seems to be encouraging introducing breaking changes more often without any migration options. Yes it's risky to depend on unexported functionality from base, we should write that down in some form and let it be known, and be more careful about only doing so when absolutely necessary in packages. And we need to be able to detect and highlight more clearly when it's happening, so we know about it in advance. Is this one of the things Lint.jl can or does check for? Have many established packages started using Lint.jl in their development process the way CI and coverage tools are being used?

We shouldn't be introducing frivolous breakage, even in unexported functions, without good reasons for doing so (there are plenty of good reasons as the language develops, but that's a difficult line to draw in policy terms). There are enough cases where using unexported functionality is being done and necessary, that "break things at will" as a written policy means things will be broken more often. Until we have the hardware resources to run PackageEvaluator reliably, 100% daily, and/or on pre-merged PR's, this makes working against -dev versions of Julia even less stable. We've done a really lousy job of keeping users away from -dev versions. Turns out people like new features, and releases are taking time. When users are the ones finding breakage in packages before the package developers do, and are just reporting issues instead of helping fix the problems with PR's, that's a sign that too many people are on -dev. And that we're not doing a good job of teaching enough people how to participate in fixing the bugs they find when using unstable versions of Julia.

As I see it, base has more developer resources than packages. Less effort here for more breakage in packages doesn't seem like a good tradeoff, unless we manage to change the onboarding process of who uses -dev versions of Julia and how breaking changes get fixed across the ecosystem.

[StefanKarpinski]

I think the issue here may be that we need separate mechanisms for expressing that something is public and expressing that it should be exported (in the current sense).

On Aug 16, 2015, at 7:48 PM, Tony Kelman [email protected] wrote:

I don't like making this a policy, as phrased it seems to be encouraging introducing breaking changes more often without any migration options. Yes it's risky to depend on unexported functionality from base, we should write that down in some form and let it be known, and be more careful about only doing so when absolutely necessary in packages. And we need to be able to detect and highlight more clearly when it's happening, so we know about it in advance. Is this one of the things Lint.jl can or does check for? Have many established packages started using Lint.jl in their development process the way CI and coverage tools are being used?

We shouldn't be introducing frivolous breakage, even in unexported functions, without good reasons for doing so (there are plenty of good reasons as the language develops, but that's a difficult line to draw in policy terms). There are enough cases where using unexported functionality is being done and necessary, that "break things at will" as a written policy means things will be broken more often. Until we have the hardware resources to run PackageEvaluator reliably, 100% daily, and/or on pre-merged PR's, this makes working against -dev versions of Julia even less stable. We've done a really lousy job of keeping users away from -dev versions. Turns out people like new features, and releases are taking time. When users are the ones finding breakage in packages before the package developers do, and are just reporting issues instead of helping fix the problems with PR's, that's a sign that too many people are on -dev. And that we're not doing a good job of teaching enough people ho w to participate in fixing the bugs they find when using unstable versions of Julia.

As I see it, base has more developer resources than packages. Less effort here for more breakage in packages doesn't seem like a good tradeoff, unless we manage to change the onboarding process of who uses -dev versions of Julia and how breaking changes get fixed across the ecosystem.

—
Reply to this email directly or view it on GitHub.

[timholy]

Yep. Currently, the best indication is: is the function documented? If not, it's likely not intended as part of the public API. All the exceptions raised so far are in that camp.

[ScottPJones]

Even things that are not intended to be part of the public API should have internal documentation though,
just so that somebody besides the original author (or the author years later) can figure out what is going on.

[timholy]

devdocs

[ScottPJones]

Having devdocs is very good, but is not a replacement for having internal documentation of functions.
If you have to deal with code whose lifespan is measured in decades, it is rather important.
I'd like to think that Julia will still be relevant, being updated and supported in 30 years from now.

[hayd]

Perhaps help could say that the method is not exported?

It's impossible to support every internal method change between dev versions (e.g. 0.4-dev) but perhaps it would be worthwhile comparing major versions. i.e. upon 0.4.0 what internal methods have changed or been removed since 0.3.

[ScottPJones]

I think the issue here may be that we need separate mechanisms for expressing that something is public and expressing that it should be exported (in the current sense).

This is really not essentially different from what I've been suggesting, except that it is opt-in rather than opt-out (of being a public, not exported function/method/type).
Doing it opt-in, as @StefanKarpinski suggested, is probably better, since it seems that it is less common in Julia to have things public but not exported, than to have things not exported with the expectation that they are "private" in some sense.

@hayd It would be a good idea of having help display that, yes, esp. if there is a way of marking something as part of the public API, but not exported.
No, you wouldn't want to try to compare internal method changes, the real problem right now is that you can't distinguish between what is really internal and what really _is_ part of the public API.

[tknopp]

@StefanKarpinski, @JeffBezanson: Could you clarify if the intention of export was that exported functions form the public interface while unexported functions are private? We see that there are currently exceptions (submodules like Pkg and Profile) but it before putting to much weight on the exceptions it would be interested what the purpose of export was/is.

[StefanKarpinski]

export controls what is available via using – that's it at the moment. It's an open question whether we need a different notion of public/private than that. One idea is to separate the concept of "exported" from "public": exported things are made available by using whereas public names still need qualification but are visible; the implication would be that anything not public would no longer be accessible even with qualification. Of course, you would still be able to sneak around that restriction using eval, but it would make it much more obvious that you were, in fact, doing something sneaky. It would probably make sense to allow non-const public bindings to be written to externally as well. So that would mean you'd have this kind of thing:

module Foo
    xx = 0
    const ro = 1
    public rw = 2
    public rwi::Int = 3
    usable = 4
    export usable
end    

using Foo

xx                  # not defined
ro                  # not defined
rw                  # not defined
rwi                 # not defined
usable => 4

Foo.xx              # not allowed
Foo.ro => 1
Foo.rw => 2
Foo.rwi => 3
Foo.usable => 4

Foo.xx = -1         # not allowed
Foo.ro = -1         # not allowed
Foo.rw = -1
Foo.rwi = -1
Foo.usable = -1     # ???

Foo.xx = "zz"       # not allowed
Foo.ro = "zz"       # not allowed
Foo.rw = "zz"
Foo.rwi = "zz"      # not allowed
Foo.usable = "zz"   # ???

Having export and public be orthogonal may make sense. This requires some thought.

[tknopp]

I know that export is currently about scoping. In my opinion public and export are not orthogonal and it would complicate the system to much. We IMHO need a single mechanism to say what the (public) interface of a module is.

Can't the submodule use case be solved differently? I.e. the submodule exports the function and the mother module does not reexport the individual functions but only the scoped ones?

[mauro3]

@tknopp: your last suggestion does not work in the case of Base.Profile.print as exporting print would clash with Base.print. Renaming it could work though.

[ivarne]

Exporting Profile.print() would work, but we'd have to change sysimg.jl to explicitly import only @profile, instead of calling importall. (or am I just totally wrong?)

[hayd]

See also https://github.com/JuliaLang/julia/issues/12064#issuecomment-119900260 which I think is similar (privacy of fields) and touches on this.

My thoughts are the same as on that one: what are you going to do when users want to access some private method, stop them doing it completely? If there's a workaround what's the benefit? and if there is a workaround why not just use underscore to _denote_ private, like in python, rather than enforce it.

_Does the fact that no-one has put a package together since that discussion suggest there's little appetite for this????_

[carnaval]

Whatever the decision is, there has to be a workaround. If there is something I hate it's the computer not doing something because it "knows better". I know better, you silly pile of microwaved sand.

[tkelman]

you silly pile of microwaved sand

Heh. I agree, enforced privacy is annoying when debugging and doesn't seem like the right solution for Julia. What I think I'm looking for is a technical-debt-detector so people other than the immediate author of the risk-of-breaking uses of non-exported code (even if it's the same person, but revisiting code 6 months later kind of thing) have an easier way of identifying problems before they happen. Breaking things in base for good reasons helps the language move forward, but the fallout of leaving things in a broken state for weeks afterwards in packages is bad for users. Maybe Pkg.add and Pkg.clone in -dev builds of Julia should have big noisy warnings, "you are using an unstable development version of Julia, if you find bugs in packages please help fix them."

[ScottPJones]

Enforced privacy may be annoying when debugging, but it is a lifesaver when trying to keep your sanity over years (or decades) of maintaining a codebase.
Solution: have the enforced privacy on by default, but be able to turn it off by a command-line switch, have have it off by default for debugging builds.

[StefanKarpinski]

Whatever the decision is, there has to be a workaround. If there is something I hate it's the computer not doing something because it "knows better". I know better, you silly pile of microwaved sand.

:heart: @carnaval

[rsrock]

Better trademark that quote, or it's going to end up on t-shirts and you'll miss out on a heap of cash, @carnaval!

Didn't Leah Hanson once say that one of the nice things about Julia is that it doesn't feel like you're fighting the compiler? I'm really afraid these proposals will end up with us cursing and beating the microwaved sand.

Here's what I don't like about this proposal (which seems to have veered off course, btw):
1) Not being able to use what's sitting in front of my face, because the pile of sand won't let me.
2) "public static final" hell. Throwing decorators around like that just makes code noisy and unreadable. Unfortunately there isn't a kill switch that will remove these from the code itself (I suppose I could "sed" everything if I really needed to)
3) My favorite of all, reading code written by scientists (my kin) where we try to work around all of these unnecessary restrictions. I've seen getters and setters defined in class A that will call other getters and setters in class B that then get or set something in class C. Encapsulation? We don't need no stinkin' encapsulation!

If you must have these "features", there's always Java for you.

But I'd like to try to pull this back on course. The topic here is whether the interface extends beyond the exported methods, and how to deal with things like deprecations. I had always thought that the only "public" methods and fields were the exported ones, but the Pkg and Profile examples show that public scope is wider than that. However, @timholy's point that things like Pkg and Profile methods are documented is a good one. So I'd propose that no deprecations are needed unless the item is 1) exported, or 2) documented.

When I use or modify something that's not in class 1 or 2, I always comment it as such. If it changes or goes away, my future self will understand why. That way, I'm never angry at the pile of sand in front of me, I just go and fix it.

(And if we must have something like privacy, _foo and __foo seem perfectly adequate. These should be kept to a minimum as a matter of style, to indicate "dangerous" code. Everything else public by default)

[ScottPJones]

At the very least, with a public keyword such as in @StefanKarpinski's example, you could:
1) make it a rule that registered packages "follow the rules" (and if they really need something to be made public, they can always submit a PR)
2) have a command line switch to turn on enforced privacy (if people prefer it off by default)
3) the very nice Lint.jl can check for violations.
It would make the author's intent totally clear, and prevent nasty surprises like when I made some of the internal UTF functions more generic, and accidentally broke JSON.jl and all the packages that depend on JSON.jl.

(I do agree with @carnaval, BTW, whatever the decision, we'd need a workaround. The idea is to make developers lives _easier_, not annoying)

[ScottPJones]

@rsrock, see my comment https://github.com/JuliaLang/julia/issues/12647#issuecomment-131666031.
Even totally internal functions need to be able to be documented. Not doing so just makes life hell for your future self and others who need to understand the code.
To me, having to decorate things with _ is just a pain, I'd rather add 7 characters (i.e. public) in front of the very few un-exported things that I really do want as part of the public API, than make have to prefix all my functions, fields, etc. names with _, _every_ time I use them.

[hayd]

Perhaps it would be good to count what things (in Base, or in external packages) could/should be internal, so that we have some idea of what we're talking about here.

_(IMO that the _-prefix is really not that bad, but I guess I think "public by default" makes more sense...)_

[rsrock]

Comments in the code itself would seem to cover that situation.

Also, if you're really making absolutely everything private by default, consider how difficult you're making it for folks who have to use your code. Sometimes that private stuff is actually useful. In general, it seems that folks slap the private label on stuff without considering if it's actually necessary. Is it really really critical that no one touches your uber_foo() method? Usually not.

@hayd, just to be clear, I'm advocating for "public by default", using _ only when necessary to mark "danger, private!"

[ScottPJones]

For example, I've seen a ton of code that accesses .data when working with strings. Problem is, that doesn't always work even now, and there will be major breakages if we try to put in better string handling.
That is why just saying, "we don't care about encapsulation" gets you into serious trouble down the line, where you are prevented from making any improvements in the language, because too many people are accessing internals directly.
"public by default" is the direct opposite of what most people seem to think is the case in Julia,
many people have told me that if it is unexported, you don't have to worry about changing it or deprecating it.
A public keyword, as @StefanKarpinski showed, would only be needed sparingly, for the few cases where unexported names really are part of the API.

[tknopp]

This thread is (again) hijacked about a discussion of enforcing privacy. Why can't we have a single keyword that say: "This is part of the public interface". I see that there are currently issues with submodules and scoping but can't these be solved? From a higher perspective its pretty unintuitive that submodules do not export their functions and this IMHO is some inconsistency that could/should be solved.

I am also linking #8005 where export/public were also discussed.

[nalimilan]

I like the way it works in R, and it looks like it would work fine in Julia: all functions in a package are private by default, and they can be accessed via pkg:::function() (ordinary functions can be called via function() when the package is attached, or pkg::function() if not).

I think it would be cool to adapt this to Julia, and make functions only available via Pkg..function when not explicitly exported or declared public. That syntax still allows using private functions without too much burden when debugging or if you're willing to take the risk, but still makes the unstability quite clear. Lint.jl would easily catch this kind of use, and it could be forbidden or discouraged for registered packages.

[mauro3]

To keep on hijacking this issue: I agree, the .. would be ideal for this. (It would also work with #1974, if that comes.) Then, a keyword would be needed, probably public as suggested above. And maybe one more, say public_all, to make all methods, etc. public, which would then be equivalent to how it is now. Or maybe, similar to baremodule, we could have publicmodule.

I think this would also solve #12069.

[ScottPJones]

The only thing I would suggest differently from what @mauro3 just said, is to use something other than .., because that is already wanted for intervals.
Maybe %%?

[ScottPJones]

Or .!?

[rsrock]

@tknopp: I think we already have a single keyword that says "this is part of the public interface". It's export. I agree with you further up that public and export are not orthogonal when it comes to defining the interface.

@ScottPJones: regarding "public by default", we have to be careful to define the two senses of "public". Because I can directly access fields and methods, I consider Julia to be a "public by default" much the same way that Python is. See https://docs.python.org/3/tutorial/classes.html?highlight=private#

In C++ terminology, normally class members (including the data members) are public (except see below Private Variables), and all member functions are virtual.

However, that's separate from defining the public interface (the real topic of this issue). I gather that you're beating yourself up over causing some breakage when you changed some exported functions. My advice: don't beat yourself up over this. If you have changed unexported and undocumented functions / fields, it's not your problem. (That said, we shouldn't go around making frivolous changes and churning the code either)

[MithrandirMiles]

I’m not beating myself up about anything, I just feel, from 29 years of being the principal architect (that was ANSI standardized, so I’m not to blame for the original syntax!) of a language that originally did not have any distinction between publicly accessible and not, until I managed to get that added to the language, that being able to keep things encapsulated is critical to good software engineering, and building reliable code.
The bad case I had was changing en unexported / undocumented function, which broke JSON.jl, which broke a whole ton of packages. :cry:
I’m not saying either that in Julia, it should not be possible to go around the system, in need, for debugging, etc.
Since most people seem to feel that by default, most things that are not exported should be treated as private, the simplest case would be to have a public keyword to indicate something (method or field) is part of the public API (which would only needed for non-exported names, a rare case).
Accessing them as pkg.!function() or typeinstance.!field would not be burdensome (and would be much less frequent than all the cases of having to add _ to denote “private”), and could easily be searched for, and checked by Lint.jl.
Everything could remain publicly accessible by default, with just a command line switch for a strict mode, which would be useful for production code (which is what I am/will be using Julia for)

[tknopp]

@rsrock: Yes until this thread my mental model was also that export is what defines the public interface. So I am absolutely happy with this. Tim pointed me to the exception with submodules. And in my opinion it makes more sense to think about how to "fix" the submodule use case (i.e. these submodule have to go through some export at some point) then to introduce a new keyword which is not orthogonal.

[rsrock]

@StefanKarpinski

export controls what is available via using – that's it at the moment. It's an open question whether we need a different notion of public/private than that.

I'd argue that we don't need a different notion of public/private. To take your example

module Foo
    _xx = 0
    const ro = 1
    rw = 2
    rwi::Int = 3
    usable = 4
    export usable
end    

using Foo

xx                  # not defined
ro                  # not defined
rw                  # not defined
rwi                 # not defined
usable => 4

Foo.xx              # not allowed, but you can cheat with _xx
Foo.ro => 1
Foo.rw => 2
Foo.rwi => 3
Foo.usable => 4

Foo.xx = -1         # not allowed, but you can cheat with _xx
Foo.ro = -1         # not allowed b/c const
Foo.rw = -1
Foo.rwi = -1
Foo.usable = -1     # ???, should be fine

Foo.xx = "zz"       # not allowed, but you can cheat with _xx
Foo.ro = "zz"       # not allowed b/c const
Foo.rw = "zz"
Foo.rwi = "zz"      # not allowed b/c Int
Foo.usable = "zz"   # ???, should be fine

Advantages of doing things this way:
1) This is what we already do. Plenty of examples in Pkgs (~1500 in my .julia, with admittedly imperfect counting).
2) No code is required. (well, perhaps a bit. The assignments in blocks 3 and 4 don't work currently)
3) No. 2 means that it's easy to change in the future, or to add packages that define other privacy capabilities, if absolutely necessary.
4) If you want namespaces, use import. The public interface can still be defined through the exports, however.

I think all it needs is some documentation. I'd be happy to take a stab at this in a PR. Where should such a thing live, and what files should I edit? I know that the manual is in a state of flux at this precise moment.

[rsrock]

@tknopp

Tim pointed me to the exception with submodules. And in my opinion it makes more sense to think about how to "fix" the submodule use case (i.e. these submodule have to go through some export at some point) then to introduce a new keyword which is not orthogonal.

Agreed

[jdlangs]

Could export be expanded to include fully-qualified names? That way the public interface is anything exported but the author still has fine-grain control over what using brings into local scope.

module SomeMethods
pub1(x) = ...
pub2(x,y) = ...
internal1(x) = ...
internal2(x) = ...
export pub1, pub2, SomeMethods.internal, SomeMethods.internal2
end

or

export pub1, pub2
export SomeMethods: internal1, internal2

[rsrock]

@ScottPJones Likewise, I have plenty of experience in the restrictive, Java and Java-like interfaces, and the more open, Python and Python-like systems. I greatly prefer the latter, especially for technical computing (the target audience of Julia).

Since most people seem to feel that by default, most things that are not exported should be treated as private, the simplest case would be to have a public keyword to indicate something (method or field) is part of the public API (which would only needed for non-exported names, a rare case).

But that's the problem-- I want the interface to be defined through export (+ docs), but to be able to access methods or fields, and I don't want the pile of sand to scold me about it. In other words, I want it to be somewhat troublesome to define something as private. I want you to have to stop and think "does this really, really need to be private?" The friction it introduces for privacy is a positive feature of this system.

In my reply to Stefan, I forgot to link the relevant section of the Python manual where they discuss privacy. I think they do it right, and they do have a rather long track-record to show that this _ and even __ business works pretty well.

https://docs.python.org/3/tutorial/classes.html?highlight=private#tut-private

[rsrock]

@phobon You can do this right now, with import vs. using:

julia> module SomeMethods
       foo() = 1
       bar() = 2
       export foo
       end
SomeMethods

julia> import SomeMethods

julia> foo()
ERROR: UndefVarError: foo not defined

julia> SomeMethods.foo()
1

julia> bar()
ERROR: UndefVarError: bar not defined

julia> SomeMethods.bar()
2

julia> using SomeMethods

julia> foo()
1

julia> bar()
ERROR: UndefVarError: bar not defined

julia> SomeMethods.bar()
2

[jdlangs]

@rsrock The intention was to let the package author annotate what methods are considered the public interface without also requiring them to make them visible via using. Anything not exported would then be liable to change without warning or deprecation.

Essentially combining @StefanKarpinski 's public keyword with the export statement.

[rsrock]

I see, to cover cases like Profile.print()? I suppose that export SomeMethods.foo could then be a no-op. That way, using SomeMethods will still require you to use SomeMethods.foo(). Interesting idea, that means that export could define the entire public interface after all.

[mbauman]

+1 to a export SomeModule: names. These names would show up in names(SomeModule), and tab completion could be restricted to names(SomeModule, #= all= =# false).

Also note that you _can_ completely restrict access to local bindings with let. But I don't want to encourage such behavior in general. :)

[StefanKarpinski]

So export would not actually export anything? Confusing

[jdlangs]

Well, it wouldn't be a total no-op, since as @mbauman suggested, the module reflection methods would know that such identifiers are meant to be public. It would imply changing the meaning of export from the definition you gave though.

[StefanKarpinski]

My point is that then the term "export" would have lost all appropriateness and we might as well rethink what we call it. The business of using import to extend things is already pretty confusing.

[rsrock]

True. To add to this import doesn't import what's exported. Perhaps a rename would be in order.

[ScottPJones]

Seriously, let’s say that @StefanKarpinski ’s public keyword is added, and also a .! syntax to override it (and to address #1974, see @JeffBezanson s comment https://github.com/JuliaLang/julia/issues/1974#issuecomment-71538055), and the default is to act exactly as now, so that absolutely no change to current code is required, and a command-line switch is added for a "enforce accessibility" mode.

What would that mean?
People who want everything to be public don't have to do anything at all, they wouldn't even notice the change.

People who really care about encapsulation, long term reliability, not accidentally breaking user code could use the public keyword.

We _might_ want to require that packages that are _registered_ be able to work with the "enforce accessibility" switch turned on (they can still use .! to get around that, but those can be trivially
searched for if somebody wants to modify something that is non-public, and people using .! would know that it is their responsibility to fix things if things break.

Lint.jl could use the information to give warnings to people using non exported, non public names
(without warning unnecessarily for things that are non exported but meant to be public).

Maybe a special public * syntax could be added also, to be used at the beginning of a module, to simply say that everything not exported is meant to be public.

[MithrandirMiles]

Another idea, get rid of export, and add public, where you can give the fully specified name, if you only want it accessible fully specified, such as people have suggested above, but without the jarring clash of export not really meaning what it does (as Stefan rightly said)

[quinnj]

Things to add to add to enhance namespace management:

• public keyword
• .! syntax
• command-line switch for "enforced accessibility"
• require reigstered packages to enforce accessibility
• public * syntax
• make export a no-op, but work for modules

Seems simple enough :stuck_out_tongue:

But seriously, lot's of good ideas/examples here. I think, given the breadth of the issue here, we may need to take a step back and examine a broader set of changes that wholistically address the issues here; I'm just not sure a syntax change or two will get us there.

[ScottPJones]

With a couple more months under my belt (and some more free time), I'd feel confident to submit a PR to do just that (although I didn't say to make export a no-op, rather to deprecate it and replace it with public, and add the capability to say public Foo.bar, as other people have suggested for export).

I think you are right, and also Stefan's comments about import, some serious thinking about how everything fits together seems necessary, not just isolated syntax changes here and there.

[mbauman]

Adding new reserved keywords is a multi-year project, so let's not blithely jump into that unless we're absolutely certain.

I do think tab completion is a great place to draw a line between public and private. It'd be wonderful if Profile.tabtab only listed the supported API for folks like me that can never remember what things are called, but that can also be done today with an Internals submodule. Heck, that might be the best approach for modules that intend to be used fully-qualified.

[StefanKarpinski]

@quinnj's point is not that we shouldn't do it because it's hard to implement but that it's a bad plan because it's a ton of features and complications. Which I agree with – too much stuff.

[ScottPJones]

I don't get that at all from what @quinnj said: he seems to be advocating for _more_ changes, not less.

think, given the breadth of the issue here, we may need to take a step back and examine a broader set of changes that wholistically address the issues here.

I agree with that, I think it is time that instead of just patching over problems, and people complaining every now and then about import / export etc. being confusing, etc., thought needs to be put into handling a bunch of those issues in a consistent fashion, that people will be happy to live with for the rest of the life of the language.

[rsrock]

Following @quinnj (and now @ScottPJones, as I'm typing this) and thinking about this a bit more, I figured it was time to read the manual before suggesting any solutions. Here's what the docs say about each of these (paraphrasing). All of this is in the "Modules" chapter.

import: 1) control which names from other modules are visible. 2) operates on a single name at a time (?? not sure this is true). 3) allows functions to be extended with new methods.

export: specify which of your names are intended to be public.

using: module will be available for resolving names as needed.

Although I agreed with @StefanKarpinski on one of our proposed changes with export SomeMethods.foo(),

So export would not actually export anything? Confusing

...the actual definition of export seems to allow such use, in the sense of "Export a public interface". So we could stick with export instead of splitting hairs with public and export, in my opinion.

In Python, from foo import * pulls in everything without an underscore into the local namespace, and export / public is not even an option. So we're already more restrictive than they are.

As an aside, @StefanKarpinski also mentions

The business of using import to extend things is already pretty confusing.

Perhaps (at the cost of adding a keyword and code churn): extend? E.g. extend Base.getindex()

[mikewl]

Another question should be: public default or private default. So instead of explicitly marking the public interface, explicitly mark the private portions of the module or type.

As has been mentioned, there should also be a simple way to escape the encapsulation. If I want to do something with the internals I don't want to jump through hoops to please the microwaved sand gods. That being said, the way to escape encapsulation must be distinct enough to be noticeable when bug hunting.

[tknopp]

I also think that the submodule issue should be discussed in relation to the other module discussions (e.g. the import thingy Stefan mentioned)

The origin of this thread initialized by @IainNZ was, however, about plain (non-submodule) functions in Base and I fully agree that that these need no deprecations warnings. Non-exported public submodule functions are a different business and it seems to be appropriate to open a dedicated issue since this has been become to broad.

[tknopp]

Thanks @rsrock for suggestion reading the manual. It says

Within a module, you can control which names from other modules are visible (via importing), and specify which of your names are intended to be public (via exporting)

I think this sentence makes it pretty clear that export==public is the approved rule (with submodules forming an exception).

[ScottPJones]

@tknopp, the problem is, that simple black or white definition is simply not good enough, and doesn't match what people are actually doing, in base or in packages.
Many times you _don't_ want to pollute the namespace by exporting the name, but you _do_ want the qualified name to be accessible, visible to help, etc. (and conversely, there are many times when you make it clear when some name is not meant to be public)
Where I worked, breaking customers code was a big no-no, so it was very important to only make public things that you were sure you wouldn't be changing in the future. The constant breakage that goes on currently with julia might be OK for some people, but will need to stop soon as the language matures.

[tknopp]

The constant breakage that goes on currently with julia ...

Scott, your statement is totally unfair for the people that put a lot of effort into Julia: There is no constant breakage of the public interface (-> export, see the manual) of Julia. Additionally, the stable Julia version (0.3) is rock solid and the minor releases do not break any packages. The development branch is a different thing and the recommendation for the "customer" is to not use it.

[ScottPJones]

It's not an attack, it's just what I've seen since I've started with Julia in April.
As I noticed at JuliaCon, many people need the features that are only available on 0.4, and hence have no other choice.
Also, unless you want a permanent split between 0.3.x and 0.4, a la Python 2.x vs. Python 3.x, then at some point people will have to deal with the major changes between the two.

I'm not saying the changes are a bad thing at all, just that Julia could use some better mechanisms to help people isolate things that are meant to be public (but not necessarily exported, for name pollution reasons), and ones that really are internal, which should be stayed away from unless you want to risk high chances of future breakage.

[KristofferC]

Where I worked, breaking customers code was a big no-no, so it was very important to only make public things that you were sure you wouldn't be changing in the future.

Comparing the development of Julia to the development of some locked down commercial corporate software is in my view not very relevant. I believe most people working with Julia expects (and desire) rapid development with the trade off that things might have to break from time to time.

Also, just because 0.4 is even more awesome than 0.3 and people want to use it, doesn't mean that we can suddenly stop developing it. If you are on development branch then things might break. It is as simple as that. If you need something stable AND the features of 0.4 I guess you have to wait a couple of weeks.

[ScottPJones]

I believe most people working with Julia expects (and desire) rapid development with the trade off that things might have to break from time to time.

I heard quite a bit of grumbling at JuliaCon about breakages. I think the issue is really, what can be done to _minimize_ breakage, while still allowing rapid development of the language.
Compat.jl and the deprecations are good examples of things that do help that goal, and I think that being able to make a clear distinction between what the author believes to be part of the public API or not also would help that goal.

[mbauman]

This is getting all jumbled up with other issues.

I like Iain's initial suggestion — non-exported functions are officially not supported unless otherwise documented — with some of Tony's caveats about avoiding gratuitous breakages. As someone who has added a deprecation for a non-exported method, I'd add that I did so because _in that case_ it was really easy… but I definitely don't want to set a precedent that all internal methods should be deprecated. If we notice that an internal function is getting used often, we should identify why and work on moving it towards a public API.

It'd be very interesting if we could search through packages and identify uses of internal functions. To make that work, though, we do need some mechanism for making a name public but not exported to support things like Profile.print(). Perhaps it's as simple as having documentation in the __META__ dict.

[tknopp]

I don't think this is enough. We need a way to address the submodule case in order to write a function public_interface(m::Module). Since internal functions also can have documentation it is too indirect to go through __META__.

[mbauman]

Since internal functions also can have documentation it is too indirect to go through __META__.

I agree that going through __META__ is a little indirect, but I think that, at least in Base Julia, only public functions should have docstrings. Internal methods can get comments in the code. I've been missing apropos since the Doc switchover, and a reasonable way for it to be re-implemented is to search through all the values of the __META__ dictionaries. If internal methods began showing up there it'd make the situation worse.

[ScottPJones]

I disagree strongly about internal methods not being able to have docstrings. I might have fairly complex internal methods, that I want to have real documentation, and annotate things like what exceptions they throw, etc.

[tknopp]

@mbauman: True. But these private docstrings could be accessible through some other functions. Thats why I think it would be better to have a direct way to define the interface of a module (i.e. export + some clever improvement to let Print.print go through some sort of export). If you have public_interface(m::Module) it could be used in apropos for filtering the actual API.

[ScottPJones]

@tknopp, I think private docstrings should be accessible from help, but maybe by doing something like doing two ?. About export, I agree with Stefan, that the name export doesn't make as much sense anymore, so I would advocate public as a preferred alias, i.e. public Print.print, foobar, blech, exactly as has been suggested for extending export.

[timholy]

@ScottPJones, I suspect you're in the minority about wanting internal methods to have docstrings. That's what code comments are for.

[ScottPJones]

When you are programming / debugging on a large project, you'd like to be able to have help come up, and not have to go digging into the source code all the time. Code comments simply don't cut it.

[toivoh]

I also think that allowing docstrings on internal functions would be nice,
seems strange to go through the trouble of implementing nice documentation
facilities and then restrict them so that they can't be used with all
functions.

[timholy]

And we go around in circles again. There are two camps: people who want to make internal functions easier to use (so they should have docstrings), and people who want to make sure our API is so nice that there's never any need to call an internal function. I'm in the latter camp. The absence of docstrings for internal functions is a way of forcing us to get the API right.

Code comments are just fine for documenting something whose usage you _actively_ want to discourage.

[tknopp]

Tim you are not alone. It would be pretty strange if our public API documentation would include documentation of internal functions. I don't care if there is some help_internal function though.

[ScottPJones]

@timholy You are forgetting the people who have to develop the software that _uses_ those internal functions. I'm not trying to make internal functions easier to use _outside_ of the module they are in, rather make it so it is clear they _are_ internal, which doesn't happen now, but also, to make it easier for the people maintaining the package or module, or for the poor programmer that gets a backtrace with that internal function, who'd like to quickly find out what it is supposed to be doing. That's why I thought a ?? at the REPL, which could call a help_internal, would be nice.

[mbauman]

We have @less, @edit, etc. We're going down yet another rabbit hole here. I only brought up docstrings because they seem to be an easy and straightforward way to define the public interface by convention right now, without adding any new language features.

[StefanKarpinski]

This discussion is getting nowhere quickly and isn't going to need to be settled for several months. How about we all let this sink in and simmer for a bit and revisit the issue when it's more pertinent.

[rsrock]

@StefanKarpinski I just finished some doc edits (two minutes ago) that attempt to lay out how things currently work. It could either help frame the discussion, or it could get the pot boiling again.

Shall I hold off on the PR?

[StefanKarpinski]

No, go for it – improvements to the documentation of how things currently work are always good.

[rsrock]

Ok, it's in #12696

[aviks]

That is a good advice @StefanKarpinski . But allow me to make a _meta_ point

The issues raised here go the heart of what kind of a language Julia is. Is it a permissive language or a restrictive one. Is it a language that makes many weird things possible, or a language that prevents many bad things happening...etc.. etc... (There is no value judgement in either of these, nor are they binary.. but decisions like this go a long way to define the feel of a language)

For things like this, I think any movement towards "design-by-committee" are dangerous. Which does not prevent me from expressing an opinion, not should it, for anybody. I'm sure they all are useful input. But we should all be wary of all our wishes coming true.

[tknopp]

@aviks: These are absolutely great points and if one follows the issue tracker and the mailing list you can see that the core maintainers see Julia as a permissive language. One issue is that there are sometimes newcomers having a different background and a very strong opinion which is expressed in verbose form so that one has to read these issues carefully to understand which is the "spirit" of Julia.

[ScottPJones]

@tknopp One can still have the language be permissive, and at the same time add support for programmers to express their intentions, and help make things more reliable.
As @aviks said, it is not a binary issue - the language _can_ be both permissive, and yet have mechanisms that be used to help prevent bad things from happening.

[tpapp]

In the meantime, until this issue is fixed, maybe it would make sense to standardize on some sort of style convention for functions which are not part of an API, which could even go in the style guide.

A suggestion from existing practice: many libraries use a _ prefix for "internal" functions, and for the packages I have installed,

$ grep -ro 'function _' ~/.julia/v0.4 | wc -l
752

If eg .! or some similar syntax is introduced, uses of SomeModule._hey_you_shouldnt_use_this() can easily be found and replaced. If this issue is never resolved, at least we have a convention that is orthogonal to docstrings and other issues.

[IainNZ]

Tidying up old issues/PRs.

[tpapp]

@IainNZ: how was this issue resolved?

[IainNZ]

I believe indirectly its been resolved by there being a 1.0 soon, which has (or will have) guarantees. There wasn't much value to be had from this years-old discussion.