Julia: module feature: allow access for public exports only

[DemiMarie]

Currently, when importing libraries in Julia, one cannot selectively import only the exported symbols of a package without bringing those symbols into the global scope. This is often not wanted, and indeed the equivalent in C++ (using namespace) is often recommended against.

[vtjnash]

use import pkg

[DemiMarie]

@vtjnash That doesn't do what I want. Consider these two files in the same directory:

Test1.jl:

module Test1
export foo
foo() = 1
bar() = 2
end

Test2.jl:

push!(LOAD_PATH, ".")
import Test1
print(Test1.foo())
print(Test1.bar())

This outputs 2. I am looking for something that I can use instead of import, such that:

• the first print expression will succeed (because module Test1 will have been brought into scope)
• the second print expression will fail (because bar() has not been exported).

That is, I want:

• qualified access to the module's public API
• no access to the module's implementation details.

[MichaelHatherly]

https://github.com/MichaelHatherly/PrivateModules.jl provides a @private macro that does this kind of thing:

@private module Test1
export foo
foo() = 1
bar() = 2
end

julia> import Test1

julia> print(Test1.foo())
1
julia> print(Test1.bar())
ERROR: UndefVarError: bar not defined

[StefanKarpinski]

There was some recent discussion about how the state of issues around modules and namespacing didn't reflect the general interest in improving them. I've reopened this and added a "modules" label for such issues.

[JeffreySarnoff]

It would be nice to import two or more modules that share one or more exported symbols and designate which module is to be used when all or some of the symbol[s] are used without a module name prefixed (and then when used, not to have warnings about overwriting the so-dominated symbols generated).

module  ModuleA
     export honey
     function honey() return "ModuleA's honey" end
end
module ModuleB
     export honey
     function honey() return "ModuleB's honey" end
end

# this almost does it -- but there is no way to discriminate some from all
using ModuleB
importall ModuleA
honey() # "ModuleA's honey"

[JeffBezanson]

@JeffreySarnoff that's a good point, but to me there is a world of difference between qualified and unqualified names. A lot of care _absolutely_ must go into how foo by itself is resolved.

@DemiMarie I don't believe the analogy to using namespace is correct; import pkg does not bring all of its symbols into the global namespace, only the symbol pkg. I can certainly appreciate why people don't want lots of names dumped into their namespace.

But by the time you've written Test1.bar, why not just give the value of, well, Test1.bar? In this case I subscribe to the "we're all consenting adults here" attitude. To restrict this, we'd be going to a lot of trouble just to slap you on the wrist and prevent you from accessing something. For example, how would you debug the module, e.g. test Test1.bar interactively? I'm against complexity that only serves to prevent me from doing things.

[DemiMarie]

The problem is that "import" treats exported and non-exported symbols
identically. If I type import foo; foo.bar, I don't know if bar is
part of the public interface of foo or just an implementation detail.

On Aug 12, 2016 1:58 PM, "Jeff Bezanson" [email protected] wrote:

@JeffreySarnoff https://github.com/JeffreySarnoff that's a good point,
but to me there is a world of difference between qualified and unqualified
names. A lot of care _absolutely_ must go into how foo by itself is
resolved.

@DemiMarie https://github.com/DemiMarie I don't believe the analogy to using
namespace is correct; import pkg does not bring all of its symbols into
the global namespace, only the symbol pkg. I can certainly appreciate why
people don't want lots of names dumped into their namespace.

But by the time you've written Test1.bar, why not just give the value of,
well, Test1.bar? In this case I subscribe to the "we're all consenting
adults here" attitude. To restrict this, we'd be going to a lot of trouble
just to slap you on the wrist and prevent you from accessing something. For
example, how would you debug the module, e.g. test Test1.bar
interactively? I'm against complexity that only serves to prevent me from
doing things.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/JuliaLang/julia/issues/17948#issuecomment-239516450,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGGWB3jv_jc33unmLYtoLlsFlHR-66Q3ks5qfLRFgaJpZM4JhgyD
.

[JeffBezanson]

I agree it is good to know which is which. names(foo) will tell you just the exported names in a module. Currently tab completion shows all names, but I think it would be good to have a setting to only tab-complete exported names. Documentation also helps; it would be good for ?pkg to show a list of exported names, especially if the package lacks documentation.

[DemiMarie]

I would also like an easy way for this to be enforced at runtime.

On Aug 12, 2016 4:24 PM, "Jeff Bezanson" [email protected] wrote:

I agree it is good to know which is which. names(foo) will tell you just
the exported names in a module. Currently tab completion shows all names,
but I think it would be good to have a setting to only tab-complete
exported names. Documentation also helps; it would be good for ?pkg to
show a list of exported names, especially if the package lacks
documentation.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/JuliaLang/julia/issues/17948#issuecomment-239550844,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGGWB8BShMJoQeOSF9MZW6SXNbhyaX1zks5qfNZqgaJpZM4JhgyD
.

[JeffreySarnoff]

@JeffBezanson Yes, I agree. My perspective was forward -- resolving APIs, however they may become done. It seems there needs to be an active way to assert dominance (in the form of how unqualified names are resolved when there is more than one qualified form available). Maybe there is another way to express this API is a refinement of the that one. Maybe there is a better way -- this one I noticed.

[tlnagy]

Documentation also helps; it would be good for ?pkg to show a list of exported names, especially if the package lacks documentation.

@JeffBezanson That would awesome! My preference is that we should hand-hold users more on the documentation side than on the runtime side. I like that Julia separates nonexported functions from exported ones, but doesn't hide them completely.

[StefanKarpinski]

It's entirely possible that a post-1.0 version of Julia will add access controls that allow some degree of enforced encapsulation, but preventing people from doing things is not a high priority for 1.0.

[DemiMarie]

I am not particularly interested in mandatory encapsulation. I am more interested in making it easier to avoid using private functions, without polluting the global namespace.

[StefanKarpinski]

I'm confused by your statement – how is that interest not satisfied currently?

[DemiMarie]

Currently, there is no way to get access to the public symbols of a module
without either injecting them into your scope or making them appear
indistinguishable (as far as the consuming code appears) from symbols that
are merely implementation details. So when you write import X, X.a
will get a whether it is public or not. But if you write using X, then
the only difference between exported and non-exported symbols is that a,
if exported, is injected into your global scope.

I don't need for the abstraction barrier to be airtight. But I do want it
to be obvious when I am using something I probably have no business
accessing.

Perhaps this can be done by convention: have a convention that private
symbols are in inner modules whose name starts with an underscore, say.

[helgee]

While I like the Python convention of prefixing private members of classes and modules with an underscore due to its simplicity it is not very aesthetically pleasing ;-)

What is missing is a mechanism that allows one to import a single name but only if it is part of the public interface of the module, e.g. like shown below:

module Encapsulate

export iampublic

iampublic = 1
iamprivate = 0
end

# `using` always respects the visibility settings of the module

# Imports all exported symbols into current scope
using Encapsulate
# Imports only `iampublic`
using Encapsulate: iampublic
# Throws an error or a warning
using Encapsulate: iamprivate

# `import` does not care about the visibility settings of the module.
# We are all consenting adults here...

# Imports the `Encapsulate` module
import Encapsulate
# Imports only `iampublic`
import Encapsulate: iampublic
# Imports only `iamprivate` although it is not exported 
import Encapsulate: iamprivate

[tkelman]

I have thought it felt a little backwards that using Mod: foo is more conservative than import Mod: foo, since the latter allows extension. (Compared to just using Mod which is less conservative than import Mod due to bringing in the exports.) More behavioral distinction between the two might make it a little more obvious when to use one vs the other.

[helgee]

import is also less conservative than importall which does not feel right.

[DemiMarie]

One option would be a way to declare some symbols as private. Getting
access to them would require reflection (say get_module_var(SomeModule, :somePrivateField)

[toivoh]

I like the idea of restricting using to only import exported symbols.
When it comes to qualified access to members of a module: if we ever get getfield overloading then it could be natural to let MyModule.iampublic work, but require MyModule..iamprivate with a ...

[rapus95]

If access restriction is ever being added, would it be out of scope to add a more complex rights management in order to disable the IO part (and maybe others independently) of julia? (in order to make it a fully virtual runtime w/o any reality interaction and thus having a "save" environment)

Concrete reason would be that i'd really love to be able to "outsource" parts of a program that i've written so that a user can manipulate it (crazy example: i've written a multiplayer game and want to provide access to some underlying methods in order to change behaviour of some things in a pretty complex way, like manipulating the weather based on ingame events)
But i obviously neither want any user to be able to shutdown nor misuse the server (for like running a DOS or sending viruses)
I know there a a lot of other ways (depending on the situation/use case) to realize such a game, like writing a DSL, doing a whole AST check for "invalid" calls (that would be my solution if i had to solve it right now, similar to Reflection/SecurityManagers in Java) or doing the "calculations" clientside and only communicate by keywords (thus simply restricting what is considered senseful on the server)
But i'm always happy to be lead to even more creative/prettier/faster or otherwise better solutions.

That's just my 2 cents for the restriction idea. As i said, it's not about time but about scope (are there chances that julia will ever support such a use case)

[JeffBezanson]

A more thorough capability or access control system is definitely a bigger problem. A library for launching missiles will export its launchMissiles function since that's the entry point, but that doesn't mean everybody is allowed to launch missiles. This issue is really only about convenience, making it easier to avoid depending on private functions. For those purposes it's ok if there are workarounds like eval, but for real security the goal is to make it as close to impossible to launch the missiles as you can.

[StefanKarpinski]

Agreed. For security, you really want a microservices approach with an audited / verified kernel.

[tkelman]

o/t, but cloudabi has some really neat ideas with capability based security there

[JeffreySarnoff]

cloudabi is Windows averse:

_Nuxi CloudABI is a new runtime environment for Unix-like systems_

It is nice of them to allow others to use their relatively more secure versions of the cloudabi, 'nix zeitgiest apps and these other commonly used packages:

autoconf, automake, bash, bison, boost, buddy, bzip2, c-runtime, cairo, cairomm, cloudabi-reexec, cloudabi, cloudlibc, cmake, compiler-rt, coreutils, curl, cxx-runtime, diffutils, expat, findutils, flac, flex, freetype, fribidi, gawk, gettext, giflib, glib, gmpUpgr, grep, help2man, icu4c, jasper, jpeg, json-c, lcms2, libarchive, libatomic-ops, libcroco, libcxx, libcxxabi, libebml, libevent, libexif, libffi, libgcrypt, libgpg-error, libid3tag, libksba, libmad, libmatroska, libmngRena, libogg, libpng, libressl, libsamplerate, libsigcxx, libsndfile, libsodium, libtasn1, libtheora, libtomcrypt, libtomfloat, libtommath, libtompoly, libtool, libunwind, libvorbis, libwebp, libxml2, libxslt, libxspf, llvm, lua, lz4, lzo, m4, make, memcached, mpfr, nettle, ninja, nspr, openjpeg, opus, pcre, pcre2, picosat, pixman, pkgconf, python, qpdf, re2, sed, snappy, speex, speexdsp, taglib, texinfo, tiff, tomsfastmath, uriparser, x265, xz, yaml, zlib

[cstjean]

I don't need for the abstraction barrier to be airtight. But I do want it to be obvious when I am using something I probably have no business accessing.

In Common Lisp, PACKAGENAME:symbol can only access exported symbols, while PACKAGENAME::symbol is for all symbols. It worked well. Too bad Module..symbol already has a meaning.

[StefanKarpinski]

Module..symbol could be reclaimed. There's also Module$symbol which is already deprecated in 0.6 and will be fully available in 1.0.

[mauro3]

As far as I can tell, this is a dup of https://github.com/JuliaLang/julia/issues/12069

[KristofferC]

Yes, looks very much like a dup.