Chapel: Public and private use statements

[lydia-duncan]

Today, uses of modules spread transitively through use chains, meaning that the following code works:

module A {
  var a = 10;
}
module B {
  use A;

  var b = 12;
}
module C {
  use B;

  var c = a + b; // "a" is visible here because it was visible to module B
}

While this means that Chapel code which relies on groups of modules that work together ultimately need to write fewer explicit use statements, it is easy to see situations where a module writer would like to not give their clients access to the modules they themselves require. To provide this control, we would like to be able to specify a use statement of the form:

private use Foo; // or
public use Bar;

similar to how symbols defined within a module can be declared as public or private. The keywords public and private will not be required, and if omitted will use the default setting.

Should the default visibility of a use statement remain public? We plan on going through our use statements in the library and built-in modules and making a decision based on both that and user input.

The big question though is in regards to our point of instantiation rules. Today, Chapel instantiates generic functions completely at the call site - on the one hand, this allows them to be called on types that were not available from the scope in which the function is defined:

module A {
  proc foo(arg) { ... }
}

module B {
  use A;

  class MyClass { ... }

  ...
  var c = new MyClass();
  foo(c); // Works because of point of instantiation
  ...
}

On the other hand, it allows function calls within the body of the generic function to be hijacked to a more local version at the call site, potentially causing security risks. Our instantiation of generic functions works today in part because of the transitivity of module uses as described above. Changing the behavior of module uses is a good time to revisit our rules about point of instantiation, to ensure that this set up continues to be the right decision going forward.

[mppf]

re public & private use, another way to look at is something like Python's import, where the symbols are available only when prefixed by the module name. Such a thing would break most transitive uses.

re. point of instantiation rules, I wonder if we could prefer the version at the definition point and fall back on a version at the call site if the required symbol is not available at the definition point?

[lydia-duncan]

For the public/private comment, that would be providing the same access as we do today: you can access a symbol in another module with the module name prefix as long as Chapel knows the module exists. If we wanted to tighten up imports like what you are describing, we would probably want to require use statements to make those accesses, and I think we didn't want to do that when we last discussed it

For the point of instantiation rules, that could work. Though maybe the intention is to allow for an overload for the specific local type (to prefer foo(x: bar) at the callsite over foo(x) at the definition point). There's more details to this situation that I haven't been able to dive into yet (though I have 20 tabs open on it in another browser window)

[mppf]

@lydia-duncan - I happened to be poking at a related area in the compiler and I noticed a problem with point-of-instantiation (where it is not implemented correctly). See issue #6252. I'd like to discuss more what (if any) alternatives there are to point-of-instantiation. However I suspect we'll want to have such a discussion is some point-of-instantiation issues (probably not on this issue).

[mppf]

See #6272 adding CHIP 20 about point-of-instantiation.

[bradcray]

@lydia-duncan / @mppf: Given that Bryant (arguably) ran into this as mentioned on issue #13041 (which I've forked off to a standalone issue #13118), I'm wondering this week what it would take to implement private uses. I know the reason that we originally felt we had to make use transitive by default was due to point-of-instantiation challenges or concerns. But I also think that our world has changed a bit w.r.t. point of instantiation since then, so have lost track of whether we believe we have a clear path forward for this item or not. Do either of you have a sense of that?

[lydia-duncan]

Iirc, the path forward is relatively straight-forward, but if we want to flip the switch on whether use statements are public or private by default, there's going to be a fair bit of fallout to dig through.

[bradcray]

I'm not proposing switching the default (at least at this point); just looking to provide the option to have a private use (and probably public use as well, though that would be no different than what we have today).

[BryantLam]

A note of caution as the proposed change would impact the module discussions in https://github.com/chapel-lang/chapel/issues/12923#issuecomment-495632972 and #10946.

public use means something completely different in Rust. In Rust, "pub use" is for re-exporting symbols from one scope to another, which is a handy feature for doing internal refactoring of code without changing the exposed public API (the symbols that are moved can be re-exported to the top package scope from its new location). For example:

// Rust
// I don't think Chapel has a way to do something similar.
mod A {
    pub use self::B::*;
    mod B {
        pub use self::C::*;
        mod C {
            pub fn hello() {
                println!("Hello!");
            }
        }
    }
}

fn main() {
    use A;
    A::hello();
}

I'd personally advocate for default-use-is-private (or import if changing the default on use is too unappealing). Also, I'd want a more powerful wildcard -- it would let us get rid of the standout use Enum behavior..

module A {
  var a: int = 1;
  module B {
    var b: int = 2;
  }

  enum Color {
    red, green, blue
  }
}

//
// Today's Chapel with default-use-is-public
//
{
  use A;
  writeln(a);
  writeln(B.b);
}
{
  use A.Color;
  writeln(red);
  // I understand why this behavior exists, but ...
}
{
  use A only Color as C;
  writeln(C.red); // Better for defensive programmers.
}

//
// Modified Chapel with default-use-is-private
//
{
  use A;
  writeln(A.a);
  writeln(A.B.b);
}
{
  use A.B;
  writeln(B.b);
}
{
  use A.Color;
  writeln(Color.red);
}
{
  use A.Color.*;
  // More wildcard support. Emulates "public" behavior if truly desired.
  writeln(red);
}

[bradcray]

public use means something completely different in Rust.

I wasn't understanding what was completely different between this example and what Chapel does today before transliterating. For those in a similar boat, the answer is that while this would work:

module A {
    use B;
    module B {
        use C;
        module C {
            proc hello() {
                writeln("Hello!");
            }
        }
    }
}

proc main() {
    use A;
    hello();
}

And this would work:

  C.hello();

This would not:

  A.hello();

because we still consider hello() to be in module C even though A has made it available through the transitive/public uses; we don't think of A as defining hello() itself.

[bradcray]

Thinking about the above use-case some more on my way home tonight, it seems to me that forwarding would be the more Chapel-consistent way to have C's symbols be directly available within A:

module A {
    forwarding B /* optionally: only hello */;
    module B {
        forwarding C /* optionally: only hello */;
        module C {
            proc hello() {
                writeln("Hello!");
            }
        }
    }
}

proc main() {
    use A;
    A.hello();
}

[mppf]

Another syntax option would be something like use B.* as . or something along those lines.

[lydia-duncan]

I'd personally advocate for default-use-is-private

I suspect we all feel that way to varying degrees. But the fact that we've been in the other world for so long means that there will be fallout from making the switch. Still, I would strongly advocate for it being a pre-Chapel 2.0 decision.

[lydia-duncan]

I think I prefer @mppf's renaming strategy to expanding the cases where forwarding is used, but I won't lose sleep over it

[bradcray]

I'd personally advocate for default-use-is-private

The proposed import variant (#13119) could be private by default (which is what you proposed as well when reading beyond this excerpt).

[lydia-duncan]

I found some old tests I wrote back when we first thought about this feature, so would like to run the more controversial of them by people before locking a particular behavior in.

Test A (access of secondary method):

definesSecondary.chpl:

module TypeDefiner {
  class Foo {
    var a = 1;

    proc methodB(x: int) {
      return a + x;
    }
  }
}

module SecondaryDefiner {
  use TypeDefiner;

  proc Foo.other() {
    writeln("I'm a secondary on Foo!");
  }
}

module User {
  private use SecondaryDefiner;

  // Ensures that using a module privately doesn't break secondary methods
  // defined in modules which don't contain the original type definition.
  proc main() {
    var foo: Foo = new Foo(3);
    writeln(foo.methodB(2)); // Should be 5
    foo.other();
  }
}

accessSecondary.chpl:

use User; // from definesSecondary.chpl
use TypeDefiner; // same

// Verifies that you cannot make use of secondary methods obtained via
// another module's private use.
var foo: Foo = new Foo(5);
foo.other();

I expect this pair of .chpl files to give an unresolved call error when accessSecondary is the main module, and to work when the first file is compiled on its own. Do folks agree?

Test B (access of type defined in a privately used module):

includesType.chpl:

module Inner {
  class Foo {
    var a = 1;

    proc methodB(x: int) {
      return a + x;
    }
  }

  proc Foo.other() {
    writeln("I'm a secondary on Foo!");
  }
}

module User {
  private use Inner;

  // Verifies that making a use private doesn't interfere with the availability
  // of types defined in that use to the scope which uses it.
  proc main() {
    var foo = new Foo(11);
    writeln(foo.methodB(4));
    foo.other();
  }
}

accessType.chpl:

use User; // from includesType.chpl

// Verifies that you cannot make use of types obtained via another module's
// private use.
var foo: Foo = new Foo();
writeln(foo.a);

I expect this pair of tests to give an error finding Foo when accessType is the main module, and to work when the first file is compiled on its own. Do folks agree?

[mppf]

Test A: definesSecondary.chpl seems reasonable when compiled on its own. accessSecondary.chpl looks like it has errors any time it's compiled (main module or not) because foo.other() will not resolve.

Test B: Similarly, includesType.chpl seems reasonable when compiled on its own. accessType.chpl has an error because Foo is not a visible symbol in it (main module or not).

[BryantLam]

https://github.com/chapel-lang/chapel/issues/6093#issuecomment-501474275 -- Is private use equivalent to use only in this context? For definesSecondary.chpl (TIO) and includesType.chpl (TIO), substituting use only doesn't have either of the "first code compiled by itself" actually compile because it complains about Foo visibility. I expect this behavior regardless of the secondary-methods question, so I wouldn't expect any of the four codes to compile because Foo wouldn't be found in all of them except accessTypes.chpl. For accessTypes.chpl, assuming definesSecondary.chpl worked, I agree that foo.other() shouldn't resolve.

If intended, as a user, it's somewhat baffling that doing private use still allows me to initialize with Foo rather than Inner.Foo in includesType.chpl. The same idiom is in definesSecondary.chpl too, but I admit I'm confused with how a transitive public-use-inside-private-use can cause a public use that I'm not sure how to reason about which symbols are added to the current scope's namespace despite the supposed protection afforded by private use.

What does private use do?

[mppf]

Is private use equivalent to use only

No, private use just means that the symbols now available from use are private to the module using them. private use A; still makes all the (public) symbols in A visible in the module doing the using, so it can do e.g. someFunctionInA().

Not importing all of the symbols and moving to a default supporting things like use Inner; .. Inner.Foo is another issue.

[bradcray]

Not importing all of the symbols and moving to a default supporting things like use Inner; .. Inner.Foo is another issue.

(specifically issue #13119).

[lydia-duncan]

@BryantLam - the important thing about a private use is its impact on clients of the module with the use statement. It means that you can bring the symbols of A into your module's scope for your convenience without worrying about their impact on your users.

[lydia-duncan]

That's why the distinction between accessType.chpl and includesType.chpl is important - we expect private use to impact accessType as a client of the module in includesType with the use statement, but not to impact includesType itself.