Chapel: Private use seems to hide methods

[bradcray]

Summary of Problem

The following code demonstrates that a private use can hide a method call, which seems incorrect to me. My intuition is that if I have an instance of a type, I should be able to call public methods on that instance regardless of what modules are used within my current scope.

Steps to Reproduce

Source Code:

For example, for this code, I'd expect to be able to call x.foo() from module O in this example:

module M {
  class C {
    proc foo() {
      writeln("In C.foo()!");
    }
  }
}

module N {
  private use M;

  proc bar() {
    return new C();
  }
}

module O {
  private use N;

  proc main() {
    var x = bar();
    writeln(x.type:string);
    x.foo();
  }
}

But instead, I get:

20: In function 'main':
23: error: unresolved call 'owned C.foo()'
23: note: because no functions named foo found in scope

Compile command:
chpl foo.chpl

Configuration Information

• Output of chpl --version: 1.20

[bradcray]

@mppf, @lydia-duncan: I'll be curious whether you agree that this is a bug. What I'm finding in practice is that code deep within the array/domain code ceases to be able to find routines in the BlockDist module along certain code paths since it is not used by default along private use chains. I.e., I can't do everything with my block-distributed array that I'd expect to be able to because its methods are disabled since I can't see the BlockDist module via a direct public use.

[bradcray]

@chapel-lang/perf-team: This is what was thwarting me heading into the meeting today. Michael was correct that the performance slip was the result of a canResolve() returning true where it used to return false.

[lydia-duncan]

I keep implementing that XD

Obviously my instinct is that it shouldn't be allowed since you're returning something you got from a private use. However, I think that conflicts with Bryant's desire for reexporting things and hiding the hierarchy, so I'll muse on this a bit more and get back to you

[bradcray]

my instinct is that it shouldn't be allowed since you're returning something you got from a private use.

Didn't we have a near equivalent case to this that was related to private declarations within a public module rather than private uses? (where I think we eventually convinced you to get over that instinct?) I can't quite come up with it quickly...

[mppf]

@bradcray - I'm not sure I have an opinion about this yet - but I wanted to ask - would your answer change for a secondary method? E.g.

module M {
  class C { }
  proc C.foo() {
    writeln("In C.foo()!");
  }
  proc baz(arg: C) { }
}

module N {
  private use M;

  proc bar() {
    return new C();
  }
}

module O {
  private use N;

  proc main() {
    var x = bar();
    writeln(x.type:string);
    x.foo(); // should this one resolve?
    baz(x); // this one shouldn't, right?
  }
}

Either way - the free function baz wouldn't be available in O, right?

Anyway I can argue both sides of this.

Rationale for having the methods be available: As in this case, a function might return something without its type actually being visible. In that case one wouldn't be able to use the type unless methods are available.

Rationale for not having the methods not be available: Methods are just an alternative way of writing a function, and so they shouldn't have different visibility rules than free functions. In particular, if the methods are always visible, then ambiguity on methods cannot be resolved with only/except/as in use statements and also it is unclear how some methods could be private but not others. Instead, two things need to happen. 1) The compiler needs to warn in the case that a public function is returning a type that is not publicly visible; 2) the author of N needs to publicly export C (and also its methods, if exporting the type C did not already do that).

To my mind, the 2nd of these has a more reasonable justification, FWIW.

[lydia-duncan]

Cases that are similar:
(Note: I'm talking about classes, but I think the same applies to records and type declarations)

Case 1: I got the type through explicitly naming a function that returned an instance

module A {
   class Foo { ... }
   proc returnsFoo(): Foo { ... }
}
module B {
  proc main() {
     var x = A.returnsFoo();
     x.someMethod();
  }
}

Behavior: compilation error because we haven't used A. This used to be the same as 1b before we only allowed access of modules when a use of them exists.

Case 1b: I got the type through explicitly naming a function that returned an instance (and there was a limited use)

module A {
   class Foo { ... }
   proc returnsFoo(): Foo { ... }
}
module B {
  use A only; // or use A except *;
  // or possibly worse: use A except Foo;
  proc main() {
     var x = A.returnsFoo();
     x.someMethod();
  }
}

Behavior: compiles and runs
We had to add explicit handling to allow this to work - this is why I said "I keep implementing this", and I think the case Brad is thinking of. The original implementation did not allow that behavior and we ultimately decided that was the wrong choice. I think consistency would be helpful between 1b and the case in this issue, regardless of whether we follow this strategy or the one I mentally default to.

Note that if we change our minds on this case and no longer want it to work, I would like to also have a warning/error when a use statement excludes a type that is involved in the signature of an included function, regardless of if it is in the argument types or in the return type.

Case 2 (future looking): The class itself is declared private

module A {
   private class Foo { ... }
   proc returnsFoo(): Foo { ... }
}
module B {
  use A;
  // or use A only;
  // or use A except *;
  // this one will not work since you can't explicitly name private symbols in limitation clauses:
  // use A except Foo;
  proc main() {
     var x = A.returnsFoo();
     x.someMethod();
  }
}

I would argue that this should not work. If you can't name the type in the limitation clause, I don't think you should be able to use it in any form. Additionally, I would like public functions that require private types as arguments to generate an error or at least a warning, and feel similarly about public functions that declare their return type to be a private type.

Case 3 (also future looking): The class is public, but the method is private

module A {
   class Foo {
       ...
       private proc someMethod() { ... }
   }
   proc returnsFoo(): Foo { ... }
}
module B {
  use A;
  // or use A only;
  // or use A except *;
  // or possibly worse: use A except Foo;
  proc main() {
     var x = A.returnsFoo();
     x.someMethod();
  }
}

This one gets into the weeds of class method and field privacy, which I think should mostly stay on #6067

[lydia-duncan]

I should also add that there are ways to re-export the symbol and its methods/fields if we don't adjust use statements - the module with the private use statement could include public wrapper functions for the type's methods and fields and a public type declaration that can be used instead

[bradcray]

would your answer change for a secondary method?

For the case you showed, my answer wouldn't change / I'd give the same answer. I think of secondary methods as largely being a syntactic variant of primary methods and don't think (?) we treat them differently in practice, nor do I think we should. If the secondary method was defined in a different module than the class itself, then I think it's a trickier question (but not the one I'm worried about here).

Case 1b ... I think the case Brad is thinking of.

Yep, that's the one (no wonder I couldn't think of it... it didn't relate to privacy). I view this case as pretty symmetric to the case here: In both cases, I somehow legally got an instance of a type, and I think I should be able to call methods on that type. If the type itself were private, then I agree with you that it's arguably not appropriate for a public function to return instances of it (but that's not the case I'm worried about here).

I should also add that there are ways to re-export the symbol and its methods/fields if we don't adjust use statements - the module with the private use statement could include public wrapper functions for the type's methods and fields and a public type declaration that can be used instead

I'm not convinced that this is practical or would even necessarily handle all cases. Take the following motivating example (which led to this issue): The file ChapelArray.chpl implements the user-facing methods on Chapel's arrays and domains; similarly, the files ArrayView*.chpl implement various other user-facing methods such as slicing, reindexing, etc. The goal of both files is to permit various implementations of arrays and domains (including user-defined cases) to be plugged in, supporting an arbitrary number of domain maps. But if the ChapelArray and ArrayView files can't see or call a method on the underlying array and domain classes without explicitly useing the BlockDist.chpl or MyUserDefinedDomainMap.chpl modules that define them, then the plug-in abstraction breaks down (they would have to know all possible domain maps, including user defined ones, to work). I think the reason we largely get away with this today is because we depend on abstract base classes to say what methods are available and dynamically dispatch. But this breaks down for methods that can't be inherited or dynamically dispatched (like methods returning params or types). I don't see a way that re-exporting would help with this case.

But more generally, it feels surprising to me to have a legally obtained instance of an object that you can't call methods on (since methods feel part and parcel of what an object carries around with it).

[e-kayrakli]

Just as an additional datapoint, a variation of this I bumped into was about inheritance:

// ParentClassModule.chpl:
module ParentClassModule {
  class Parent {
    proc foo() {}
  }
}
// ChildClassModule.chpl:
module ChildClassModule {
  private use ParentClassModule;
  class Child: Parent { }
}
// main.chpl
use ChildClassModule;
var c = new Child();
c.foo(); // should this call resolve?

I expected c.foo() to resolve. While working on #14731, I wanted to make this use private, too, but blocked by this.

FWIW, the case in the original post seems a bit more controversial to me, but @bradcray's following statement makes sense to me:

But more generally, it feels surprising to me to have a legally obtained instance of an object that you can't call methods on (since methods feel part and parcel of what an object carries around with it).

[vasslitvinov]

I would like Chapel to have consistent module-based privacy. For example, things declared private are not visible outside the module -- and so are not accessible. Ditto things in a module when that module is not appropriately use-d.

The advantage of applying this principle to the above examples is uniformity in the language, which IMO outweighs the potential benefits of doing the contrary. A couple of specific points:

• Brad's array/BlockDist case - I am unconvinced that the generic array code should invoke BlockDist-specific functionality. So the compiler correctly prevents this from happening.

• If a function returns a type that is not visible - it makes sense to me that I can't do anything with it. In my mind it is akin to the class BlockStmt; declaration in C++: I can pass it around, I can't invoke any methods on it.

I do not like class-based privacy. That's a separate topic.

[mppf]

From @lydia-duncan's first comment:

I keep implementing that XD

If we have this working in some cases and not others, and we decide to turn it off altogether, it would be a breaking change, wouldn't it? IIRC we had some special rules for operators here.

To make an informed decision on the language design direction here, I think we need to consider:

• methods on records
• methods on classes
  
  
  
  • how does the choice interact with inheritance (like in @e-kayrakli's example above)
  
  
  • will virtual dispatch make this callable even if they are not visible in some cases?
  
  
• are there any other languages with module-based visibility we can draw upon?

I also wonder if some kind of "re-exporting" idea would address some of these examples (where the involvement of a particular implementation module should not be visible, but where one would want that implementation module to contribute methods to certain types...).

I think that there's a philosophical point here too. Say e.g. module M has a class C with a method called method. What is the point of private in modules?

• is it to control namespace creep? (In that case maybe it is fine for methods to be available, e.g. C.method)
• or, is it to control which methods and functions can be called outside the module? (In which case presumably we wouldn't want the methods to be available? And presumably we wouldn't allow private methods to override public ones?)

Maybe thinking of C kindof like a submodule would help?

private use M;
public use M.C; // Make C available and re-export its methods

In the above, since C is publicly available, so to should be methods within it. We had to opt in to making it public, but we can do that while hiding other things in the module implementing it.

Lastly, in @bradcray's original example the core of the issue is here:

  private use M;

  proc bar() {
    return new M.C();
  }

My current reaction is that this code is unreasonable, because bar is public and yet if fully typed it isproc bar(): C and C isn't public. I'm thinking that public functions should have their function signature consist only of public types. (Otherwise... from a certain reductive point of view, how would you even be able to call such a method? E.g. imagine that in a separate compilation world, all of the private types aren't even available outside of the module, so the compiler couldn't even know how big they are etc to make stack variables).

[mppf]

I think consistency would be helpful between 1b and the case in this issue

I've been mulling this one over a bit and I'm currently more inclined to agree with what I understand @bradcray and @lydia-duncan were saying above.

I've also been thinking about my previous thought:

I think that there's a philosophical point here too. Say e.g. module M has a class C with a method called method. What is the point of private in modules?

• is it to control namespace creep? (In that case maybe it is fine for methods to be available, e.g. C.method)
• or, is it to control which methods and functions can be called outside the module? (In which case presumably we wouldn't want the methods to be available? And presumably we wouldn't allow private methods to override public ones?)

I think that the purpose of module operations (use, import, and re-exporting in whatever form it takes) is to control namespaces. As such they do not need to have any impact at all on nested namespaces. For example, if I have a module M with a (public) submodule N - if I public use M or private use M then that shouldn't affect the visibility of symbols inside of N - just whether or not N itself is visible. I think there's a corresponding argument for classes and methods. (And if none of that made any sense... I can make a few code examples).

I think part of the issue is that private use M might be interpreted as doing two different things:

1. Making the fact that M is used in this module hidden for namespace reasons (i.e. simply to control namespace creep)
2. Making the symbols available in M not callable outside of this module because they are now private. This is more of a re-export operation (and see the re-export issue #13979).

Now, I don't think that 2 is reasonable at all. I don't think re-exporting something to make something more private makes sense. As a result, I think that the private use should allow methods in the types available to be called outside of the module (at least, unless those methods themselves are private).

I have not changed my viewpoint on this part of my last comment, however:

My current reaction is that this code is unreasonable, because bar is public and yet if fully typed it isproc bar(): C and C isn't public. I'm thinking that public functions should have their function signature consist only of public types.

[mppf]

Actually the two-interpretations-of-private idea can also apply to this thought:

My current reaction is that this code is unreasonable, because bar is public and yet if fully typed it isproc bar(): C and C isn't public. I'm thinking that public functions should have their function signature consist only of public types.

If bar wants to return C, and C is public but not currently visible, who am I to stop it? It seems that private use might make C not be visible, but C being private would be based on whether or not it was declared public or private in its original module (#6067).

So if I continue with the interpretation that private use is simply about controlling namespace creep - and doesn't really mean the same thing as private would on a declaration - then a function returning C when C itself is not visible is OK as long as C is public at its declaration.

In fact it would be easy to write similar code without using private use at all, e.g.:

module M {
  module N {
    class C { }
    proc makeC() { return new C(); }
  }

  proc bar() { return N.makeC(); }
}

module Test {
  use M only bar;
  bar(); // returns a type that wasn't visible but wasn't private
}

[lydia-duncan]

Resolved by #15003