Chapel: how are type methods inherited?

[vasslitvinov]

This language-design issue raises these related questions:

(A) Should a type method defined on a parent class be invocable on its child class?

(B) Should a child class be allowed to override a type method on its parent class?

(C) If yes on A and B, should a type method in the child class be required or allowed or disallowed to have an override annotation?

(D) If yes on A and B, what type should be this inside a type method defined on a parent class and invoked on a child class?

C++, Swift, and present-day Chapel answer YES to (A) and (B).

Swift requires override, which I am inferring from this page, where it says:

A subclass can provide its own custom implementation of [...] type method [...] that it would otherwise inherit from a superclass. This is known as overriding. [...] any overrides without the override keyword are diagnosed as an error [...].

Tests: there are tests/futures in test/functions/typeMethods

[LouisJenkinsCS]

(A) Should a type method defined on a parent class be invocable on its child class?

Yes.

(B) Should a child class be allowed to override a type method on its parent class?

Yes.

(C) If yes on A and B, should a type method in the child class be required or allowed or disallowed to have an override annotation?

Required.

(D) If yes on A and B, what type should be this inside a type method defined on a parent class and invoked on a child class?

If the child class does not override it should be the parent's type. If it does override it should be the child type.

[mppf]

I think the main challenge in this case is terminology - is it overriding or overloading?

From the compiler's viewpoint, it is overloading, but I can see arguments that overriding is how users will think of it. However these type methods never dynamically dispatch, so are fundamentally different from the other situation in which the override keyword is used. That is the reason this is not obvious.

[lydia-duncan]

I agree with Louis on the answers to A-C. I'm uncertain about D.

[lydia-duncan]

I think about this as more to do with the relationship between the parent and the child types rather than specific to type methods. I think if there's something that is true for the parent type, or a feature that is inherent to the parent type, the compiler should assume that it is applicable to child types and should allow users to specify when this is not the case.

The dynamic dispatch question is interesting. I would need to think more about its implications.

[bradcray]

From the compiler's viewpoint, it is overloading, but I can see arguments that overriding is how users will think of it. However these type methods never dynamically dispatch, so are fundamentally different from the other situation in which the override keyword is used. That is the reason this is not obvious.

Are type methods subject to the same type of hijacking scenarios that caused us to introduce the override keyword?

[mppf]

Are type methods subject to the same type of hijacking scenarios that caused us to introduce the override keyword?

I think so. At least https://github.com/chapel-lang/chapel/blob/master/doc/rst/developer/chips/20.rst#unexpected-base-class-additions could be written with type methods.

[BryantLam]

I agree with Louis and Lydia on Questions A-C.

Question D is a bit hard to understand the subtleties. What are the options being considered and what are their tradeoffs / what's the impact of choosing one option versus another? Are there use cases to motivate a choice?

For example, Rust-style object-construction static functions (e.g., Animal::new() returns Self) could be _imitated_ if the typed this always referred to the child type:

// If typed `this` always refers to child type, this code would be valid.
// Today, DO NOT USE THIS CODE. It compiles, but prints: _owned(Animal)
class Animal {
  proc type make(): owned this {
    return new owned this();
  }
}
class Sheep: Animal {}

proc main() {
  var sheep = Sheep.make();
  writeln(sheep.type :string);
}

That said, traits in Rust are implemented on the type itself; e.g., Sheep actually has the static type method new(), so this isn't a perfect analogy.

Another view would be to leave it as is today and the typed this should always refer to the parent type if the parent's type method was invoked by a child type.

As far as I can tell, C++ doesn't have this issue because you cannot self-referentially refer to your own type; e.g., CRTP is explicitly typed (no typed this equivalent) unless you do it yourself explicitly. Rust doesn't have this issue because there is no inheritance; only traits and those are implemented on the "child" types.

[mppf]

@vasslitvinov @bradcray - it seems to me that we're considering requiring an override annotation on type methods overriding a parent method.

(C) If yes on A and B, should a type method in the child class be required or allowed or disallowed to have an override annotation?

That seems like something we should try to get in soon, assuming it is not too hard to implement, but I am not certain if we are all in agreement on this direction.

[bradcray]

To me, "required" seems most consistent with other override cases, right? Has anyone argued for having it be allowed or disallowed instead? (I've only re-skimmed the responses today).

[vasslitvinov]

Requiring override makes sense to me.

[e-kayrakli]

There seems to be consensus on A-C and I agree with those directions.

On D: I feel closer to this being Parent's type rather then Child's if the Child doesn't override. However, my arguments are not very concrete:

• I am worried that a child type can use an implementation that is not implemented for it and cause wrong behavior
• This behavior would also be aligned to the current behavior of non-type methods:

class Parent {
  proc foo() {
    writeln("foo says: this is a ", this.type:string);
  }
}

class Child: Parent {
  proc bar() {
    writeln("bar says: this is a ", this.type:string);
  }
}

var c = new Child();
c.foo();  // says Parent
c.bar();  // says Child

FWIW, my particular use case for this is serialization of locale objects. I am playing around with distributed array creation which involves copying locale arrays around. And currently locales are implemented with a class hierarchy. Ideally I'd like to be able to have methods like proc type chpl__deserialize(data) and proc type chpl__serialtype etc on locales.

[vasslitvinov]

We can think of the this formal of a type method as a variation of a general type-intent formal. Here are a couple of possible interpretations:

class C {
  proc type typeMethod() ......
}

// D1: as a typed formal
proc typeMethod(type thisFormal: C) ......

// D2: as if using a where-clause
proc typeMethod(type thisFormal) where thisFormal <= C ......

this will have the parent type under D1 and the child type under D2.

[e-kayrakli]

We can think of the this formal of a type method as a variation of a general type-intent formal. Here are a couple of possible interpretations:

Among these interpretations, D1 is a more intuitive for me.

[mppf]

this will have the parent type under D1 and the child type under D2.

Will it though? I thought that type arguments allowed subtying; and thisFormal: C is of generic type; so it can be instantiated with a child type can it not?

Off-issue, I heard some support for having the type this in the parent class type method, when called from a child class, instantiated with the child type.

[vasslitvinov]

type thisFormal: C is of generic type; so it can be instantiated with a child type

It is legal to pass a child type to thisFormal. Currently, it will still be instantiated with the parent type.

I think we instantiate with the parent type consistently in several situations. When I discovered it first, it surprised me. Now I am ok with it. I do not know any real-life use scenarios that would argue one way or the other.

Choosing the interpretation D1 or D2 (or some other) is still meaningful, regardless of this.

[BryantLam]

The effective behavior of type methods in an inheritance hierarchy should be the same as if they were regular methods. Anything other than similar, consistent behavior is madness because it will be hard to remember.

• Edit: Argh. The code in https://github.com/chapel-lang/chapel/issues/13154#issuecomment-543316582 is a a valid argument for the Parent type. See next post below for counterargument.

type this in the parent class type method, when called from a child class, instantiated with the child type.

This is the behavior I would want, but my only argument is because this code doesn't work how I want it to (#14291):

class BaseError: Error {
    var x: string;

    override proc message() {
        return x;
    }

    proc type make(x: string) {
        return new owned this(x);
    }
}

class SpecificError: BaseError {
    var z = x + x;

    override proc message() {
        return x + z;
    }

    // proc type make(x: string) {
    //     return new owned this(x);
    // }
}

proc main() {
    var x = BaseError.make("base");
    writeln(x);
    var y = SpecificError.make("specific");
    writeln(y);
}

# chpl version 1.21.0 pre-release (cba06531)

# As is, the output is:

BaseError: base
BaseError: specific

# If you include the commented-out block of code and define again
# the same, redundant method, output is what I would want:

BaseError: base
SpecificError: specificspecificspecific

But maybe choosing the child type isn't a great idea if it breaks consistent behavior with regular methods.

To throw more confusion onto the fire:

• The child-type behavior is more aptly relegated to constrained-generic interfaces, so maybe I'm trying to fit a round peg in a square hole. Having it be the child type would certainly be more convenient than doing some kind of CRTP nonsense.
• Arguably, the child type is more useful in many scenarios. I can't think of when I would want the parent type when the parent's type method is called on the child type.
• https://github.com/chapel-lang/chapel/issues/14343#issuecomment-546793763 bends my mind and I don't know how it works, but it is the behavior that I sometimes would have wanted from even the regular virtual-dispatched methods (https://github.com/chapel-lang/chapel/issues/13154#issuecomment-543316582 [TIO]). I don't know the full implications for advocating for the child type in both type methods and regular methods... Probably a bad idea.

[BryantLam]

I added some futures in #14354. I can see now why there are valid arguments for the parent type or the child type. I think the answer should be the child type because of this future.

If (a) the child calls a parent's type method, (b) that type method calls another this.type method, and (c) the child has overridden the second type method, today, that overridden type method won't be called. I argue that this behavior is not the one that the user would intend if these were non-type methods.

To make this more concrete, here's a table:

DIFF OUTPUT / ACTUAL
==== ===============
     Grandparent.foo
     Parent.foo
       Parent.bar
  x  Parent.foo
  x    Parent.bar    <- Should definitely be Child.bar

     EXPECTED
     ========
     Grandparent.foo
     Parent.foo
       Parent.bar
  ~  Child.foo       <- Should instantiate as Child.foo to get towards Child.bar
       Child.bar

 Legend:
  x  Result is different between OUTPUT and EXPECTED
  ~  Result is different between type-method-dispatch and instance-method-dispatch

[BryantLam]

Now, a related question: How does super.<type method> work?

In regular inheritance (instance-method-super.chpl), if a child calls a parent's method that then calls super.method, the super will resolve to the grandparent because the method is defined and called from the parent.

With type methods, the same should be true even though we have chosen to instantiate the type methods onto the child. super.<type method> should still resolve to the grandparent even though the type method is in the child!

This sounds confusing, but because it is a type method that is, for all intents and purposes, exactly the same as the parent's type method that it was instantiated from, there isn't a reason to call the parent's version of the type method from the child's super.<type method>. Doing so would be duplicative work (e.g., reinitialize the same variables with the same values). The exception, of course, is if the parent's type method is overridden in the child.

Here's another table:

DIFF OUTPUT / ACTUAL
==== ===============
     Grandparent
     Parent
       Grandparent
  *  Parent
       Grandparent

     EXPECTED
     ========
     Grandparent
     Parent
       Grandparent
  ~  Child         <- This result is a consequence of type-method-dispatch
       Grandparent <- Parent.foo is not called because it's the same as Child.foo!

 Legend:
  x  This result is different between OUTPUT and EXPECTED
  ~  This result is different between type-method-super and instance-method-super

Edit: Or Chapel could simply not allow super inside a type method for simplicity. This is fine too.

[bradcray]

Vass summarized what some other languages do for (A) and (B), but in a quick scan of this issue, I'm not seeing what they do for (D). Is there a precedent here in C++ or the like that we can learn from?

[BryantLam]

C++ uses the parent type.

The code emits ??? as a placeholder because C++ doesn't have an easy way for classes to self-reflect on its own type. When Child::foo runs, it prints Parent::bar, so ??? == Parent.

By using the parent type, C++ users cannot statically dispatch to Child::bar. Frankly, there's no good / easy way in C++ to emit my expected output.

Rust doesn't have this problem. Rust has no inheritance; only traits. Here is similar code in Rust, but the use of traits makes the result obvious.

Other languages to look at would be Java, some functional languages with inheritance, and possibly Python, though Python is interpreted.

[bradcray]

I'm feeling nervous about using the parent type for the case when a child type calls into an inherited parent class type method. Specifically, I'm thinking of a revised version of the example from Bryant's comment above:

class Animal {
  proc type make(): owned this {
    return new owned this();
  }
}
class Sheep: Animal {
  type parasiteType;
}

proc main() {
  var sheep = Sheep.make();
  writeln(sheep.type :string);
}

Here, having the new owned this(); in make() try to construct a Sheep rather than an Animal will fail because Sheep's default initializer requires a parasiteType argument. This suggests to me that the author of Sheep should be required to add their own override proc make() in order to support making new sheep. The alternative would be to issue errors when the parent type method tries to do things that aren't valid on the child (e.g., "can't find a 0-argument initializer for Sheep") but that strikes me as being more fragile and unpleasant as a result (e.g., "your sub-class broke my nice class hierarchy!").

While this doesn't support the power of dynamic dispatch on type methods that Bryant was requesting for their power / intuitiveness in his follow-up argument, that also seems justifiable to me given that type methods are, by nature, statically dispatched.

I suspect, but am not sure, that this approach also has the advantage of simplicity in implementation.

[bradcray]

Following on from a comment Engin made above, the following example reinforces for me that having the non-overridden parent class type method always consider this to be the parent class's static type is the right thing to do.

class C {
}
class D: C {
}

// These two overloads of foo() are similar to me to defining a type method on C and an overridden
// type method on D where `t` is the stand-in for `this`.
proc foo(type t: C) {
  writeln(t:string);
}

proc foo(type t: D) {
  writeln(t:string);
}

// The following case is like having only the parent class type method
proc bar(type t: C) {
  writeln(t:string);
}

// The following is completely generic, so adapts to whatever static type is passed in.  But it doesn't 
// have an analogue in the type method world because there's no way to leave the receiver without
// an associated type there
proc baz(type t) {
  writeln(t:string);
}


// mix of static and dynamic types
var x: C = new C();
var y: C = new D();
var z: D = new D();

// these print C/D/C/C/D following the static type of the foo() arguments:
foo(C);
foo(D);
foo(x.type);
foo(y.type);
foo(z.type);

// these print C/C/C/C/C following the static type of bar()'s argument
bar(C);
bar(D);
bar(x.type);
bar(y.type);
bar(z.type);

// these print C/D/C/C/D since baz() is generic
baz(C);
baz(D);
baz(x.type);
baz(y.type);
baz(z.type);

[dlongnecke-cray]

If we are aiming for consistency in look and feel with normal methods (as it seems we are, given that we've agreed to use the override keyword), then I think the example from @e-kayrakli https://github.com/chapel-lang/chapel/issues/13154#issuecomment-543316582 illustrates why it is important to use the parent type unless overridden.

I think what @bradcray has written above establishes a strong intuition of how we should expect type methods to behave (it's a good test to add for this feature, as well).

In general, I think this all boils down to @vasslitvinov's example above https://github.com/chapel-lang/chapel/issues/13154#issuecomment-543339298, which says that we can interpret this in one of two ways.

The only way that makes sense, given the static nature of type methods, is:

// D1: as a typed formal
proc typeMethod(type thisFormal: C)

As the other interpretation breaks class encapsulation.

// D2: as if using a where-clause
proc typeMethod(type thisFormal) where thisFormal <= C

A parent class cannot possibly anticipate all the different subtypes of thisFormal, nor should it have to (that would be bad form, and counterproductive to the whole point of inheritance).

As such, I think we can say the following:

• A type method that is inherited but is not overridden uses the _parent_ type for this.
• A type method that is inherited and overridden uses the _child_ type for this.

Brad makes a good point below, which is to remember that type methods are statically dispatched, which means that whether or not the parent or child overload is called depends on the static type of the receiver!

If you still feel really strongly that D2 above is the right way to go, please say so!

For now, I'm going to move forward with this interpretation (no pun intended), which will mainly just involve writing tests (wooo!).

[bradcray]

A type method that is inherited and overridden uses the child type for this.

But just to make sure nothing's lost in translation: If the actual's dynamic type is the child, but its static type is the parent, the type method would call the parent's version of the method (and thus the type of this would be the parent) since type method dispatch is based on static typing.

[dlongnecke-cray]

That's a fair clarification, yeah. No dynamic dispatch is happening.

[dlongnecke-cray]

In the current stage of my implementation, type methods can override and they can also have param or type return types.

Are we OK with letting type methods have param/type return types, but not regular methods?

If we are, then it this point it would seem reasonable to let _all_ methods with param/type return types be overrideable. Otherwise we will emit errors for normal methods (akin to "methods with param/type return types cannot be marked override") that do not occur for type methods.

In my mind, allowing methods with param/type return types to override seems reasonable as long as we recognize the difference between override and virtual dispatch (which it seems like we do, for the most part).

[edit - typos, grammar]

[mppf]

it would seem reasonable to let all methods with param/type return types be overrideable.

Seems reasonable to me - as you said

we recognize the difference between override and virtual dispatch

but there might be some surprise there I havn't thought of.

[bradcray]

it would seem reasonable to let all methods with param/type return types be overrideable.

I agree that this would be reasonable, though it should be clearly documented that because they are evaluated at compile-time, the dispatch will be based solely on the static type of the object, not its dynamic type.