Chapel: lifetime checking: implicit coercions between kinds class pointers

[mppf]

It seems likely that we'll want to allow implicit coercion from an 'owned' to a 'borrow'.

Allowing an implicit coercion from 'borrow' to 'owned' would break the analysis.

Do we want this implicit coercion? Or any others related the effort?

• owned to borrow
• Owned(Child) to be a subtype of Owned(Parent)
  
  
  
  • raw to borrow
  
  

Ended up with: owned, shared, unmanaged can all coerce to borrow. Subtype coercions are also possible.

[psahabu]

There might be cases where the user wants to pass ownership to another scope, which this coercion would preempt. If we go forward with this coercion, then there needs to be a way to express the other pattern.

[mppf]

There might be cases where the user wants to pass ownership to another scope, which this coercion would preempt. If we go forward with this coercion, then there needs to be a way to express the other pattern.

There already is a way to express that. Say you want to pass ownership from one function to another function that it calls. The called function just needs to take in the value to "get" ownership from by in intent (or possibly ref if it wants to control when ownership is taken). Another way to put it is, if Owned is the way we represent an owned class instance, Owned already includes methods to "move" ownership, including the = operator.

[mppf]

If we have

class Parent { }
class Child : Parent { }

We also need Owned(Child) to be a subtype of Owned(Parent)

[psahabu]

The called function just needs to take in the value to "get" ownership from by in intent (or possibly ref if it wants to control when ownership is taken).

I'm good with in intent being the way to pass ownership, with default being coerced into a borrow.

We also need Owned(Child) to be a subtype of Owned(Parent).

I'm pretty sure my programming languages professor literally jumped up and down to emphasize that U being a subtype of T does not imply that W(U) is a subtype of W(T). That said, given that Owned is a meta-construct on ownership rather than a direct modifier of the contained type, this coercion makes sense.

[vasslitvinov]

It makes a good sense to make a (Child) to subtype the same modifier on Parent.

Yes for owned->borrow conversion implicitly.

I am not excited about 'in' intent to mean "pass ownership" implicitly. 'const in' is the default intent and it would be messy to explain that the default intent means one thing and 'const in' another.

The called function just needs to take in the value to "get" ownership from by in intent (or possibly ref if it wants to control when ownership is taken).

How are we going to check the lifetime when a class pointer is passed by ref then the callee takes over the ownership? We'd need to look inside the function, at the very least, which violates "interface invariance".

Also, just like with in intent above, argument intents would be too subtle of a way to distinguish passing ownership from borrowing.

Whenever a borrowed pointer is converted to an owned pointer, it should be a compiler error or, with a cmdline flag saying "soft lifetime checking, please", a compiler warning.

W.r.t. lifetime conversions, I suggest we start out with more restrictions until we have experience and/or user requests.

[mppf]

I am not excited about 'in' intent to mean "pass ownership" implicitly. 'const in' is the default intent and it would be messy to explain that the default intent means one thing and 'const in' another.

I hope that this is stemming from confusion about what I was saying.

const in is not the default intent for a record such as Owned. While we use the term const in to describe the default intent for a class instance, I think that's more terminology than anything else; we could also say "it's a reference to the object's fields".

I think in intent is a reasonable way to pass ownership when the argument type is Owned. I'm not advocating it pass ownership when the argument type is e.g. MyClass. For Owned in particular, it already works this way and it wasn't my intention to re-litigate that decision. At least not in this issue.

How are we going to check the lifetime when a class pointer is passed by ref then the callee takes over the ownership? We'd need to look inside the function, at the very least, which violates "interface invariance".

We can assume that it can take in ownership - which, for Owned, means that it can invalidate the value passed to it. The lifetime checker doesn't currently think about whether the Owned/Shared continues to store a value - it just checks that a borrow doesn't live longer than the Owned/Shared variable. I think it's interesting to check for "invalidations" such as myOwned.clear(), since these can lead to use-after-free, but that's the subject of issue #8382 and would require different analysis than what I have so far (and I think it's more plausible that this other analysis might need to look into called functions, but I'd like to learn that from experience).

W.r.t. lifetime conversions, I suggest we start out with more restrictions until we have experience and/or user requests.

I don't know what you mean by "lifetime conversions". Do you just mean "Coercions between Owned and borrows"? "Start out with more restrictions" seems to me to be at odds with what you've already said - which is that the 2 implicit coercions I'm proposing are ones that we should support immediately. I'm not proposing anything else in this issue so I'm not sure how the idea to "Start out with more restrictions" helps this discussion? Maybe you can clarify what you meant?

[vasslitvinov]

W.r.t. lifetime conversions, I suggest we start out with more restrictions until we have experience and/or user requests.

Meaning +1 to starting with just the proposed two (owned->borrowed and child->parent) and adding more later as we see fit. Sorry for being unclear.

[vasslitvinov]

Michael's comment makes me realize that I am unclear about a couple of things.

• What are the differences in handling lifetime/ownership (ex. when passing to a function or returning) for records vs. classes?

• When we talk about lifetime/ownership of class instances, what is the type of the corresponding Chapel variable/formal/field/return value? Is it...

  • a class type with a compiler-supported annotation (somewhat like 'sync int')?
  • a library record that follows all Chapel record rules, with a field of a class type?
  • a magic record that follows all or most Chapel record rules and that the compiler knows about and does something about?
  • something else?
  • does the answer depend on the annotation i.e. owned vs. borrowed vs. unchecked?

@mppf could you clarify? What do others think it should be?

[mppf]

@vasslitvinov - I'm not sure what you mean by "lifetime/ownership"

What are the differences in handling lifetime/ownership (ex. when passing to a function or returning) for records vs. classes?

The checker I implemented does 2 types of lifetime checking:

1. ref
2. class instance borrows

These two parts happen to be implemented together but you could think of them as totally separate analyses.

I think it makes sense to reserve the term "borrow" for 2.
In that regard, only values of class type can be "borrowed". If a record stores a class instance, the compiler needs to know if it's a "borrow" or if it's got a field that's supposed to be "owning" it. For the prototype, I did that with a pragma.

Let me try to answer the question again:

What are the differences in handling lifetime/ownership (ex. when passing to a function or returning) for records vs. classes?

This is a lot of questions:

1. What happens when you pass a borrow into a function?
2. What happens when you pass a record storing a borrow in a field into a function?
3. What happens when you return a borrow from a function?
4. What happens when you return a record storing a borrow from a function?
   ...

Do you really want me to answer each of these individually? I think it makes more sense for you and I to chat, outside of this issue, at this point.

[bradcray]

Just a quick note to point out that int has a different default intent than atomic int and sync int so that gives us ground to have owned C have a different default intent than C (continuing with the theme of "if it's part of the language, it probably deserves a keyword modifier that I added to issue #8374).

[mppf]

So, if we allow consider Owned(Child) to be a subtype of Owned(Parent), where class Child : Parent, does that imply that you should be able to pass an Owned(Child) into a const ref argument of type Owned(Parent)? This is particularly relevant since const ref is the default intent for a record.

See also issue #8489.

[mppf]

This issue hasn't really engaged with the topic of separate types for "raw" pointers, but if we added such a thing, we'd have to decide about coercions between a :MyClass (i.e. a borrow) and :Raw(MyClass). These could conceivably go in either direction. Rust decided to only allow coercions from a borrow to a raw pointer, because going the other way would not guarantee the borrow had the safety properties normally guaranteed. Thus the user has to use a cast in that case. That approach arguably would add significant overhead to using a "raw" pointer in Chapel - and limit its usefulness as a transitional tool.

[mppf]

At the moment, I'm leaning towards allowing coercions from raw pointers to borrows, and not the other way. Rust raw pointers are fairly different from an explicitly deleted Chapel class.

[bradcray]

I'm leaning towards allowing coercions from raw pointers to borrows

Under some of the approaches, that seemed natural to me as well, from the perspective of wanting to write a generic routine and be able to call it with either a raw or an owned object, but I'm not sure I understand all the implications. Would it imply that we wrap some sort of layer around the raw pointer for the routine? Or would we clone the routine as though it were generic and have stricter/different checking on the raw vs. the owned version? Or something else?

[mppf]

a generic routine called with either a raw or owned object would be instantiated for each, I don't see any reason we'd do any particular wrapping.

Or would we clone the routine as though it were generic and have stricter/different checking on the raw vs. the owned version?

Now I'm wondering if you're talking about a generic routine? If it wasn't generic, normal coercion would apply (which long ago we did in a wrapper).

I feel like I'm missing something though - could you write a code snippet?

[bradcray]

I'm sure I'm the one who's missing something. From your earlier comment, I was thinking that simply coercing a raw object into a borrowed one would somehow loosen the amount of checking we could do on it (perhaps incorrectly). So if that was the case, I was imagining that we might want to either temporarily make it less raw somehow, or to clone for the two cases so that owned actuals sent to borrowed formals wouldn't lose any safety.

But maybe I've made something out of nothing. My intuition is that if the routine borrows the object it's basically saying "I don't have anything to do with its ownership" which seems as true for owned pointers as for raw ones (where the user has the responsibility of its ownership). So your response makes me more confident that my intuition might simply be right here and that I jumped to wrong conclusions earlier (?).

[mppf]

Yeah.

So if that was the case, I was imagining that we might want to either temporarily make it less raw somehow, or to clone for the two cases so that owned actuals sent to borrowed formals wouldn't lose any safety.

Safety is a pretty philosophical thing. Rust makes some particular choices in the name of safety (so allowing raw -> borrow coercions would indeed be "less safe" from their point of view). However we can make different rules about what safety properties we're interested in enforcing. We already know those properties != the Rust properties.

In particular, if we allow coercions from raw -> borrows, we're basically saying that if you use raw pointers in your code, the compiler won't guarantee the safety of them. But that seems so obvious as to potentially be not even worth saying, which is why I think I havn't said it yet.

In Rust, there is a desire to limit which regions of the code are working with "unsafe" stuff. At the same time, the coercion from raw->borrow might make a failure at runtime (e.g. use-after-free) possible in code that is not obviously using raw pointers. Vs if we didn't have such coercions - then we could say whichever code is doing the explicit casts is responsible for making sure the borrows are safe (which is the Rust philosophy on this point). But, from a whole-application view (which I think we tend to take), we can still point a finger at whichever code is creating raw pointers and not handling them correctly.

I also think that raw -> borrow coercions fit with our general goal of having some compile-time checking of pointer "safety" properties without making it too hard to use the pointers. The above might be interpreted as just another way that the Chapel lifetime checker is "incomplete" vs Rust's.

[vasslitvinov]

My way of saying - and supporting - the above:

• raw -> borrow coercion does not make the unsafe pointer any more unsafe. Philosophically this is fine. Let us allow it.

• borrow -> raw creates something potentially unsafe from something safe, implicitly. Philosophically, I suggest that we require users to label "unsafe" code as such explicitly, so do not allow this coercion implicilty.

This way we can have unsafe pointers passed into a function that accepts borrows and is perfectly "safe" w.r.t. lifetime checker. In my view this is perfectly fine for us.