Problem-solving: Coercion reconsidered and unified

[vrurg]

Coercion in its current implementation is just and only a particular case of a routine invocation and no more than that. Even in this role its implementation is incomplete and somewhat incorrect.

It's hard to separate the details of what's wrong or not implemented from the ideas of how to fix things. Thus most of this ticket content will follow in the second comment.

[vrurg]

Primary Proposal Target

First, I'd link back to tickets which inspired me: rakudo/rakudo#1285, Raku/problem-solving#137, Raku/problem-solving#22. Experimental draft of proposed changes implementation is in PR#3891. The most essential changes from the draft are:

• Coercion type must be a full-fledged type object akin to definite or subset. It's a nominalizable with its target type been the nominal coercion type. Nominalization chaining supported.
• Metamodel::CoercionHOW implements unified coercion protocol.
• Coercion type typechecks only against its target type but accepts both target and coercion types. I.e. it makes sense for Int(Str) to be Numeric but not Str, while Str ~~ Int(Str) is True.

Coercion Protocol

Defines the exact way a coercion is implemented. The current proposal is to have it consist of three ordered steps (assuming $value is what we apply coerce to):

• $value.TargetType is the current classical shortcut
• $value.COERCE-INTO(TargetType) is an alterative to the previous step. For a multi COERCE-INTO it could be possible to inject a candidate for coercing into a previously unsupported type. Provides an alternative to augmentation. Dubious.
• TargetType.COERCE-FROM($value) is a fallback of despair. As it was discussed in #137, for $value of a type which is heavily based on internal, non-public state correct coercion could be impossible without direct introspection of private attributes. And even then some of the state could be defined by class-local symbols. Yet, there are numerous cases where complete of good enough coercion is possible. For example, it is certainly better to have a multi method COERCE-FROM(Str:D $value) than augment core Str for absolutely the same result.

The protocol is implemented by Metamodel::CoercionHOW.coerce(). Correct implementation requires the metaclass to be able to determine if a COERCE-* method can be invoked with particular arguments. Due to cross NQP->Raku call, a Capture object cannot be used with cando method call. For this reason it makes sense to transition cando into a multi method with a second slurpy-arguments candidate.

Parameter Class Issues

Current implementation of coercion for a routine parameter suffers from being too specific. Coercion is supported explicitly and the implementation of it is spread across several locations in bootstrap, Parameter class, and in grammar actions.

I propose to reduce the implementation to the minimal necessary amount by delegating the actual coercion to the metaclass. All other locations would only be responsible for maintaining typeobjects data mandatory to perform correct coercions. By this I mean instantiation of generics and doing nominalizations where necessary.

There is a performance penalty for generically typed parameters which is described in PR#3891 and I'm reluctant to duplicate it here.

There is a big problem about Parameter class though which I have no good solution for yet. The problem is caused exactly by the explicit implementation of coercion for a routine invocation. What happens now is that a coercion type created for a signature is getting "disassembled" into a Parameter object. So, introspection of sub foo(Int(Str) $v) signature would not give us a parameter of a coercion type but of a nominal type. Worse, it would result in totally incorrect type because for the example provided we'd get Str as the type of $v which is totally absolutely wrong! This happens due to a misconception in handling of Parameter by grammar actions. Perhaps it was an attempt to work around a type matching problem, but the outcome is catastrophic.

Unfortunately, I don't see a really good solution for Parameter problem as any fix would most likely have a backward incompatibility in it. Because of Parameter is a part of bootstrapping, it is hard if not overall impossible to versionize it implementing the fix for 6.e only. Besides, it's hard to foresee possible fallouts of cross-version incompatible Parameter objects.

So far, the least problematic solution to me would be:

• Make Parameter a real single-type entity by introducing an attribute $!type which would always be exactly what a parameter was declared with. Kind of an exception: outcome of a generic instantiation.
• Method type of Parameter becomes an accessor of $!type instead of $!nominal_type.
• Accessor nominal_type is added.
• Both $!nominal_type and $!coerce_type become caches of $!type. Whereas $!nominal_type can and should be used for many type-based operations where use of nominalizables is not possible, $!coerce_type would become purely informative and could be deprecated.
• $!coerce_method should be deprecated. It's uses are to be replaced with coercion protocol.

This would limit incompatible changes to the private implementation details with the only publicly visible incompatibility of type method return changing to target_type nominalization for coercives. I don't expect this to be the cause of a massive ecosystem fallout.

[vrurg]

Basically, the problem with Parameter is what prevents me from completing the PR into something fully functional.

[vrurg]

A quick look through greppable6 results for use of Parameter in modules doesn't look like there would be a lot of breakages in case of gradual change of the class structure. It's very likely that DispatchMap would stop working. But it is explicitly marked as experimental and potentially breakable by changes to Rakudo internals.

[niner]

Just waving my hands here without much real knowledge, but if coercion types
become fully fledged types in their own right, couldn't they be compatible
enough with the nominal types so backwards compatibility of Parameter can be
maintained?

[vrurg]

@niner Not really. A coercion is not a nominal type and for anyone it'd be a surprise to get it from $!nominal_type. Besides, for Int(Str) it is so much incorrect to get Str as the parameter type (and this is what we currently get in $!nominal_type) that it has to be fixed.

Luckily, none of nominal_type, coerce_type, and coerce_method are specced. I'm worried about specs of Parameter.type accessor for coercing parameters, but quick grepping didn't come up with any results.

[vrurg]

BTW, I'm experimenting with throwing away $!nominal_type, $!coerce_type, and $!coerce_method altogether. So far got broken find_best_dispatchee. I expected it to handle any kind of parameter time equally but it's probably requiring nominals to work correctly.

[jnthn]

A few rough thoughts...

What happens now is that a coercion type created for a signature is getting "disassembled" into a Parameter object.

About why it's this way: coercion types were a fairly late arrival in the language, and came from a generalization of the parameter syntax Int $foo as Str, which meant to take an Int, but coerce it into a Str - or in today's money, Str(Int). When this new syntax appeared, it was implemented in terms of the existing mechanism. I suspect we can change $!nominal-type for $!type and update places that poke at the attribute. Everything outside of core should have been going through the methods, and we can probably mostly maintain the outer interface on those.

TargetType.COERCE-FROM($value)

The original speculation here was that a multi method new taking a positional argument would be used as the coercion fallback. I think I still prefer that over introducing a COERCE-FROM.

The protocol is implemented by Metamodel::CoercionHOW.coerce(). Correct implementation requires the metaclass to be able to determine if a COERCE-* method can be invoked with particular arguments. Due to cross NQP->Raku call, a Capture object cannot be used with cando method call. For this reason it makes sense to transition cando into a multi method with a second slurpy-arguments candidate.

I'm not sure this is really the best hook to hang this off. I think coercion would be better implemented as a dispatcher (in terms of new-disp); in fact, I already did start stubbing a coercion handling dispatch in. That way we can wire it all together far more efficiently.

[lizmat]

@jnthn How would that work with something like:

my Str(Int) $a = 42;

[jnthn]

@lizmat That was always going to need to be implemented as part of assignment. That would once have been a real pain (when scalar internals were all in C), but these days assignment to scalars is implemented as a spesh plugin (master) or a dispatcher (new-disp), meaning in the latter case it's a neat fit there also.

[vrurg]

my Str(Int) $a = 42;

This was an easy one and is already working.

[vrurg]

TargetType.COERCE-FROM($value)

The original speculation here was that a multi method new taking a positional argument would be used as the coercion fallback. I think I still prefer that over introducing a COERCE-FROM.

Not that I would strongly oppose using new. I even like it in a way. But the following points are making COERCE-FROM a little more attractive to me:

• It comes in pair with COERCE-INTO, unless it is decided to drop off this one.
• It is more explicit and thus would make sources a bit more readable.
• It allows to have slightly different semantics for object creation and coercion. For example, Session.new($string) might mean creating a named empty session; but Session(Str) might create a session from a JSON data.

Though for the last one I'm still thinking if Session(JSON(Str)) would be better option after all despite its verbosity.

In either case, using new instead of COERCE-FROM is only a matter of changing a single string as not a big deal at all.

The protocol is implemented by Metamodel::CoercionHOW.coerce(). Correct implementation requires the metaclass to be able to determine if a COERCE-* method can be invoked with particular arguments. Due to cross NQP->Raku call, a Capture object cannot be used with cando method call. For this reason it makes sense to transition cando into a multi method with a second slurpy-arguments candidate.

I'm not sure this is really the best hook to hang this off. I think coercion would be better implemented as a dispatcher (in terms of new-disp); in fact, I already did start stubbing a coercion handling dispatch in. That way we can wire it all together far more efficiently.

The protocol implementation is only about 40 lines of code with comments and blank lines. The primary purpose is to define the rules. Besides, if I understand things right, there must be an entry point into the dispatcher. To provide a user with a way to invoke manual coercion I think Int(Str).^coerce("42") would do the good job. It's just so that when new-disp is finally in place the current code of method coerce will be replaced with a dispatcher invocation. Unless I miss something here.

What currently bothers me is that most of the changes in rakudo/rakudo#3891 are done to the areas to be affected by the upcoming RakuAST and new-disp projects. Those are actions, world, spesh plugins. Even in bootstrap I currently need to get find_best_dispatchee working with the changes in Parameter. Does it even worth finishing the PR? On one hand, half of it would be thrown away. On the other – it is unlikely to happen within the next few months and this time could be used to polish the new CoercionHOW and Parameter. Any advise here would be appreciated.

[vrurg]

@jnthn I would strongly need your opinion on a new problem popped up.

The old coercion semantics was so that it delegates to Mu via type pretense. The new semantics makes it nominalizable with delegation to its nominal type. This seemingly broke a lot of specs and I started investigating Int() == 0 case which is supposed to pass despite a warning. It previously worked due to .Numeric method re-delegated directly on Mu as a result of type pretense. Now it requires :D on Real::infix:<==> arguments to get closer to the required behavior. I.e. in this case it falls through to Numeric::infix:<==> and attempts .Numeric on the coercion type which results in a warning and then dies at self.new where self happens to be Int().

I considered the idea of nominalizing type objects prior to invoking a method on them. But rejected it later because this would not allow a class developer to react to non-nominal invocant:

class Foo {
    method special-new(Foo:U:) {
        if self.HOW.archetypes.nominalizable {
            return self.^nominalize.new(self.^constraint_type);
        }
        self.new;
    }
}

So, I decided to go other way around and make Mu.new determine if nominalization is needed prior to instantiation. The outcome is the following constructs are now working:

class Foo { }
subset Fubar of Foo;
say Foo(Str).new.raku;
say Foo:D.new.raku;
say Fubar.new.raku;

And the following also works as expected:

class Bar {
    has Int(Str) $.bar is rw .= new;
}
my $bar = Bar.new;
say $bar.bar.raku;
$bar.bar = "42";
say $bar.bar.raku, ", ", $bar.bar.WHAT;

Without the changed Mu.new semantics the above .= new construct was failing. As far as I understand, manual handling of a definite for this case could also be eliminated now.

Evidently, there is spectest fallout caused by the changes. I just have started analyzing it. But got two major cases worth considering already.

Here is the first one from APPENDICES/A02-some-day-maybe/misc.t:

subtest 'same exception with and without type smiley for failing coercion on var' => {
    plan 3;
    my \XTAD = X::TypeCheck::Attribute::Default;
    throws-like ｢class { has Int() $.x = "42"}.new.x｣,           XTAD, 'no type smiley';
    throws-like ｢class { has Int:D() $.x = "42"}.new.x｣,         XTAD, ':D (1)';
    throws-like ｢class { has Int:D() $.x = "42"}.new(:x("43"))｣, XTAD, ':D (2)';
}

It basically forbids a coercion to be used as attribute type. To my view it's totally incorrect assumption.

The second case is tested in a couple of locations and winds down to disallowing instantiation of a subset meaning that Fubar.new from above must throw. Not sure about this one as on one hand a subset hides its nominal type and thus makes Fubar.new harder to understand. On the other hand, Fubar is Foo after all. So, why not?

The last important detail. Unfortunately, there seem to be no way to versionize coercions. They're parametric meaning that Int(Str), for example, gets instantiated by the core and becomes permanently 6.c versioned. For this reason there is no reliable way for it determine wether it should re-delegate to Mu or to its target type.

The bottom line for now is such that either we explicitly disable certain uses of coercions like with type captures, for example. Or specs are to be changed for 6.c/6.d.

[vrurg]

The latest status on development of the related PR.

Most of the failing tests were taken care of by fixing bugs. A few needed adjustments similar to the above mentioned APPENDICES/A02-some-day-maybe/misc.t for which I have changed Int() to Int(Rat) because the former now works and accepts strings. But since the meaning of the test is to check for the sameness of the exception thrown, the change doesn't break this.

The biggest remaining problem for now are subsets passing Subset.new where it previously expected to throw. In the essence, I don't see why should we demand this to fail since a nominalizable unwinds to a nominal type which we can instantiate. And, basically, nominalizables are nothing but just wrappers. For this reason I don't see why is it possible to has Int:D $.foo .= new, but not has UInt $.foo .= new. Especially considering that Int:D could be defined as subset IntD where *.defined. And another view to the situation is from the point that the following two are basically the same:

subset BigVal of Int where * > 1000;
has BigVal $.foo;
has Int $.foo where * > 1000;

Except that the first one doesn't allow .= new.

The point could be made that a subset actually obscures it's underlying type. But to my view it's no big difference to an attribute obscuring the type it's declared with.

So far, the only real problem I see in this is that 6.c specs would actually be changed retroactively. But on the other hand, the change is permissive and allowing a behavior is not as bad as breaking or disabling it.

[raiph]

@vrurg

the change is permissive and allowing a behavior is not as bad as breaking or disabling it.

I think it would be helpful to decide whether or not this can and should be established as a permanent official principle of Raku evolution, either with or without qualification, and, if so, preferably with immediate effect retroactively to 6.c.

---

It seems to me that keeping Raku as free as possible to evolve to clean up after mistakes is of enormous strategic importance. One aspect of that is the freedom to evolve entirely new languages atop all that's been built to date. But this issue of trying to have our cake (improve a language even if it breaks some existing code) and eat it too (avoid breaking existing code on an arbitrary basis) seems like a fantastic thing to become ever better at. And keeping careful long term track of the issues that highlight problems and solutions related to that is presumably of tremendous value.

It feels like one big value coming from this issue (and several others you've been pushing, including the one where you provided me a link to jnthn's doc on language version change protocols/principles), is your discovery of wrinkles as you wrestle with backcompat commitment issues that arise as you try to improve/evolve the language. Perhaps the "language" tag is the right one for ensuring the issues you open that are of interest in terms of "backcompat snafus" / "impediments to language evolution" are tagged and findable via their tag(s). Or is there a need for a more specific tag?

[vrurg]

the change is permissive and allowing a behavior is not as bad as breaking or disabling it.

I think it would be helpful to decide whether or not this can and should be established as a permanent official principle of Raku evolution, either with or without qualification, and, if so, preferably with immediate effect retroactively to 6.c.

I'm afraid, not everything is that simple. In certain cases third party code may rely on something to fail intentionally and has it as a design pattern. Enabling this "something" could be a major breakage in this case. Luckily, the case of subsets should not be a problem. Especially considering the following:

class Foo {
    method foo(::?CLASS:U:) {
        say "Foo!"
    }
}
subset F of Foo;
F.foo; # Foo!
my $foo = F.new; # You cannot create an instance of this type (F)

This is a rather weird discrepancy which is not the case anymore in the submitted PR.

Or is there a need for a more specific tag?

I don't think this needs a separate tag. language is good enough.

[jnthn]

This is a rather weird discrepancy which is not the case anymore in the submitted PR.

What's weird about it? Method dispatch is orthogonal to representation, and whether creating instances is possible is a property of the latter.

[vrurg]

This is a rather weird discrepancy which is not the case anymore in the submitted PR.

What's weird about it? Method dispatch is orthogonal to representation, and whether creating instances is possible is a property of the latter.

The weirdness in the eye of an unexperienced user, that's what I thought about. Otherwise it's exactly the behavior I don't want to break by implicit type nominalization.

[jnthn]

I've been considering the proposal some more, and here's a few thoughts.

COERCE-INTO has variance issues

Most immediately, I think COERCE-INTO is not going to work out well, due to the way variance works out with that API. When I want to coerce into some type T, then the result I get back should have the property r ~~ T. However, given some type B where T ~~ B, if I implement multi method COERCE-INTO(B) { B.new }, then this candidate will match T, but return B and B !~~ T. Every COERCE-INTO we would implement would be vulnerable to accepting subtypes and then not producing something of that type. Thus it seems to be to be a bad idea.

At the same time, there's something unsatisfying about the current state of affairs with regards to providing coercion implementations on the source type. My initial feeling when I saw COERCE-INTO - and before I realized the variance problem - was that it would be unfortunate to provide two competing ways to implement coercions in the source type (either implementing a method Int() { ... } or implementing a multi method COERCE-INTO(Int) { ... }). I wondered if we might instead be able to reform the current approach sufficiently that existing uses of it naturally fall under the new scheme.

A counter-proposal for improving source-provided coercions

Today, coercions are not reified in the MOP in any explicit sense; but rather exist simply as methods. This is the primary thing I propose to change.

When we encounter a method definition, we check if the name of the method corresponds to a typename that is in scope. (It would be highly odd to write a coercion method for a type that is not in scope; how do you plan to make an instance of it?) If the method name corresponds to a typename, then the method is registered with the metaclass as a coercion for that type. Coercion types, when the coercion should be performed, then query this coercions table in order to decide how to coerce. For now we'll assume the invariance of the current approach is retained (e.g. a Str coercion only handles precisely requests for coercions to Str, and would not, for instance, handle coercions to Stringy), however below I'll sketch a variance extension too.

I propose the following semantics:

1. In order to be considered a coercion, the method must be named after a type that is in scope and have a signature with arity 1 and no parameter declarations aside from an explicit invocant.
2. Single-part method names, like Str, declare both a method and a coercion. A method name like IO::Path will resolve to the IO::Path type correctly and declare such a coercion (noting that coercions are keyed on the type they resolve to, not the string name of the type). [Conjectural: however, as of 6.e, they will no longer result in a the declaration of a method named with the string "IO::Path", and declaration of a multi-part method name that does not resolve to a type name will become an error.]
3. Unless the candidates involved are marked multi, it is an error for two coercions for the same type to be specified. This again is on type equivalence, so e.g. my constant Path = IO::Path; method IO::Path() { }; method Path() { } would be a compile-time error [conjectural: from 6.e if there are ecosystem compatibility issues, however, this seems unlikely].
4. Coercions are inherited. If there are multi candidates, they are gathered and chosen according to the usual multiple dispatch rules. Most probably this is implemented by synthesizing a proto for each target coercion type. The proto of a method that is registered as a coercion method must by an onlystar proto, either by explicit declaration or as a result of generation.
5. [Conjectural: a coercion scope declarator is introduced, such that one could write coercion method Foo() { }. This would have the effect of registering the method as only a coercion rather than as a standard method, and further raising a compile time error should Foo not resolve to a type name or should the signature not meet the requirements. Note a scope declarator is considered the appropriate peg because it impacts upon the installation of the symbol, which is exactly what scope means.]
6. [Conjectural: a coercion method automatically receives a return type constraint of the target type of the coercion, thus method IO::Path() { ... } implicitly means `method IO::Path(--> IO::Path) { }.]

Variance

Today, a coercion method for Str can not handle a coercion to Stringy, even though it would meet the property of returning something that does Stringy. With coercion dispatch handled distinctly, however, we have a chance to resolve this. We should also consider the case of upcasts that are not simply no-ops. For example, today we have a method List on Array, which is not identity, but rather strips the Scalar containers. Thus we could have dispatch semantics for coercions as follows:

1. A topological sort is performed on the coercion types, to produce a grouped ordering, just as happens in the case of multiple dispatch if all candidates are arity 1. However, the arrows in the narrowness graph are reversed - that is to say, the first group contains the widest type(s). For example, if we had coercion candidates to Numeric, Int, and Num, then the ordering groups would be ((Numeric,), (Int, Num)). (It's somewhat reassuring that we find the arrows reversed in a situation where we're wanting the opposite variance.)
2. The groups are searched for a type that is accepted by the target type (that is to say, if we put the target type on the right of the smart match, however in reality we'll do the judgement with istype). For example, suppose we wish to do a coercion to Numeric. The first group is considered, and Numeric ~~ Numeric comes out True, thus we have a result. Now consider a target type of Int. Numeric ~~ Int comes out False, so we proceed to the second coercion group, rejecting Num since Int !~~ Num, and accepting Int since Int ~~ Int. Now let's instead consider the case where we only provide a Str coercion and want to coerce to Stringy. Thus there's one group with one coercion. Since Str ~~ Stringy, this coercion is considered applicable. If we only declared a coercion to Stringy, but a Str is requested, then since Stringy !~~ Str, then it would not be considered applicable.
3. As in multiple dispatch, an ambiguous offering is an error. For example, if a type only offers Int and Num coercions and we wish to coerce to Real, then it is an error.
4. There may be a follow-up real multiple dispatch to decide which multi candidate to actually use (typically definedness constraints, but in principle a class could write multi methods with where constraints on the invocant).

[vrurg]

Now it is my turn to think over it. For the moment I see a few points which may require postponing the implementation until RakuAST is able to handle it – simply due to the complexity of the task, possibly making it unreasonable to work on the current grammar/actions.

@jnthn, to complete the picture I'd like to consider the option for the opposite path of COERCE-FROM. I still think it would be useful to allow a developer to distinct between a normal new invocation and a coercion case. Perhaps there is a solution to it: the method could be invoked upon the coercion type itself. I.e. technically:

sub foo(Foo(Any) $v) { ... }
foo(42);

would result in Foo(Any).new(42). new could then choose a particular path depending on wether self is Foo or Foo(Any). For now multi-dispatch doesn't make a difference between the two, but in the future this could be taken care of allowing to:

multi method new(Foo: $source) { ... }
multi method new(Foo(Any): $source) { ... }

[jnthn]

I still think it would be useful to allow a developer to distinct between a normal new invocation and a coercion case.

Need to ponder this (and other things mentioned in various of the posts) more, but my thinking was that idiomatic new usage for construction takes named parameters, and thus a constructor taking a positional argument is already somewhat distinctive.

[vrurg]

Haven't had any spare time time lately to work on the issue and now leaving for a small vacation. So, the only thing I have to mention is that positional new is already used by some modules. And the logic they implement is not necessarily fits well into coercion requirements. So far my consideration is that it'd be easier to produce PRs for such modules and take care of a coercive invocator instead of either changing their logic cardinally, or leave them unaware of the new semantics altogether.

[jnthn]

@vrurg Yes, I guess there's a risk of "false positives" in such cases; I guess it depends how common it is. If we do introduce a new name, though, maybe it can simply be COERCE rather than COERECE-FROM, given the issues with a COERCE-INTO idea.

[jnthn]

@vrurg And of course: enjoy your vacation!

[vrurg]

Leaving it as a note. I consider COERCE[-FROM]/new approach susceptible to the same variance problem, as @jnthn noticed with COERCE-INTO. In a very simple case where Bar doesn't know about Foo and consequently doesn't have a coercion method for it:

class Foo {
    method COERCE(Bar:D \val --> Foo) { ... }
}
class Fubar is Foo {
}
sub fubar(Fubar(Bar) $param) { ... }

fubar will get a Foo instance in $param which is certainly not what is expected.

So far, it seems that the solution lies in the area of complete encapsulation of source-coercion routines within their respective class via use of submethod. Something like:

class Foo {
    multi submethod COERCE(Bar:D \val) { ... }
}
class Fubar is Foo { ... }

In this case Fubar(Bar) will fail because there is no known coercion path for it.

[vrurg]

Even though rakudo/rakudo#3891 is merged now and thus this issue can be closed, I'd leave one more comment as currently there are no other sources of information about what's been done.

After all, COERCE is supported in both method and submethod variants. To mitigate the unwanted effects of variance problem a coercion typechecks against its target type and throws if the check fails. Actually, the recommended approach for writing a COERCE is to consider it a kind of new and use self for instantiating:

multi method COERCE(Foo:D $v) {
    self.new: ...
}

This is not necessary for a submethod COERCE because it would only be invoked if the coercion target is the class where the submethod is declared:

class Foo {
    method COERCE(...) {...}
}
class Bar is Foo {
    submethod COERCE(...) {...}
}
class Fubar is Bar { }
sub foo(Bar() $v) { ... } # submethod Bar::COERCE will be invoked
sub bar(Fubar() $v) { ... } # method Foo::COERCE will be invoked

Method new of the coercion target type is invoked as a fallback candidate is no acceptable COERCE found. To let the new find out about the coercion context, dynamic variable $*COERCION-TYPE is provided which is bound to the coercion target type.

[vrurg]

Closed via rakudo/rakudo#3891