Dotty: SIP: Widening of case class apply to better emulate ADTs

[odersky]

Motivation

In Scala, case classes are used to emulate algebraic data types. But the emulation is not exact, insofar as a case class represents its own type whereas a constructor of an ADT has as result type the type of the underlying ADT type. For instance, using

trait Option[+T]
case class Some[T](x: T) extends Option[T]
case object None extends Option[Nothing]

the type of Some(1) is Some[Int], not Option[Int]. While useful, this can also cause problems for type inference. This SIP proposes an alternative class and object syntax, using : for extends. The only difference between the two forms concerns the auto-generated apply method of a case class. If extension is via a :, apply has as return type the conjunction of all parent types following the :.

_Example:_ Using

trait Option[+T]
case class Some[T](x: T): Option[T]
case object None: Option[Nothing]

we have as most precise typings:

Some(1): Option[Int]
None: Option[Nothing]

Proposal in detail

In SyntaxSummary.txt change

TemplateOpt       ::=  [`extends' Template | [nl] TemplateBody]

to

TemplateOpt       ::=  [(`extends' | `:') Template | [nl] TemplateBody]

Add the restriction that : can be used only for case classes. (Not sure we need this, should we allow `:' for normal classes as well? It would make no difference there).

Augment the definition in the spec on Case Classes as follows.

If the case class definition is defined with a : extension clause, as in
C[tps](ps1)...(psn): parents {...} the following apply method is generated instead, where
P1, ..., Pn are the declared or inferred parent types of C:

object C {
 def apply[tps](ps1)...(psn): P1 & ... & Pn = new C[tps](xs1)...(xsn)
}

Augment the definition in the spec on object definitions as follows: After the existing

It is roughly equivalent to the following definition of a lazy value:

    lazy val m = new P1 with ... Pn { this: m.type => stats }

add the clause: if the case object definition is defined with a : extension clause, the associated object is given the type P1 & ... & Pn, as in:

lazy val m: P1 & ... & Pn = new P1 with ... Pn { this: m.type => stats }    

[sjrd]

The "proposal in detail" does not explain how None gets type Option[Nothing] rather than None.type, as said by the example.

[julienrf]

What's the motivation in supporting both, extends and :? This adds complexity to the language.

[DarkDimius]

This proposal does not cover case, when case class is intended to be published with a type different from superclass. Something similar to:

trait Option[+T]
trait OptionPrivateInterface[+T] extends Option[T]
case class Some[T](x: T) extends OptionPrivateInterface[T]: Option[T]
case object None: OptionPrivateInterface[Nothing]: Option[T]

[odersky]

@julienrf We can't change the semantics of extends. So if we want ADT semantics we need to use a different mechanism.

@darkdimius Indeed, an alternative would be to allow both :' andextends'. Not sure whether it's worth it. In that case I would swap the order:

case class Some[T](x: T): Option[T] extends OptionPrivateInterface[T]

[odersky]

@sjrd Right. I'll added some wording to the proposal.

[japgolly]

Regarding the motivation for this change, Some(1) returns a Some[Int] but a Some[Int] _is_ an Option[Int] so the types are accurate and not a problem, yes? So is it fair to say that the only problem is type inference?

[odersky]

Sent from my iPhone

On 26.02.2016, at 05:53, David Barri [email protected] wrote:

Regarding the motivation for this change, Some(1) returns a Some[Int] but a Some[Int] is an Option[Int] so the types are accurate and not a problem, yes? So is it fair to say that the only problem is type inference?

Yes I think that's a fair characterization.

—
Reply to this email directly or view it on GitHub.

[ritschwumm]

if apply's result type is changed, shouldn't unapply's parameter type changes, too?
could be useful to make a prism from an apply/unapply pair.

[japgolly]

So is it fair to say that the only problem is type inference?

Yes I think that's a fair characterization.

In that case what do you think about attacking this problem at its root and improving type inference? I think it would be a better use of energy because not only will it solve this issue but others too.

In fact, I think it would be fantastic if at some point in the future, the type inference part of scala/dotty were in some mostly isolated module (or package) with some documentation on how to contribute (going from zero-knowledge to PR-ready) so that people in the community could add their own examples to the tests and make improvements to the logic. I'd say type inference is a bit of a sore point once you go beyond the basics, this issue is an example of something that seems simple from the outside but doesn't work.

This syntax enhancement might be nice in its own right but preventing the need for a syntax enhancement would be even better! There'd be cheers if type inference were improved and applause if users could easily contribute.

[ritschwumm]

@japgolly this isn't about enhancing syntax.
scala conflates the notion of a type having constructors (in the haskell sense) with the notion of a type having subtypes. this is a step towards un-conflating those two.
my gut feeling says the direction is right, but the step might not be large enough.

[OlegYch]

case class Some[T](x: T): Option[T] extends OptionPrivateInterface[T]

seems counter intuitive

this seems more consistent with other type ascriptions

case class Some[T](x: T) extends OptionPrivateInterface[T] with OtherPrivateInterface: Option[T] with OtherInterface

[Ichoran]

Could we achieve the same thing in a more general way by allowing type aliases to widen?

That is, we'd use

sealed trait Option[+A] {}
object Option {
  case class Some[+A](a: A) extends Option[A] {}
  case object None extends Option[Nothing]
}
type Some[A]: Option[A] = Option.Some[A]
type None: Option[Nothing] = Option.None

to get a user-facing Some and None that were widened to Option of the appropriate type? (And the self-typed versions would still be available, in this case as Option.Some and Option.None?)

[odersky]

On Fri, Apr 29, 2016 at 10:21 PM, Ichoran [email protected] wrote:

Could we achieve the same thing in a more general way by allowing type
aliases to widen?

That is, we'd use

sealed trait Option[+A] {}object Option {
case class Some+A extends Option[A] {}
case object None extends Option[Nothing]
}type Some[A]: Option[A] = Option.Some[A]type None: Option[Nothing] = Option.None

to get a user-facing Some and None that were widened to Option of the
appropriate type? (And the self-typed versions would still be available, in
this case as Option.Some and Option.None?)

What exactly would a widened type alias mean? I have no good intuition for
this.

• Martin

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/lampepfl/dotty/issues/1093#issuecomment-215870178

Prof. Martin Odersky
LAMP/IC, EPFL

[Ichoran]

A type alias is essentially referentially transparent in that

type Foo = ComplicatedBarThingy[SomethingAboutBaz]

and you simply do a textual replacement of Foo with ComplicatedBarThingy[SomethingAboutBaz], you will in most cases get exactly the same behavior, whether it be from constructors or methods on companion objects or implicit search or whatever.

The proposal would be that you detach the constructors/methods from the declared type. So

type Foo: SuperType[Bar] = SubType[Bar]

would appear in type signatures as SuperType[Bar], but it would use the constructor and companion object methods of SubType[Bar], promoting any return values of SubType[Bar] to SuperType[Bar] along the way, unless those return values already were SubType[Bar] in SuperType[Bar] or they are part of an unapply.

Just as type Foo[A] = SomeType[Bar, A] effectively removes the unnecessary boilerplate of using SomeType[Bar, A] over and over, type Foo[A]: SuperType[A] = SubType[A] would remove the unnecessary boilerplate of casting SubType[A] to SuperType[A] all the time.

There are a few cases where it wouldn't work and you'd have to fall back to the un-aliased case (e.g. I can't think offhand of a good way to make typeclasses work the right way), but 95% of the time it should do the trick, I think.

[odersky]

On Sat, Apr 30, 2016 at 8:27 PM, Ichoran [email protected] wrote:

A type alias is essentially referentially transparent in that

type Foo = ComplicatedBarThingy[SomethingAboutBaz]

and you simply do a textual replacement of Foo with
ComplicatedBarThingy[SomethingAboutBaz], you will in most cases get
exactly the same behavior, whether it be from constructors or methods on
companion objects or implicit search or whatever.

The proposal would be that you detach the constructors/methods from the
declared type. So

type Foo: SuperType[Bar] = SubType[Bar]

would appear in type signatures as SuperType[Bar], but it would use the
constructor and companion object methods of SubType[Bar], promoting any
return values of SubType[Bar] to SuperType[Bar] along the way, unless
those return values already were SubType[Bar] in SuperType[Bar] or they
are part of an unapply.

I don't understand "but it would use ...". What is "it"? When precisely
would they be used?

Cheers

• Martin

Just as type Foo[A] = SomeType[Bar, A] effectively removes the
unnecessary boilerplate of using SomeType[Bar, A] over and over, type
Foo[A]: SuperType[A] = SubType[A] would remove the unnecessary
boilerplate of casting SubType[A] to SuperType[A] all the time.

There are a few cases where it wouldn't work and you'd have to fall back
to the un-aliased case (e.g. I can't think offhand of a good way to make
typeclasses work the right way), but 95% of the time it should do the
trick, I think.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/lampepfl/dotty/issues/1093#issuecomment-215985396

Prof. Martin Odersky
LAMP/IC, EPFL

[Ichoran]

Given type Foo: SuperType[Bar] = SubType[Bar], any instance of x: Foo would be treated as x: SuperType[Bar], but new Foo(myBar) would be translated to (new SubType[Bar](myBar)): SuperType[Bar], and Foo.mkFrom(myBar) would be translated to SubType.mkFrom(myBar): SuperType[Bar] assuming that mkFrom performs Bar => SubType[Bar].

[odersky]

OK, that's clearer now. But that would mean that a type definition like Foo introduces also a term Foo. That seems drastic and will likely break things.

[Sciss]

I think this vastly complicates the language and the cognitive burden on people learning Scala. What's wrong with allowing one to override the apply method of the companion object, like:

object Some {
  def apply[A](x: A): Option[A] = new Some(x)
}
case class Some[A](x: A) extends Option[A]

Another possibility would be to add an annotation, something like

@apply[Option] case class Option[A](x: A) extends Option[A]

But in general we already have Option.empty and Option.apply. You can use those if needed.

[Sciss]

Ah, apologies, I was thinking this was Scala not Dotty. (Still I would find concurrent : and extends confusing).

[dwijnand]

It would generally be very useful to redefine a case class's companion apply and unapply method. For instance to evolve case classes while maintaining binary compatibility.

[Ichoran]

@odersky - In one sense it is drastic to allow type statements to introduce terms. But in another sense I think it is more regular, as there's a mismatch between direct and typeclass-evoked functionality with type as it stands now. Now it detaches you from your companion object and/or static methods (e.g. type X = String is not the same as alias X = String because X.valueOf(1) doesn't work) and there's no real way to get it back. It may be too big a change, though. (If you allowed only the type A: X = Y form to introduce a term, though, it wouldn't be source incompatible with anything existing, and if you did want a more alias-like behavior then you could type A: X = X.)

[nafg]

It doesn't make sense to me, currently case is just a modifier applied to regular class syntax that generates stuff. Allowing to replace extends with : is a huge syntax change. It doesn't make sense to only allow that in such a limited context, which until now is otherwise a regular syntax context.

I think it's a laudable goal, but if we're tweaking syntax for ADTs, how about also addressing the cumbersomeness of defining ADTs? How about a way to define all the cases in one statement? I think once you come up with syntax for that, you could kill two birds with one stone and say that when the cases are defined in the short form, the factories use the base type rather than the subtype.

@Ichoran do you remember the syntaxes we were discussing?

[nafg]

@odersky why is this more worth language footprint than proper enums?

[odersky]

@nafg It would be much simpler than proper enums. But I agree that it is a borderline case to be considered. Note that you can already express the idiom, albeit in a convoluted way:

abstract case class Some[T](x: T)

object Some {
   def apply[T](x: T): Option[T] = new Some[T] {}
}

The abstract is necessary so that an apply method won't be generated automatically.

One might consider not generating an apply method if one exists already, but that fails because
it would also avoid an apply method when there are overloaded variants of the apply given, which is
quite a common case in practice.

[sjrd]

One might consider not generating an apply method if one exists already, but that fails because
it would also avoid an apply method when there are overloaded variants of the apply given, which is
quite a common case in practice.

Why not simply avoid generating an apply method if one exists already _with the same parameter types_?

[odersky]

@sjrd

Why not simply avoid generating an apply method if one exists already with the same parameter types?

It would be nice if it were so simple. Unfortunately, it's a chicken and egg problem. To know that there is no such apply method, you have to elaborate all other apply methods., which means: determining their parameter type. This can in turn cause other case classes to be completed (i.e. have their type determined) which will have to decide whether they need an apply method generated. It's not clear at all how to avoid CyclicReference errors in these situations.

[nafg]

On Sun, May 1, 2016, 4:27 PM odersky [email protected] wrote:

@sjrd https://github.com/sjrd

Why not simply avoid generating an apply method if one exists already with
the same parameter types?

It would be nice if it were so simple. Unfortunately, it's a chicken and
egg problem. To know that there is no such apply method, you have to
elaborate all other apply methods., which means: determining their
parameter type. This can in turn cause other case classes to be completed
(i.e. have their type determined) which will have to decide whether they
need an apply method generated.

Why should determining a type require fully computing apply methods? That
sounds like an architectural weakness.

It's not clear at all how to avoid CyclicReference errors in these

situations.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/lampepfl/dotty/issues/1093#issuecomment-216069800

[odersky]

On Sun, May 1, 2016 at 10:45 PM, nafg [email protected] wrote:

On Sun, May 1, 2016, 4:27 PM odersky [email protected] wrote:

@sjrd https://github.com/sjrd

Why not simply avoid generating an apply method if one exists already
with
the same parameter types?

It would be nice if it were so simple. Unfortunately, it's a chicken and
egg problem. To know that there is no such apply method, you have to
elaborate all other apply methods., which means: determining their
parameter type. This can in turn cause other case classes to be completed
(i.e. have their type determined) which will have to decide whether they
need an apply method generated.

Why should determining a type require fully computing apply methods? That
sounds like an architectural weakness.

Believe me, it's complicated. Experiments and pull requests to overcome
this are appreciated of course! But I personally won't touch it. I have
seen too many failed attempts already.

• Martin

It's not clear at all how to avoid CyclicReference errors in these

situations.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/lampepfl/dotty/issues/1093#issuecomment-216069800

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/lampepfl/dotty/issues/1093#issuecomment-216071318

Prof. Martin Odersky
LAMP/IC, EPFL

[odersky]

I decied to write up a document explaining the namer/typer interface. That should give us more clarity whether something can be done. But note we still have not addressed the case where we would like to have None: Option[Nothing].

[Sciss]

we still have not addressed the case where we would like to have None: Option[Nothing]

I still don't understand why Option.empty and Option.apply are not sufficient.

[ritschwumm]

@sciss Option.apply is different, it returns None for a null value

[OlegYch]

@Sciss this is for any ADT, not just for Option, and repeating an apply method for every constructor is cumbersome, besides most of the time constructors with too specific return type are not useful and just clutter interface

[nafg]

@odersky

@nafg It would be much simpler than proper enums.

Perhaps simpler to change the spec, but it would be a much greater cognitive stumbling block. There's nothing intuitive about adopting C++/C#'s "extends" keyword (the :) but only in certain places, and the difference has to do with something random.

I would be more open to the following, instead of only allowing it on case classes:

If a : is used in a class definition in pace of extends, then the compiler will generate an apply method on the companion taking the same arguments as the primary constructor with a return type of the type following the :. If the class is a case class, then this apply method is generated instead of the normal case class apply method (i.e. with the class type as its return type).

Thus, : becomes a general rule, that intersects with case class simply by taking precedence in the one area of conflict.

However, ONCE AGAIN, WE _REALLY TRULY SERIOUSLY VERY BADLY_ need a lightweight syntax for defining ADTs and enums. So can we PLEASE PLEASE PLEASE get syntax for that, and then revisit the "problem" that is the premise of this issue in that context?

[mdedetrich]

I think everything has been pretty much said in this debate, but I strongly disagree with using : as syntax for the special case of extend. Its conflating the syntax with something that has a very consistent meaning in scala (either specifying a variable has a type, or typecasting a variable). Neither of those things has anything to do with what this ticket is solving.

Its also terrible from an aesthetics point of view, it breaks the ability to easily read and segment different pieces of code to quickly get meaning (Scala in general is very good at this, the : conception would make this a lot harder)

Even using a keyword (implements anyone?) would be better. In fact I think thats a pretty good solution for consistency sake.

i.e.

trait Option[+T]
case class Some[T](x: T) implements Option[T]
case object None implements Option[Nothing]

Looks pretty nice.

@nafg
I am kinda against adding lightweight syntax to ADT's in general, because its yet another thing someone has to learn, and the even though the current syntax is a bit boilerplatery, it does signify to the user how case classes/objects are implemented (i.e. they are a special type of class)

[nafg]

@mdedetrich I think it does have some connection, at least on an intuition level. E.g.,

def x: T

means x has type T

similarly

class X(): T

means X() has type T

But I agree it's a stretch, and combined with the fact that it's sort of camouflaged in somewhere, it will just add confusion.

On the other hand if we add and ADT/enum shorthand, it should be pretty obvious from looking at the code what it does, so it might affect the "spec footprint" more, but it would not make the language as much harder to learn (or to read without fully learning).

As far as the concern you allude to, them such syntax would mask the expanded equivalent, there's plenty of precedent in scala for one feature expanding to equivalent code, such as case class (in theory, though the extractor is short-circuited), for comprehensions, and lots of function syntax (literals, literal shorthand, partial application).

[mdedetrich]

But I agree it's a stretch, and combined with the fact that it's sort of camouflaged in somewhere, it will just add confusion.

Its also visually really bad, because you have multiple : on the same line, it gets really hard to distinguish between type annotations and some special case of extends.

On the other hand if we add and ADT/enum shorthand, it should be more or less obvious from looking at the code what it does, so it might affect the "spec footprint" more, but it would not make the language as much harder to learn (or to read without fully learning).

Yeah precisely

[odersky]

@mdedetrich Note that `:' is a C#-ism where it means extends. C# people have no problem parsing it so I guess it's a matter of what you are used to. But I am not hung up on the syntax (nor on the SIP as a whole).

[odersky]

If people have suggestions how to express ADTs without too much fuss and good integration with classes that would also be interesting to discuss!

[lihaoyi]

My current feeling js that a small amount of boilerplate reduction like this isn't worth changing the parser. I don't disagree that this would be covenient, but I disagree with the "let's bake it into the syntax immediately" process that's being proposed.

Shouldn't this be possible with an annotation macro? If not, I think it should be - this is not the only "tweak" to case classes i can imagine - and definitely won't be the only tweak someone wants in future.

Ideally I think major syntactic additions like this should exist in 3rd party compiler plugins or annotation macros that become popular and ubiquitous before being included into the core language.

[martinring]

For me one of the most annoying places where this is commonly an issue are folds:

For your example with Options, consider this simple example:

val options = Seq(None,Some("Blub"))
options.foldLeft(None)(_ orElse _)

which yields

error: type mismatch;
  found   : Option[String]
  required: None.type
       options.foldLeft(None)((x,y) => x orElse y)
                                         ^

However this would not even be fixed by this solution. It would just change the error to:

error: type mismatch;
  found   : Option[String]
  required: Option[Nothing]
         options.foldLeft(None)((x,y) => x orElse y)
                                           ^

So I would also vote for improving type inference instead. Even though I understand, that it is far from trivial (not to speak of decidable)

[ryantheleach]

C# people have no problem parsing it so I guess it's a matter of what you are used to.

It's not having : approximately equal to extend that is the problem, it's having 2 different syntax elements that effectively mean extend, one having special behaviour.

If you were proposing using : instead / as well as extend without the special behaviour I believe people would be more receptive but it would still just be yet another thing for beginners to learn.

It's like the Procedure Syntax debate, but worse, because it has hidden teeth to bite you with.

[megri]

To me, the payoff does not seem worth the complexity added by introducing this new form of extension. An annotation would make more sense. In fact, since the mechanics for pattern matching are well defined, it would make some sense to make the class-specific case an annotation itself.

How about

@adt trait Option[A]
class Some[A](a: A) extends Option[A]
object None extends Option[Nothing]

for abstract data type hierarchies (@adt implies sealed), and

@case class Foo(a: Int, b: String)

It seems the more Scalaesque way, especially as the desired effect is really only seen in the automatically generated methods. Extensible and transparent through meta programming at that.

[ivanopagano]

@megri I don't dislike your idea, apart from the fact that I would be more explicit in requiring the sealed modifier to the @adt trait, adhering to principle of least astonishment.

That said this solves the issue of inferring a proper case class type, but it doesn't reduce the syntactic burden in any way

[odersky]

@megri I am not sure we'll have sufficiently powerful annotation macros to be able to reach into companion objects of child classes of @adt annotated parents to change their apply methods. This requires a lot of meta power. But in general, the proposal to have some modifier/annotation on the base trait only looks appealing.

[odersky]

I conclude that the current proposal does not have enough support, so I'll close it. I think it's best to keep thinking of enums and adts and once someone has something more concrete create a new proposal in a separate issue.