sdk 🚀 - Non-nullable type annotation syntax

Just curious, why was '?' chosen? It seems very counter-intuitive to me, since a question mark implies some kind of optionality. In this case, we rather want to express a restriction. A more assertive exclamation mark would be more appropriate, in my opinion, but I did not study possible grammar conflicts, if that is the reason of this choice.

crelier on 3 Sep 2016

👍3

The question mark means that the type is nullable/optionally null/union of type and Null, and its absence means that the type is just itself.

lrhn on 3 Sep 2016

👍6

Any chance union types will come along with this? They seem to be natural fits.

donny-dont on 3 Sep 2016

👀1 👍1

Questions:

How to declare the type of optional parameters of a function? With question mark or without? By definition, they accept null, so we have to add question marks everywhere?
How to declare that, var is nullable? Will it be "var? x" ? Can we write "dynamic? x". (IMO both forms look stupid: "dynamic" means "whatever" already, and var is a synonym of "dynamic")
Opinion: too many question marks already. Maybe we don't need to make declaration cryptic. e.g. use some keyword instead? (though "nullable" is probably too long)
As soon as we have non-nullable types, we need "is-not-null" assertion operator. E.g. double exclamation mark: expression "x!!" means "I'm sure x is not null, but if it is, throw NPE". This is necessary to prevent monstrosity like:

if (x != null) {
  return x;
} else {
   throw new NullPointerException();
}

tatumizer on 3 Sep 2016

👍4

This seems just an experiment for now. But I'd like to note that there might be needs for addition of APIs to the standard library. For instance, Kotlin has filterNotNull and mapNotNull on nullable sequence or collection, which return one with non-nullable element type. Dart's Stream and collections would be better to have such new APIs. They can definitely be implemented in user codes, but are really clumsy to do so.

dynaxis on 4 Sep 2016

👍2

@lrhn Oh, I see. The title is misleading and the text does not clarify. It should actually be "Nullable type annotation syntax" instead of "Non-nullable type annotation syntax".

crelier on 5 Sep 2016

@tantumizer

"is-not-null" assertion operator.

This is shorter

return x ?? (throw new NullPointerException);

See also #24892

I find x!! a bit cryptic.
If the return type for the function that contains this code is a non-nullable type it should throw in checked mode anyway and make the check and throw redundant.

zoechi on 5 Sep 2016

👍2

@tatumizer

Good questions, some not decided yet.

Optional parameters: If nothing else changes, you'll have to make them nullable by adding a ? to the type. After all, that is their type. We may consider allowing you to make them non-nullable if they have a non-null default value. If we do that, we should probably also change how default parameters work by making an explicit null argument be equivalent to not having the argument at all. There is some design-space here to explore.

For a nullable var, it's not really a problem as we are moving towards Strong Mode where var means to infer the type. You can write dynamic? but since null is already assignable to dynamic, I think dynamic? will just mean the same as dynamic (so we may disallow it entirely, there is no reason to have two ways to write the same thing). Same for Object and Null.

Too many question-marks. Likely true, but they are consistently about null (except for the original ?: conditional operator).

A not null assertion (that is: take nullable value, throw if it's null, otherwise we know the type).
That could just be x as Foo if the type of x is Foo?. We will have to change as to not allow null for non-nullable types, you would then write x as Foo? to get the current behavior.
We are also considering shorthands to quickly coerce a value to an expected type. Maybe x! will check for null if x has type Foo? and it's used in a position where a (non-nullable) Foo is needed. Maybe that's just being too clever by half.

lrhn on 5 Sep 2016

👍1

@zoechi, @lrhn: "return x ?? (throw new NullPointerException)" is technically the same as "return x!!", but human meaning of it is different. There are cases where you want to say: "I'm SURE x is not null, that's it. And if I'm wrong, it's a bug!" Example:

map["foo"]="Hello";
String foo = map["foo"]!!;  // I'm sure it's not null, I've just assigned it!

Do you really believe the following is an equivalent replacement? To me it looks absurd:

map["foo"]="Hello";
String foo = map["foo"]  ??  throw new NullPointerException();

Although the example might look artificial, the phenomenon is quite common. While programming in Kotlin, I found myself using !! more often than I'd like to, for variety of reasons. It would be a good idea to create a summary of these reasons - scenarios when you are forced to declare nullable vars even though you are sure (and can prove!) that before first use, they are already initialized. Writing anything more verbose than x!! would be just adding noise to the program. (Maybe we can discover some deficiencies of the language by analyzing these reasons).

@lrhn: same argument applies to x as Foo. It's a very roundabout way of saying "I'm sure x is not null", though technically the same. (Occam's razor doesn't apply to the language b/c language is about meaning - the notion which is (almost by definition) impossible to formalize).

BTW, just to avoid misunderstanding, x!! is defined in Kotlin as (x != null ? x : throw NullPointerException()) - so if you are considering one exclamation mark for this, that's fine, it's just Kotlin believes two are better. I have no opinion on that.

tatumizer on 5 Sep 2016

If you have

String foo = map["foo"];

then ?? throw new NullPointerException(); is redundant because String already is non-nullable and checked mode should throw.

zoechi on 5 Sep 2016

👍1

@zoechi: it depends on how nullable types are implemented in dart. I think the whole purpose of them is to make things like String foo = map["foo"] illegal. Strong mode compiler must complain here. Anything less than that undermines the whole idea of nullables/non-nullables IMO.

tatumizer on 5 Sep 2016

👍1

As a clarification: if we add only ? (meaning "this type is nullable"), then the non-? type must be non-nullable.
This means that String foo = map["foo"] would statically not be allowed, _unless_ we have implicit downcasts for nullable types, too. (It wouldn't be that awkward, since A is pretty much a subtype of A|Null).

If we don't allow downcast assignments, then there must be a way to go from nullable to non-nullable.
We have the following choices:

as, conditions and ifs promoting the type: String foo = map["foo"] ?? ... basically falls into that category. This requires no change to the language.
a special operator: String foo = map["foo"]!! which checks for nullability. I guess this would also include !!. as in map["foo"]!!.bar(). As Lasse suggests, this could potentially be more general, coercing more than just nullable types. For example, it could replace the implicit downcast.

@tatumizer: I'm definitely interested in the reasons for why you had to use !! more often than you thought.

floitschG on 5 Sep 2016

👍1

@floitschG General description of the situation is "you can prove x!=null, but compiler can't".
Sometimes you can restructure your program so that compiler can figure it out, but as a result, your code becomes worse, so you choose a lesser evil. Because this is subjective, I'm reluctant to use my own code for demonstration - instead, I will use dart's own code. Opened dart sdk code at random place, and in 30 sec got a first example (https://github.com/dart-lang/sdk/blob/master/sdk/lib/collection/linked_list.dart):

E get last {
    if (isEmpty) {
      throw new StateError('No such element');
    }
    return _first._previous; 
}

You know _first is not null here, but compiler doesn't. With !! operator, last line will be written simply as
return _first!!._previous. Without !!, this cute piece will become a mess. I will post more examples later, but the search algo is simple: find any uninitialized var anywhere, and see how it's used later in the code. There's a good chance it's used without explicit "if (x != null)" guard, because it's clear from the context that it can't be null (modulo bugs).

Added later: turns out, isEmpty is used as a guard everywhere in linked_list. You need to write quite a few of !! to pacify compiler. That's basically the effect I wanted to demonstrate: the number of !! in the code is higher than one can naively expect.

tatumizer on 5 Sep 2016

Second source I opened (randomly) was splay_tree. Another program - another "universal guard", this time, it's method _splay, which returns -1 in case _root == null. Throughout the code, program "knows" that when _splay returns non-negative value, it's safe to access _root. But compiler, most likely, won't know it. You will have to use _root!! in every place where you currently have _root.

tatumizer on 5 Sep 2016

Another question: consider generic class Foo. What is the meaning of E? Is it a nullable type? Or only non-nullable type? If the latter, we would need Foo to denote nullable E, but this idea is probably not tenable. So E is nullable. Suppose we want to say that in Foo<E>, E must be NOT nullable. How to write this restriction? Maybe Foo :) Or what?

tatumizer on 6 Sep 2016

Currently the nullability experiment is about syntax only.

In Patrice Chalin's proposal (
https://github.com/chalin/DEP-non-null/blob/master/doc/dep-non-null-AUTOGENERATED-DO-NOT-EDIT.md),
nullability is a property of a type (T? is essentially a shorthand for T | Null: B.3.1), and Foo<C> would be the generic class Foo instantiated
with an actual type argument C, which is a non-null type, assuming that
C is the name of a class. The type argument of Foo<E> where E is a
type variable of an enclosing generic entity such as a generic class Bar<E> .. could be a nullable type or a non-null type, depending on the
instantiation of that generic class of which this is an instance. For
instance, if we consider an instance of Bar<int?> and Bar contains an
occurrence of Foo<E> then E is a nullable type, namely int?. If you
want to make sure that a given type is non-null then you may or may not
have an operator for it: For instance, Foo<E!> could be a Foo of
non-null E, which would in this case be int (int? with the ?
stripped off).

When a type variable can stand for a nullable as well as a non-null type it
is necessary to be a little bit smarter in code generation, such that it
will work in both cases, with good performance.

Because of complexities like this, there are quite a number of issues that
we haven't decided on, so we can't promise anything specific about these
design choices. But currently we won't even promise that there will be
anything like a ! operator for stripping ? off of a given type, we are
just looking for a syntax that will work well for ?.

On Tue, Sep 6, 2016 at 3:14 AM, Tatumizer [email protected] wrote:

Another question: consider generic class Foo. What is the meaning of E?
Is it a nullable type? Or only non-nullable type? If the latter, we would
need Foo to denote nullable E, but this idea is probably not tenable.
So E is nullable. Suppose we want to say that for Foo, we accept generic
type parameter E only if NOT nullable? How to write this restriction? Maybe
Foo :) Or what?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/dart-lang/sdk/issues/27231#issuecomment-244828272,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AJKXUhdiciCSV8iIfIi2pqtBdS5pIByZks5qnL5kgaJpZM4J0Ezv
.

Erik Ernst - Google Danmark ApS
Skt Petri Passage 5, 2 sal, 1165 København K, Denmark
CVR no. 28866984

eernstg on 6 Sep 2016

👍1

Replying to a few random things that weren't already covered:

Just curious, why was '?' chosen?

It's the same syntax used to represent nullable types in C#, Ceylon, Fantom, Kotlin, and Swift.

Any chance union types will come along with this?

We are interested in exploring union types too, but they're a big feature with a lot of consequences, so we aren't working on them right now. There's only so many hours in the day. :)

It should actually be "Nullable type annotation syntax" instead of "Non-nullable type annotation syntax".

I considered that, but the obvious response is that Dart 1.0 already has nullable type annotations—_all_ type annotations are nullable. So this is really about adding a way to express _non_-nullable types. And the way we do that is by adding new syntax for nullable types and changing the existing syntax to mean non-nullable.

I find x!! a bit cryptic.

Me too, but something along these lines might be worth doing. A big part of why we are doing an experiment around non-nullable types is to get answers to usability questions like this. How often do users need to assert that they know something isn't null when the compiler doesn't? We're hoping to implement enough of the static checking to be able to answer that confidently.

String foo = map["foo"]!!;  // I'm sure it's not null, I've just assigned it!

Another part of this experiment is determining how we need to change our core libraries to make it pleasant to work with non-nullable code. In this case, I think Map should support two accessor methods. One returns V? and returns null if the key isn't present. The other returns V and throws if the key isn't found. In this case, you'd use the latter and wouldn't need !!.

There's some interesting API design questions about which of those operations should be [] versus a named method, which is more commonly used, etc. but we need to start trying things out to get a feel for that.

Opened dart sdk code at random place, and in 30 sec got a first example (https://github.com/dart-lang/sdk/blob/master/sdk/lib/collection/linked_list.dart)

That entire class was designed around the idea that E is nullable. Once that assumption is no longer true, there are probably systemic changes you could make to the entire class so that you don't need to sprinkle !! everywhere.

It's also probably true that core low-level collection classes like this will bear the brunt of the manual null checking. They are closer to the metal and need to do things a little more manually. Higher-level application code should hopefully be able to use non-nullable types more easily.

Another question: consider generic class Foo. What is the meaning of E?

I have an answer in mind for this, which I think lines up with Patrice's proposal, but I haven't verified that or written mine down in detail yet. (That's why this issue is about non-nullable _syntax_. :) ).

The short answer is that here, since you have no constraint, E can be instantiated with either a nullable or non-nullable type. If E was constrained to some non-nullable type, it could only be instantiated with a non-nullable type.

Suppose we want to say that in Foo, E must be NOT nullable. How to write this restriction? Maybe Foo

If you want to say E is some specific type, you can give it a constraint and that implicitly constrains it to be non-nullable too, unless the constraint is nullable:

class Point<T extends num> {
  T x, y; // <-- These are non-nullable.
  Point(this.x, this.y);
}

new Point<int>(); // Fine.
new Point<int?>(); // Error! Constraint is non-nullable.

class Pointish<T extends num?> {
  T x, y; // <-- These may or may not be nullable.
  Pointish(this.x, this.y);
}

new Pointish<int>(); // Fine.
new Pointish<int?>(); // Also fine.

I don't currently plan to support a constraint that says, "The type must be non-nullable, but I don't care anything else about the type." I could be wrong, but it doesn't seem very useful to me.

munificent on 7 Sep 2016

👍2

I don't currently plan to support a constraint that says, "The type must be non-nullable, but I don't care anything else about the type

Then what is the meaning of class MyList<T extends Object>, as opposed to class MyList<T extends Object?>
?

tatumizer on 7 Sep 2016

With the approach where T? stands for T | Null, there is no difference between Object and Object?. This is because the set of objects typeable as Object already includes the object typable as Null.

You could invent an operator computing a "type difference" (think set difference), say \, and then you could express "everything except null" as Object \ Null. You might even claim that the ! operator that we might introduce in order to "strip the ? off" of a given type variable could be defined to compute exactly that (you could say that it stands for the type-level function (Type T) => T \ Null). You could also consider one of the other interpretations of nullability in the first place. But, as mentioned, we're currently focusing on finding a good syntax, so it'll be a while before those other topics come up.

eernstg on 7 Sep 2016

That entire class was designed around the idea that E is nullable. Once that assumption is no longer true,...

Sorry, I don't understand. You want to change collection classes to only accept non-nullable type parameters? But there's no syntax for that. You can't say class MyList<E where E is not nullable>. So we are stuck with everything nullable because:

I don't currently plan to support a constraint that says, "The type must be non-nullable, but I don't care anything else about the type." I could be wrong, but it doesn't seem very useful to me.

I'm afraid we have a paradox on our hands.

Regardless, in the examples above I tried to demonstrate common situations where compiler cannot figure out that the variable is non-null already, and forces programmer to write non-null assertion
Please take another look at this:

E get last {
    if (isEmpty) {
      throw new StateError('No such element');
    }
    return _first._previous; 
}

E might be non-nullable all right, but the list has no _first element, because the list is empty.
However, program does NOT check if (_first != null) - it checks isEmpty instead, the rest follows logically. Unfortunately, compiler won't get the logic, you need to use explicit !!: return _first!!._previous.
And my claim was that the situation is typical, you need to write '!!' quite often. (This situation probably can be characterised as "using semantic guards instead of low-level if (x != null) guards makes compiler complain because it's clueless about the meaning of the program").

tatumizer on 7 Sep 2016

You want to change collection classes to only accept non-nullable type parameters?

No, the collections don't have constraints and should accept nullable type arguments.

My point was that the implementation of LinkedList takes for granted a type system where E is always nullable and non-nullability is not expressible. Once that assumption is no longer true, the implementer of LinkedList, might change how it's implemented. Maybe something like:

class LinkedList<E extends LinkedListEntry<E>>
    extends Iterable<E> {
  E? _first;

  E get first {
    if (_first != null) return _first as E;
    throw new StateError('No such element');
  }

  E get last {
    return first._previous;
  }

  // ...
}

And my claim was that the situation is typical, you need to write '!!' quite often.

Yeah, you may be right, though I'd like to see how it actually works out in practice in the context of Dart and it's core libraries.

munificent on 7 Sep 2016

How will compiler behave in this situation:

int index;  // "int" with question mark or without?
int sum = 0;
for (index = 0; index <array.length; index++) {
   sum += array[index];
}

Will compiler force you to write "int? index"?

tatumizer on 8 Sep 2016

Currently, it would need to be int? index, since it's initialized with null.

There are other similar cases where the null initialization would be annoying:

int x;
if (someBool) {
  x = 499;
} else {
  x = 42;
}

If these cases are too common we will investigate definite-assignment strategies, like in Java, and see if they would fix that problem.

floitschG on 8 Sep 2016

If dart can infer the whole type from initializer, maybe it can infer "nullability" part from the lack of initializer? That is, declaration int index; is treated like int? index; (*)

Otherwise, with these question marks everywhere, program will look like a crossword puzzle or something.
This is a real showstopper, much worse than forced !!.

(*) some restrictions apply. E.g. for non-optional parameters of a function, "int" means "int", not "int?".

tatumizer on 9 Sep 2016

That is, declaration int index; is treated like int? index;

We could do that, but I think it's probably better to be explicit here at the expense of a little verbosity. I think it would be a bad thing if adding an initializer to a variable changed how its type annotation was interpreted.

Otherwise, with these question marks everywhere, program will look like a crossword puzzle or something.

The last time I poked around a corpus, I found that ~90% of variables were non-nullable, so I don't think the ? will be _too_ common. But we won't know for sure until we try, which is what this experiment is about.

munificent on 10 Sep 2016

We could do that, but I think it's probably better to be explicit here at the expense of a little verbosity.

So, int x; without initializer will not be allowed? We should always write it as int? x; , right?

The last time I poked around a corpus, I found that ~90% of variables were non-nullable

Did you poke around the corpus of tests? They might have much more of the questionable stuff.

(BTW, here a case to add to our collection of forced !!. Still the same phenomenon of "semantic check", just of a bit different flavor: http://stackoverflow.com/questions/23779678/kotlin-null-safety)

tatumizer on 10 Sep 2016

@tatumizer I worked several years in a language that didn't have null (An ERP system - mostly a database frontend - this might be different for code closer to the metal) and I only missed null when I had to deal with outside code (COM components or similar, but there was some special value provided that was converted to null when passed to outside code).

I liked it a lot to not have to deal with null at all.

I think this is the main point of NNBD that null is rarely needed, otherwise we could keep the current NBD.

A way would be (I think it was mentioned somewhere above) to define default values.

int index;

could be treated like

int index = 0;

This would be similar to now where it is treated like

int index = null;

For String it could be the empty string and for classes an instance created by the default constructor or a named constructor MyClass.default().
But I think just now allowing non-nullable types without initialization would be fine as well.

zoechi on 10 Sep 2016

👍1

@zoechi: please allow me to disagree with your opinion,. Initializing something to irrelevant value instead of null serves just one purpose: sweeping bugs under the carpet for as long as possible. Note that we are discussing the case where you don't have a value for initialization. Why use 0 then, not -1, not 42?

Dart made a correct decision to treat "int x;" as "int x = null". I see null-safety as a cure which might very well be worse than the (non-existent) disease - will lead to ugly code and more bugs, not less. But dart has a problem of self-confidence, afraid of the cries of the Boeotians... Anyway, belief is a reality, there's nothing to be done about it; dart just needs to minimize the damage.

One of the possibilities is: like in java, have special tool (or option in analyzer) to go after nulls.
The problem of java tool: AFAIK, it doesn't allow you to add permanent markup like non-null assertion !!.
My conjecture is that language might do very well without "nullable types" like int?, but with !! operator, which tells the tool to shut up.

tatumizer on 10 Sep 2016

Another paradox:

void main() {
  int x;
  print(x);
  x = 10;
  // proceed with x=10, it won't ever be null again
}

Suppose int x; is allowed without initializer. Then print(x) should be flagged, because otherwise we will observe non-nullable variable having the value null (this argument holds for any function that can accept null as parameter - not only print).

Which means that int x; should not be allowed. Even if we don't have print subsequently called, we may add it later. So we have to write int? x.

Then, regardless of "definite-assignment strategies", the following declaration should be flagged:

void main() {
  int x; // flagged
  x = 10;
  // proceed with x=10, it won't ever be null again
}

I remember having similar problem in kotlin, where my declaration was either flagged or not, depending on whether I call some debug method like printState(). Then I had to change declaration, say, from String to String?, but it never was intended to be nullable in a real context. After a while, I was going nuts: I don't remember any more which String? is a real nullable, and which is a fake one.

It seems currently dart is on its path to require fake nullables. In other words, programmer has to lie about the types, just to keep compiler happy.

tatumizer on 11 Sep 2016

It seems currently dart is on its path to require fake nullables. In other words, programmer has to lie about the types, just to keep compiler happy.

Why do you think that?

int x;

I think if there is not a defined default value (like 0), this can and should not be allowed.

zoechi on 11 Sep 2016

👍1

In dart, there can't be a default value 0 for int, because int is an object. In java, int is not an object, so it always has some value. Java has no choice but initialize it to made-up value (they chose 0, but it could be 42). However, if you use Integer x; - it will be initialized to null, not to Integer(0).

Anyway. the problem of fake nullables is not specific to ints, E.g.

void main() {
  Foo x;  // flagged - wants you to use  Foo? x, which makes it FAKE NULLABLE 
  print(x); // you may, or may not, have this line
  x = new Foo(10);
  // proceed with x -  it won't ever be null again
}

Fake nullables appear also in any scenario where compiler is just not smart enough to figure out the variable is always initialized. To avoid paradoxes like like, dart has no tenable option (IMO) but disallow "Foo x" completely, so you ALWAYS have to lie about the type when you have uninitialized var.

To avoid lying, programmer is tempted to initialize it to fake value, like Foo(42). The paradox comes from the fact that variable initialized to made-up value is much worse than null. With null, program at least crashes with NPE; with made-up value, it proceeds, leading to absurd results somewhere down the road, and there's no indication as to why we arrived at such absurd result.

tatumizer on 11 Sep 2016

I'm not sure why you would expect this code to work?

Foo x;
print(x);

The whole point of having a non-nullable value is to guarantee it has a value; you shouldn't expect to be able to declare one without a value and use it?

And if you take the print out of your example:

Foo x;
x = new Foo(10);

I'm pretty sure the compiler could figure this out; the value is always assigned before it is used and therefore is guaranteed to be valid. I agree that Dart shouldn't just pluck random values out of the air.

I think it's valid to want to be able to declare a non-nullable value and assign it later, but I don't think it's valid to want to use it before it has been. Eg., this code I think should be valid:

int x;
if (something) {
  x = 1;
} else {
  x = 2;
}

Although you could shorten this to int x = something ? 1 : 2; it feels like if the above isn't valid it could be frustrating. However, I would expect an error if you tried to _user the value_ before it was guaranteed to be assigned and I can't think of any obvious ways that wall fall down. If you have a concrete example of where it would, I think it'd be very useful to post :-)

I think even things like try blocks wouldn't be difficult (relatively ;)) for the compiler to understand:

int index;
try {
  thing();
  index = 1;
  callFunc(index); // index is assigned here, this is fine
}
finally {
  callFunc(index); // index is not valid here, it isn't guaranteed to be assigned
}

DanTup on 11 Sep 2016

In Dart, even local variables are considered to be null initialized by its semantics. But In the case of Java, local variables may be left uninitialized and the compiler ensures any use-before-assignment is flagged as illegal. On the other hand, Java's instance/static variables are all null or zero initialized by default. If I remember correctly, it was a decision to avoid unnecessary redundant assignments to local variables and wasn't related to nullability at all (Java has no nullability concept builtin).

For the record, I think such a decision to allow local variables to be left uninitialized is no longer useful for optimization point of view since modern compilers are smart enough to eliminate such unnecessary assignments to local variables.

Back to the main point, (I think) Kotlin followed the Java tradition to not decide on a default value used for non-nullable variables. In Kotlin, you may use lateinit modifier to leave even non-nullable instance variables left uninitialized. But in the case, the compiler inserts runtime checks to use sites to ensure the variables are actually initialized before uses.

I'm not sure which way is better. But I think having a default value doesn't defeat the whole point of having non-nullable type. In the end, we don't get in trouble in most cases by uninitialized (local) variables. If that was the problem, we should've been getting NPE all over the places while using Dart.

UPDATE:
My original point was that requiring local variables to be initialized are not that much a problem since their scopes are quite narrow (Maybe I'm wrong here). For instance variables, I think it's better to require explicit initialization rather than a default value.

But thinking a little bit more, requiring an explicit initialization at declaration site seems better than a default value. It's less complex and less troublesome in many aspects, IMHO.

dynaxis on 11 Sep 2016

I've been writing Kotlin in production for the last 3~4 months. For the !! operator in Kotlin, I find myself avoiding uses of it as much as possible, since I know they entail runtime checks. So I think it's ok to have a little bit more verbose syntax. Overloading x as NonNullable seems good to me.

Also for Kotlin, I think its non-nullable type support is quite well designed and seems working for me at least for now. But we don't know if the design is single way of doing it correctly.

dynaxis on 11 Sep 2016

I'm not sure why you would expect this code to work?

I don't expect anything - I'm confused! Please help me figure it out! Consider a variant of my previous program:

void main() {
  int x;
  print("Hello"); // I'm printing Hello now, not x 
  x = 10;
  // proceed with x=10, it won't ever be null again
}

My questions are:

is it a valid program?
If yes, does x have any value at the time when we print Hello?
if yes, what is this value?

tatumizer on 11 Sep 2016

void main() {
  int x;
  print("Hello"); // I'm printing Hello now, not x 
  x = 10;
  // proceed with x=10, it won't ever be null again
}

Is it a valid Dart program (in a variant of Dart with not-null-by-default nullable types)?

Probably not.
The declaration int x; declares x as not nullable and doesn't initialize it to a value.
In the simplest case, that alone is enough to reject the program.
Bob was mentioning one optimization that could be allowed - use a definite assignment analysis which checks that all _reads_ of the variable is guaranteed to occur after a _write_ to the variable, and allow uninitialized nullable variables as long as they satisfy the analysis and are never read before they have been written to at least once. Initialization counts as a write, so initialized variables are always guaranteed to be read-after-write, but a program like the following can then also be allowed:

int x;
if (foobar()) {
  x = 42;
} else {
  x = 10;
}
print(x);  // use of x guaranteed to occur after write to x.

Your program fails to satisfy this - it reads the variable x before it's written, so it would still be invalid with that optimization.

You can think of "definitely assigned" analysis as a way to allow implicit initialization of non nullable variables with a suitable value. It's extremely confusing if we just do that in general and pick some default value, but if it is always assigned before being read, _it doesn't matter_ which value we use since nobody will ever see it.

lrhn on 11 Sep 2016

@dynaxis

In Kotlin, you may use lateinit modifier to leave even non-nullable instance variables left uninitialized

lateinit? WTF is that? So you have non-nullables, nullables and lateinit non-nullables? But... if you open debugger at a place where it's not late enough for lateinit non-nullable to become real non-nullable, what value will you see?

Anyway, given your knowledge of Kotlin non-nullables, please consider the following dart program:

main() {
   List<int> list = new List<int>(5);
   print (list[0]);
}

My questions:

is it a valid program?
If yes, what gets printed?
if the answer is "null", how would you fix the language/runtime to avoid printing null for nun-nullable thing?

tatumizer on 11 Sep 2016

Is it a valid Dart program (in a variant of Dart with not-null-by-default nullable types)?

Probably not.

This is the only logical answer. Anything else hits the halting problem barrier very quickly, as the "List" example below demonstrates:

main() {
   List<int> list = new List<int>(5);
   print (list[0]); // what does it print?
}

Then language has to unconditionally disallow List<int> list = new List<int>(5);
Or what? (BTW, the formalization of what Lists you allow/disallow might be quite complicated, but this is beside the point).

The alternative is, of course, to make a bold move down the rabbit hole: introduce lateinit, latelateinit etc.

tatumizer on 12 Sep 2016

main() {
   List<int> list = new List<int>(5);
   print (list[0]);
}

That is a problematic case, and we are aware of it.
There are many different possible solutions:

Have something like class List<E?> ... enforce that the type parameter is always converted to nullable, so all lists have nullable element types. Writing new List<int>() and new List<int?>() would mean the same thing (List<int?>).
Don't have that, but List uses E? everywhere inside, so the effect is that elements are nullable (but then List<int> and List<int?> are different but equivalent, that's inconvenient).
Allow different constructors to have different type requirements. Then new List<int>(5) is invalid because that constructor requires a nullable type, but List<int>.filled(5, 0) works.
Add an optional second parameter of type T that is used for initialization: new List<int>(5, 0) works but new List<int>(5) doesn't because null isn't a valid value for that E. That solves this particular problem, but not any other similar problem. And it makes List.filled partly redundant.
Something better that I can't think of right now.

It's definitely not trivial to make everything fit together just right and give a good developer experience. We are looking at what other languages do too, but not everything translates to Dart or to what we want Dart to be. That's why we need to try things out, to see if there are other annoying edge cases like the List constructor that we haven't realized yet.

lrhn on 12 Sep 2016

@lrhn: of 6 solutions you proposed, only the last one looks like a good idea :+1:

And I even know what it could be: forget"null-safety" and return to reason! Because virtually all languages (the exception being C/C++, but we are not talking about that) are null-safe already. Any attempt to dereference null pointer causes NPE, with stack trace beautifully printed for your convenience.

Switching from C to java felt like nirvana. It doesn't get safer than that. Recently, somebody brainwashed me into trying Kotlin: played with it for a while - didn't enjoy it (though got off relatively lightly with just a moderate migraine and a couple of minor psychotic episodes, all of which cured without a trace upon switching back to java)

After kotlin, java feels like nirvana again!

Having experienced the effects of null-safety firsthand, my advice to other experimenters: please take it easy! Try to keep your stress under control, the day will come - your ordeal will be over, one way or the other!

To those whose management decides to move to null-safer languages: you have to realize that you can't afford having NPE ever again, not even once. Because for higher-ups, it would look like you managed to get NPE in a language which makes NPE impossible (according to their views and based on advertisements). So your personality may become a subject of great interest.

Therefore, forget about question and exclamation marks. They will get you confused and cause more NPEs, not less. The cure is, of course, to initialize all your values right from the start (as one correspondent suggested earlier in this thread), but to minimize the adverse effect, you have to initialize them to fake values that can't be confused with the real ones:

all ints - to something guaranteed to exceed the size of any array you have
all Strings - to the first chapter of Boccacio's Decameron
for every class Foo you have, create a subclass Foo$$$ that overrides every method and instead throws incomprehensible (for anyone but you) exception in no way resembling NPE: specifically, avoid the terms "null", "pointer", "uninitialized" and their synonyms. Use "Foo foo = new Foo$$$();" in all declarations;

This way, you get rid of all Billion-Dollar mistakes, still ensuring enough info printed for debugging :+1:

tatumizer on 12 Sep 2016

Now let's talk about semantics of null-safe. What is the meaning of "int? x" ? What is the meaning of "x!!"?

Currently (as of today - before any null-safe extras are introduced), when we write "int x", we don't specify which values it can accept. Is null a valid value? To answer this question, we have to think about the meaning of x (computer can't do it for us). But null is not any special among other values. We can as well ask: is 42 a valid value? It might be, or not. To answer the question, we have to think about the meaning of x. Same question - same answer.

Now, what happens with the addition of question types? When we write "int? x", what does it mean for us, in practical terms? Looks like it's a positive statement: yes, null is a valid value! Go ahead and assign it! But it's a wrong interpretation. It might mean simply that for technical reasons related to limits of compiler's mind, we have to declare it this way, but we don't really mean it. So the declaration should really look like:

int? x; // don't look at nullable annotation! not really nullable!!! 
          // (but happens to be null for now)

And what is the meaning of "!!" then?

int? x; // assigned somewhere
...
int z= x!!+5; // need to use !! b/c compiler cannot prove it's assigned.

We write x!! that says: throw NPE if x==null. But if x==null, runtime would throw NPE in any case, even without "null-safe" extras! The language forces you to add fake "?" annotation, and then "!!" annotation, for them to just cancel out! OMG!

To appreciate the stupidity of all this, you have to experience it personally. It's difficult to describe it in words. I traveled to another world (kotlin), saw weird things, but my language is not good enough to convey my experiences. You have to go and see it first-hand. (in fact, my notes just scratch the surface. It's all dynamic: you refactor the program - and suddenly have to change type to nullable, for no logical reason).

Curious case of mass delusion.

tatumizer on 13 Sep 2016

when we write "int x" .. Is null a valid value? .. But null is not any special among other values.

I agree that it is very useful to discuss the semantics conceptually in order to achieve meaningful and consistent technical definitions. But starting from the technical level we have two clear answers here: In current Dart, null is a valid value for x, and in a future Dart where nullability support has been added, it is not. Conceptually, I think both choices are meaningful, you just need to know which one is in effect, and you need to use that when reasoning about potential NPEs.

The value of null is indeed special, because the spec singles out that value, and also because it is used to model "does not exist", which is different from any particular regular value of, say, an int variable.

When we write "int? x" .. It might mean simply that for technical reasons .. we have to declare it this way

In that case I'd suggest that the developer refactors the code slightly, such that it is possible to use a technical device (the type int rather than int?) which reflects the intended "meaning" of the variable.

It may be a useful language enhancement to use definite assignment analysis, such that int x; can be allowed, given that x cannot possibly be evaluated before some non-null value has been assigned. It may also be a bad choice to do this, because it allows for code which is quite brittle ("I just changed blah-blah which can't possibly make a difference, so why doesn't it compile any more ... Ah maybe this method call can throw!", etc).

eernstg on 15 Sep 2016

👍1

@eernstg "refactoring slightly" may not be enough, but this is just the tip of the iceberg. More interesting case is that of collections. Let's consider the question: is List<int> and List<int?> the same type or different. 2 possibilities;

Same type. Then compiler has to be pessimistic: whenever we write int x=list[i] (no matter how the list was declared - as List<int> or List<int?>) - compiler will insist on adding a guard, which leads to a ritual of adding "!!" in every int x=list[i]!! (note !!). Also, on write, compiler has to accept nulls, so list[i]=null is accepted. Absurd.
Different types. Then consider List<Object> (or List<dynamic> - doesn't matter here):

List<Object> objList = ... // initialized somehow
List<int> intList = objList as List<int> // compiler cannot complain here!
int x = intList[0]; // compiler cannot complain here! No exclamation marks!

Note that from compiler's viewpoint, it's all legit, The check for null in line 3 is postponed until runtime
So subscript intList[0] will fail in runtime if intList[0] happens to be null.

But we still can assign nulls to intList elements, even in runtime! There's another reference to the list (objList), which we can use as in objList[0]=null - the assignment is valid both in compile and runtime. And actually, if you print runtimeType of intList, it will be List<Object>, not List<int>

So what exactly is achieved by null-safe syntax? We have only runtime-time diagnostics, which we have anyway on any dereference of null - sure, this dereference may happen a bit later, but statistically, it makes no difference, because most of the time, we do some operation over elements anyway: sum += list[i] will give you NPE on null even in current dart (obviously. dart is already null-safe).

Possible counter-argument can be: conversion from List<Object> to List<int> must be rare. But it's not.
E.g. you have JSON object from JSON parser. In this object, every list is a List<dynamic>. But if you happen to know that in fact it's supposed to be List, you can check all values for "non-null int" in a loop as a precaution and then cast to List. Very likely scenario.

The impression is that by adding a lot of punctuation and rituals and special cases and whatnot, are not much better off, just more confused.

(The situation is in no way specific to List. It occurs in every collection, and in every parameterized type in general).

What say you?

tatumizer on 15 Sep 2016

The optional nullability is intended for Dart 2 which will also has strong mode-like type guarantees.
That means that you can't assign a List<Object> to a variable expecting List<int>, nor can you cast it using as. The expression objList as List<int> must throw if objList is-not-a List<int>.
A List<int?> is-not-a List<int> (it's the other way around due to Dart's covariant generics).

That means that the line int x = intList[0]; will only be reached in situations where intList contains an actual List<int> which also guarantees that its operator[] returns int and not null.

It is true that if your program is actually flawed, adding extra type checks will just make the program throw at a different and earlier point, possibly by completely rejecting the program.
Your explicit down-cast in this example can't be caught by the compiler, you are doing everything correct in order to tell the compiler that you know what you are doing, but it will add a check at runtime to ensure that you don't break the type invariants.

The List you get from JSON decoding is hard to handle in general. You might actually know that it is a list of integers, but it will not be a List<int> anyway. You can't cast yourself into that, you have to actually wrap the list or create a new list (new List<int>.from(jsonList) or new DelegatingList.typed<int>(jsonList)) which will both do runtime checks - either at creation time or at access time - to ensure that you will never read a non-int out of the resulting list.
With a little luck, Dart 2 will make it even easier to convert to a different type of list - but you will still get runtime checks because the static checks can't see that it's safe, and it must either be safe or it must throw.

lrhn on 15 Sep 2016

👍1

Agreeing completely with Lasse, I have just one more comment: We may do something (e.g., change the placement of the type Null in the type hierarchy) such that Object is no longer the same thing as Object?, which is needed in order to create general data structures whose element type is constrained to be a non-null type (if the type argument bound is Object and that's the same thing as Object? then int? and all other nullable types are acceptable type arguments).

eernstg on 15 Sep 2016

👍1

It's hard to argue against features that might, or might not be, included in future dart versions.

The only thing I can add at this point is an obvious observation that there exists an optimal level of safety. Such a level that more safety makes you less safe. Finding this optimal level is difficult, and can be done only experimentally. Wearing seat belt make you (statistically!) more safe. Taking strong medications for a trivial ailment may make you less safe and lead to real issues, for which you might need more medications etc. There're whole industries built around false safety.

Speaking about null-safety, I don't know how many readers of this thread ever dealt with really non-null-safe environment. The example is a small microprocessor that doesn't trigger hardware interrupts while dereferencing pointers (unless we point above available memory). In such environment, if pointer happens to be 0, program proceeds as if nothing had happened, reading the content of memory location 0. It doesn't segfault. This is non null-safe, but in this environment nothing is safe.

For more modern architectures, if you write in C, program segfaults on null. This is better than nothing,
but still leaves a window wide open for bugs related to pointer arithmetic, so you can read wrong location and proceed as if nothing had happened. These bugs are hard ones.

My point is: dart is nothing like this! It's already safe. It's flexible and convenient. IMO, it already reached optimal level of safety. Adding more safety will add more complexity, and complexity breeds bugs! Instead of trivial bugs related to occasional NPE (very easy to fix), you might start having more subtle bugs. Human mind has limits, can't bear too much complexity. Any experimental data to show this "extra safety" makes you safer? I haven't seen any!

Dart it its current form is still missing some features, but they have nothing to do with nulls..
E.g., there's no convenient format for object literals. Look at flutter examples, they were designed to showcase the virtues of UI programming in dart. Lots of unreadable object literals!

In any case, the lack of popularity of dart has nothing to do with this all. It's just the language failed to find a niche for itself. Maybe flutter will help. Or Fuchsia. I'm just afraid dart 2 will destroy everything good that was achieved in dart 1, which IMO happened accidentally, while pursuing wrong goals, but the result, all of a sudden, was good!

tatumizer on 15 Sep 2016

👎1 👍1

I find it very cumbersome in Dart (and other languages) to always have to deal with nulls in every method, when there is no use for them at all. All values that I access have to be checked for null before any property or method is accessed.

Non-nullable types are like implicit asserts that free the developer from tedious work.
The analyzer already does the checks that otherwise the developer had to do with asserts and is-not-null checks. It's just more of what types already do for us, just with more fine-grained control.

If you actually want allow nulls, then all you have to do is adding ? to the type names.
In my experience, situations where one actually wants nullable types are quite rare.

Non-nullable types are not about false safety. They are about being able to expressing intent with less code.
Just because there are still other sources for bugs, doesn't mean non-nullable doesn't provide benefits. You could similarly argue that the analyzer is redundant because even when you fix all issues it shows, your code can still have bugs.

In my opinion non-nullability is not about adding a feature but removing a bug that was introduced decades ago without a convincing reason and repeated just because people were already used to it.

zoechi on 15 Sep 2016

👍1

And besides, if you like all these features discussed last year for Dart 2, you can have them right now - just take swift and use it. Swift is not a bad language, it's just for the people with different mindset. What's the point of creating a clone virtually indistinguishable from the original?

tatumizer on 15 Sep 2016

I find it very cumbersome in Dart (and other languages) to always have to deal with nulls in every method, when there is no use for them at all. All values that I access have to be checked for null before any property or method is accessed.

Always have to deal with nulls? Not sure what you mean. If you have function with 3 parameters x, y, z,
do you start your function with something like this?

if (x==null) {
   throw new NullPointerException();
}  
if (y == null) {
   throw new NullPointerException();
}
if (y == null) {
   throw new NullPointerException();
}

Please check out the source code in dart core - can you see the pattern like this anywhere?

Strangely, I never "deal" with nulls in a situation where the only choice is NPE. Let runtime throw NPE by itself. That's what null-safety really is.

tatumizer on 15 Sep 2016

do you start your function with something like this?

and writing this is pure nonsense. Just to ensure I'm null-safe when I never asked for null in the first place

zoechi on 15 Sep 2016

👍1

Sorry, I don't understand... You really do write something like this???
Never mind.

tatumizer on 15 Sep 2016

Seems like this thread is going in circles, but my two cents...

@tatumizer
IMO, it already reached optimal level of safety. Adding more safety will add more complexity, and complexity breeds bugs! Instead of trivial bugs related to occasional NPE (very easy to fix)

I'd love to know what the "very easy fix" is for NPEs. In my experience, it's a game of whack-a-mole. Your function fails because it was called with a null; what are your options?

Put null checks into your function - this is not a fix, it's too late, it's run-time and you've got invalid data, you can't do what you need to do so you crash in front of the user (or gracefully fail to do what you were supposed to and probably leave something in a bad state)
Look at the stack and fix the place that was incorrectly providing a null. What do you do about the other 100 places that called your function? Wait until you get an NPE exception stack trace for each one? Doesn't it seem a bit crap to require a bug to occur before you can fix it? Is it reasonable to expect a developer to manually review all code interactions to ensure no nulls can exist where they're not wanted?

In most cases null is just not valid - your code should not compile and you should not be able to ship it. Failing at runtime is just not a reasonable thing for an application to do.

There is no reason you should not be able to write your code such that it doesn't require nullable fields where they don't exist. If you can't explain it to the compiler, how can you expect the computer to run it?

@zoechi
Non-nullable types are like implicit asserts that free the developer from tedious work.

I'd say it goes much further than this; asserts are runtime crashes; non-null types give us these guarantees at dev time!

IMO there is absolutely no reason to force people have to deal with nulls in a modern language. How will we ever write software that doesn't crash in front of a user if we can't do something as simple as write an add(int, int) function knowing it can only be called with ints? nulls are an enormous hole on our type system and there's no reason we shouldn't be able to opt-out of them.

IMO if we want to improve the quality of software (and we should, because it's generally poor) we really need to go much further (give contracts and dependent types!) but I can't see how we'll ever get there while nulls are still kicking around :(

DanTup on 16 Sep 2016

What do you do about the other 100 places that called your function?

Who wrote the code for these 100 places bombarding you with nulls? It's what? denial-of-service attack? Or you copied/pasted same bug in 100 places?

Not sure how people write programs these days.

Compiler takes care of everything: checks for nulls and throws NPE; checks array bounds on every access; for each type cast, checks the types; checks the number and types of parameters; etc. etc. You guys already live like Kings, and still keep complaining.

You know what: try Ada - you may like it. Come back in a couple of years and tell about your experiences.

tatumizer on 16 Sep 2016

... but I can't see how we'll ever get there while nulls are still kicking around :(

They are not kicking around in Kotlin. Instead, it's Kotlin who kicks you around with its null-safety features,

During my short tenure in Kotlin, I picked up more fights with the compiler than with all compilers combined during my 40-year-long career (for the lack of better word). It's not only about null-safety (though it played a major role). While we are at it, I'd like to mention another cool feature reportedly planned for dart 2: long return.

If you never encountered "long return" concept before, here's what I learned: you can write "return x" in a function, and it really returns x, just not to the caller, but to caller's grandfather! I was writing music-generating app, and after a small refactoring, it suddenly started playing weird notes in funky combinations, in such range that only dolphin can hear. What the hell was going on? Simple: long return!

Anyhow, it all ended badly. Tired of fighting, I had to stop the whole effort. Too bad, it was a promising project. Safety comes at a steep price.

tatumizer on 16 Sep 2016

Maybe dart-misc is a better place to continue this discussion?

@tatumizer

What do you do about the other 100 places that called your function?

Who wrote the code for these 100 places bombarding you with nulls? It's what? denial-of-service attack?
Or you copied/pasted same bug in 100 places?

Not sure how people write programs these days.

If the tools don't allow you to forbid null, it's all too easy to introduce this, especially on a huge, evolving codebase. One day something is guaranteed not to be null; the next day it needs to be deletable (eg. customer information). The type system gives us no ability to know all of the places we're using an order/booking/whatever and assuming customer is non-null.

And we don't always have the luxury of writing our own code - the team I lead has almost 1m lines of code in our major project most of which was written by developers that no longer work here and an outsourced firm; the quality is not excellent but it's what we have. Sure, it shouldn't have ended up like this, but I can't change the past. Making changes and improvements is really tough when the foundations are bad. Non-nulls would give us enormous guarantees that would help us improve the quality.

Compiler takes care of everything: checks for nulls and throws NPE; checks array bounds on every access; for each type cast, checks the types; checks the number and types of parameters; etc. etc. You guys already live like Kings, and still keep complaining.

This is poor logic. Just because things were even worse before does not mean we should not make them better. NPEs are not a solution, they are a band-aid. Tell your end user that "at least that error you got didn't crash the whole server" and they don't care; you've failed. Runtime failures are failures; we need to be stopping these bugs before shipping. Sure, some devs might not care about this, but some do and we should have that option. Options are good. If you don't want something, don't use it. But don't assume that because you don't want it it's not beneficial to anyone!

They are not kicking around in Kotlin. Instead, it's Kotlin who kicks you around with its null-safety features,

You've said this many times, and I'm sure you're not making it up. However, I haven't seen any concrete examples of the problems so it's hard to really have an opinion on. If you have !!(?) all over your code where you shouldn't, search for it and post some examples; maybe they're solvable problems? (though I think posting to dart-misc would be better than keeping this issue growing!)

DanTup on 16 Sep 2016

👍2

Making changes and improvements is really tough when the foundations are bad

Here you nailed it! The problem is not nulls, and it's not insufficiently strong typing - it's lousy code that doesn't make sense. And lousy code is 99.9% of the "corpus" I have to deal with, and probably same with you, and with almost everybody working "in the wild". (Maybe google's internal code is better, but I'm not sure).

To write a good code, you have to be able to iterate. Normally, it takes me 4-5 iterations to produce something that at least makes a bit of sense. No amount of advance thinking helps - only when I start writing, I start learning to discover the structure of the problem. I need to kind of live in it to feel what's going on.

By building Gulag in the type system, the language makes this iterative process difficult. Compiler picks at trivialities. - Aha! You say this var "x" is not null, but it is! - No, it is not! You see I assigned it here! -No you didn't. And besides, this "parent" var you use over there - it may be null! -It may, but I use guard "!isRoot" before accessing - So what? How it becomes non-null if you use a random guard? etc. This quarrelling goes on and on and never ends.

Even while writing tests (for your own use) you have to jump through the hoops.

The reason to discuss the problem here (not in the mailing list) is that you can easily reconstruct the whole history of argumentation by looking back in the same thread.

It all started with seemingly innocuous proposal of allowing non-nullable types by simply adding "?" to type name (please check the first post to make sure I'm not making it up).

Then it turned out, "?" is not enough, we need non-null assertion "!!", it won't work without it.

Then we discovered that there's a problem of creating an empty list of fixed size

Then we found out that without complete separation of nullable types from non-nullable types we quickly run into paradoxes.

So by now we know that dart (2?) has to support draconian typing just to make things more or less consistent, all for the sake of non-nullable types!

But there's a bit of a problem even in dart 2, b/c then we can't really deal with JSON objects without creating copies of all our lists and maps.

And it certainly doesn't stop here. More and more functions and concepts have to be developed. Non-nullability infects all aspects of the language, barriers have to be erected everywhere.

Now let's recall where dart started. It was based on certain philosophy of static vs dynamic typing. Please revisit this article (written before dart, but it outlines the same ideas):
http://bracha.org/pluggableTypesPosition.pdf

Please start reading at paragraph "The disadvantages of mandatory typing are not as universally acknowledged". Pay special attention to the term "brittle". This word certainly comes to mind when you are trying to iterate over your design. It seems dart had a change of heart or something? because features planned for dart 2 are not consistent with this at all. It will require a whole ritual to just be able to handle JSON.

Interestingly, "strong typing" as of dart 1 doesn't stand in your way. It provides benefits, with almost no downside (at least in my experience) - because when you want more freedom, you can opt out of strong checks - e.g by using dynamic.

For those complaining about the need to check input parameters for nulls: you don't need much beyond assertion here. E.g. it can be easy as x!!. Operator "!!" won't do much harm. But if you want to check input parameters, this is not enough. E.g. you want to verify that x>0, or x!=42, or something. Maybe special kind of standard "runtime assertion" can be added to make it more concise. E.g. check(x>0, "x should be positive"). Of course, it's not hard to write your own function like this, but it would be better (IMO) if runtime assertions for ArgumentError come out of box.

(BTW, where's Gilad right now? Still with dart?)

tatumizer on 16 Sep 2016

👎1

@zoechi every time when you :-1: my post without ever bothering to read it, I will go and :+1: your post - just to make a point that every honest post, however misguided, deserves praise.

tatumizer on 16 Sep 2016

👎1

@tantumizer I don't think it's the right place for this kind of discussion and your arguments are IMHO just more of what you already explained as your opinion and I could only reply again with my opinion which would probably lead to a stackoverflow (Bob locks this issue)

zoechi on 16 Sep 2016

👍2

@lrhn

A List is-not-a List (it's the other way around due to Dart's covariant generics).

This "the other way around" leads to the same paradox. We have List, and we pass it to a function that accepts List as parameter. Now, the function is free to assign list[0]=null.

Treating Null is not-an-object, as @eernstg suggests, doesn't solve this problem, because anyway, we must be able to convert List into List (*)

There are, of course, ways to resolve the paradox - e.g. by adding "mutable" keyword, treating list as a value, implementing copy-on-write, etc (see swift). I have a conjecture that all this trickery in swift is a direct ramification of null-safety. Not sure - please opine.

(*) the root of the problem is that List essentially works as a pointer. In other words, though dart doesn't have pointers, List with a single element essentially models a pointer. So we DO have "pointers". What's going on in the example above - we cast pointer to int to pointer to int?, which, of course, quickly leads to absurd.

@zoechi : maybe you know how to design null-safe typesystem that doesn't lead to paradoxes and confusion? Your opinion is welcome, too!

tatumizer on 17 Sep 2016

To make a paradox more apparent, here's another example (seemingly unrelated to nulls):

class Fruit {}
class Orange extends Fruit {}
class Apple extends Fruit {}
class Plum extends Fruit {}

applesToOranges(List<Fruit> list) {
  for (int i=0; i<list.length; i++) {
    if (list[i] is Apple) list[i] = new Orange();
  }
}
void main() {
  List<Apple> list=new List<Apple>.filled(100, new Apple());
  applesToOranges(list);
  print(list[0]);
}

Is it a correct program? Do we expect it to fail?
If not, change Apple to int, Fruit to int?, and list[i]=new Orange() to list[i]=null - it shouldn't fail either.

tatumizer on 19 Sep 2016

Dart used to have covariant generic classes. It still does, but it used to too.

That means that the []= operator of List isn't type-safe, nor is add. That's nothing new. In Dart 2 there will have to be an implicit check of the parameter, so you can think of it as

void operator[]=(int index, Object value) {
  value as E;
  // assign value at index
}

That means that if you do:

List<Fruit> x = new List<Apple>();
x.add(new Orange());

you get a runtime error when you try to add an Orange to a List<Apple>.

So, your program is "correct" in the sense that the type system won't catch the error. We expect it to fail at runtime. That's nothing new, Dart 1 has the same problem in checked mode, and Dart 2/strong mode is fairly close to forced strong mode (with some extra bells and whistles, and a few stronger requirements around functions).

And yes, you can assign a List<int> to a variable of type List<int?> due to covariant class generics, but it will throw at runtime if you try to add null to it.

The real problem is that generics classes are covariant in the type parameter, but the type parameter may occur contravariantly in some methods. Those methods are wrong. They are also incredibly convenient. As a compromise, _all_ such methods (add, []=, etc) need a runtime check, and they should get one automatically in Dart 2.

If you look at Java, they picked a much more precise approach where you can assign a List<Apple> to List<? extends Fruit>, but not to a List<Fruit>. You can't call add on the former, so that problem is prevented, at the cost of some complexity in the type system.

lrhn on 19 Sep 2016

@tatumizer

A List<int?> is-not-a List<int> (it's the other way around due to Dart's covariant generics).

This "the other way around" leads to the same paradox. We have List<int>, and we pass it to a
function that accepts List<int?> as parameter. Now, the function is free to assign list[0]=null.

This is again covariant generics and it has nothing to do with nullability per se: You pass a subtype List<int> into a context where a supertype List<int?> is expected, which is fine by any type system. However, covariant generics makes contravariantly positioned occurrences of the type argument unsafe. For instance, list[0] = null provides null as the argument where the statically known argument type is int? but the runtime argument type is int, and that will fail because it's (actually, though not visibly) a downcast.

Treating Null is not-an-object, as @eernstg suggests, doesn't solve this problem, because
anyway, we must be able to convert List into List (*)

I don't expect the treatment of null as not-an-Object to make any difference here, because T will be a subtype of T | Null, no matter where in the type hierarchy Null is sitting. This also means that List<int> will be a subtype of List<int?> as long as List is covariant in its type argument.

(It's a separate discussion whether we want to allow a class to declare that a given type argument is invariant or contravariant, in addition to the covariance which is chosen by default. Covariant generics was chosen for Dart because the designers wanted to give programmers simpler types to work with --- in particular, simpler than Java wildcards --- in return for this particular kind of unsafety. We may or may not wish to strike that balance slightly differently in the future, such that developers can make this trade-off in more than one way; this could allow things like the list[0] = null example above to be a compile-time error rather than a run-time error).

There are, of course, ways to resolve the paradox - e.g. by adding "mutable" keyword,
treating list as a value, implementing copy-on-write, etc (see swift).

I believe Swift arrays and dictionaries are covariant and user-defined types are invariant, which is probably too weird for Dart. ;-) But, sure, there is no end to the available language designs in this domain, with associated trade-offs.

I have a conjecture that all this trickery in swift is a
direct ramification of null-safety. Not sure - please opine.

I wouldn't expect null-safety to sit at the core of this, I would expect null-safety to be just one among many different cases where variance arises. With a little luck, this means that anything that makes variance work well will be good for handling null-safety as well.

eernstg on 19 Sep 2016

@lrhn, @eernstg: So it comes down to runtime check, right? When we write list[i]=new Orange(), compiler should generate code that checks whether runtime type of value is assignment-compatible with runtime type of the list. In general, both runtime types (of list and of value) become known only in runtime.

Compiler faces a problem here: given two types T and T1 (known only in runtime), find out whether T1 is assignable to T. This is a hard problem - performance penalty might be way too high. Notice that the check has to be done on each value being assigned, like in the example

copy10Frutis(List<Fruit> from, List<Fruit> to) {
   assert(from length>=10 && to.length>=10);
   for (int i=0; i<10; i++) to[i] = from[i];  // value-by-value runtime type check!

}

Performance may degrade by ... I don't know by how much. Order of magnitude, maybe even two.
Especially in case of AOT compiler. Are you ready to pay the price so high?

Plus, of course this "actually, though not visibly" part.

I read (in HN) that users of swift are now complaining about the lack of dynamism. In other words, the type system is safe and sound, but you have problems with actually writing anything. This is 3rd problem - language becomes inconvenient, which (I'm repeating myself, sorry) leads to complexity and more bugs, and to general dissatisfaction of a programmer (which happened with me in case of kotlin. I found out that I prefer java).

All together, this means that we replace one small (IMO) problem with a bunch of harder ones.

But ultimately, whether it's worth it or not, is a judgement call. My judgement is that current dart is WAY more convenient.

Added later: sudden performance degradation after seemingly innocuous modification of code is a bug, too, and very bad one at that (IMO).

Even later: haha, just found a confirmation of my theory: "I hoped that Rust’s strong static typing would help reduce bugs, but sadly, it didn’t have much effect" - from https://medium.com/@robertgrosse/my-experience-rewriting-enjarify-in-rust-723089b406ad#.wdxskueoh

Note that this confession comes from the guy who rewrote the same program 3 times, and rust was the 3rd one, so most of logical bugs should have been eliminated already!

tatumizer on 19 Sep 2016

.. So it comes down to runtime check, right? When we write list[i]=new Orange(), compiler should
generate code that checks .. runtime types (of list and of value) ..

True. Considering the example in the comment above, let's call the actual type argument of the list T. Then we only know that T <: Fruit and there is no guarantee that Orange <: T, so a runtime check must be performed at list[i]=new Orange().

However, there's nothing stopping a compiler from generating a customized version of the method which only gets called when the actual type of list is List<Fruit> (exactly, i.e., it doesn't get called when list is a List<Apple> or indeed List<S> for any S which is different from Fruit). In that version of the compiled method it is known to be safe to assign an Orange to any entry, so no dynamic check is needed in the loop. This effectively means that a check per entry is replaced by a check per invocation of applesToOranges.

Compiler faces a problem here: given two types T and T1 (known only in runtime), find out whether
T1 is assignable to T. This is a hard problem - performance penalty might be way too high. Notice that
the check has to be done on each value being assigned, like in the example
  copy10Frutis(List<Fruit> from, List<Fruit> to) {
     assert(from length>=10 && to.length>=10);
     for (int i=0; i<10; i++) to[i] = from[i];  // value-by-value runtime type check!
  }

This is more tricky so you probably shouldn't expect a compiler to be able to do much about it. However, if we assume that it tries really hard then you could do the following: The test that the value of from[i] is acceptable as an argument to to.[]= can be abstracted: If the type argument of from is a subtype Tfrom of the type argument of to, Tto then the type of from[i], Ti is known to satisfy Ti <: Tto, so since Tto <: Tfrom we also have Ti <: Tfrom, so there is no need to check it. So we can start by checking Tto <: Tfrom, and if that's true then we run the loop with no checks; otherwise we run the loop with the check.

But, of course, a compiler is not likely to take that many steps in order to find a very special optimization opportunity. It is a trade-off: Do we want to support an invariant modifier on formal type parameters? If we do, there will be fewer runtime checks and there will be fewer reuse opportunities.

eernstg on 20 Sep 2016

@eernstg The takeaway is: it's not a trivial problem, and we agree on that.
Certainly interesting thing to think about, but practical gains are uncertain.

And because they are uncertain, we don't have justification for the feature other than "this is an interesting math problem".

I'm still going with the idea that there exists an optimal level of safety. Notion of "optimality" takes into consideration various components (other than safety itself):

Logistics of providing extra level of safety
Inconveniences caused by extra level of safety
Extra costs associated with providing extra level of safety
Potential misunderstanding of the safety mechanisms by user, leading to the loss of safety
etc.

My conjecture is that proposed measures overshoot the point of optimality by a mile.

Thanks for the discussion anyway!

/"too much safety is abhorrent to the nature of a human being." - Agatha Christie

tatumizer on 20 Sep 2016

Thank you, too! Your input is helpful in keeping us aware of the costs of more strict checking. Dart is a language where completely strict static checking has never been the goal, so we do have to be careful.

eernstg on 21 Sep 2016

And because they are uncertain, we don't have justification for the feature

Fortunately, we don't need to justify the feature, we just need to justify _running an experiment_ for the feature, which is all we're currently doing. The results of that experiment will help justify whether or not we should land it. :)

munificent on 21 Sep 2016

Then you might also find this info helpful:
https://courses.cs.washington.edu/courses/cse590n/10au/hanenberg-oopsla2010.pdf

It's not about nulls specifically, but rather about strong typing in general.
BTW, Erik is listed there as one of the contributors (see Acknowledgements section) :)

tatumizer on 21 Sep 2016

rather about strong typing in general.

That is out of scope for Dart in general, and _certainly_ out of scope for this issue, which is about adding ? syntax. Dart has already decided to have a static type system.

munificent on 22 Sep 2016

If you are looking for edge cases, here's a curious one. Currently, null can be interpreted as boolean value. That is, compiler (in strong mode) doesn't complain at print(null? 1 : 2);. Spec says it should fail in checked mode. Otherwise, null behaves as if we have extra method "booleanValue" in Null, and the value is "false". Not sure what to make of that. Feels like paradox. You might be aware of it already though.

tatumizer on 24 Sep 2016

Definitely aware of it.
The current Dart treats all type annotations as "nullable", so null can go anywhere, and nothing prevents you from trying to use a null value by calling a method on it or using it as a test condition. That's basically the two ways you can use a value in Dart, and "using as a test condition" stands out because it doesn't obviously lead to a "no such method" error because there is no method involved, unlike all other uses of null. It's still an error in checked mode because boolean conversion is defined to explicitly check for not being a bool.

If Dart gets non-nullable types, then the valid type of a test condition will likely be just (non-nullable) bool. You'll probably also not be able to do foo.bar() directly if foo has a nullable type, you need to do if (foo != null) foo.bar(); or (hopefully) something shorter like foo!.bar().

lrhn on 24 Sep 2016

It's still an error in checked mode because boolean conversion is defined to explicitly check for not being a bool.

Typo: ... for not being a null (sorry for nitpicking)

tatumizer on 24 Sep 2016

True, it only has to explicitly check for null because all other non-bools were already caught by the type parameter on the conversion closure:

(bool v){
  assert(v != null);
  return identical(v, true);
}(o)

Effectively boolean conversion does check against the value not being a bool, it just splits it into two parts: the bool type on v and the != null in the assert.
It could have been:

(b) {
  assert(v is bool);
  return identical(v, true);
}(o)

instead, it would just give a different error.

lrhn on 24 Sep 2016

Oh, now I see. Second variant is more explicit, avoids 2-stage logic.

tatumizer on 24 Sep 2016

How about this:

instead of targeting nulls, target uninitialized variables.
If compiler suspects the access before initialization - warning. (Explicitly assigning null counts as initialization, too)
Therefore, no question marks in types. Compiler just checks whether everything is initialized
to silence false warnings, allow exclamation mark

What percentage of NPEs is due to uninitialized variables? If it's 50% (very likely), we get a useful feature almost for free. I think it's more than 50%.

tatumizer on 26 Sep 2016

What percentage of NPEs is due to uninitialized variables? If it's 50% (very likely), we get a useful feature almost for free. I think it's more than 50%.

This percentage is nothing like 50% in any of our (C#) codebase. There are so many sources null; different expectations between empty collections and null, unobvious optional/mandatory parts of an object graph.

If you can't encode in your types that null is invalid, how should a caller of your function know? Runtime? What if you change it later to no longer be valid - a breaking change only found at runtime (and only when you pass a null)? Your rules make it unreasonably difficult to make a previously valid null no longer valid.

One of the selling points of Dart is to help you find mistakes at dev time, not let your user find them at runtime. Software is too complex to expect to hit every permutation of code paths and state during testing (not to mention more expensive than finding at "compile" time).

I'm not sure why you're so against non-null but you always have the option not to use it. There are lots of people that REALLY want it (blame our bad inherited codebase if you want, but improving and maintaining existing code is a real requirement for most software developers) even it will come with some edge cases.

I've had NPEs in release versions of Android apps from Google, Twitter, MS and other large software companies. This absolutely is not only an issue for sloppy/inexperienced devs that don't test.

DanTup on 26 Sep 2016

👍1

I think the treatment of null is especially tricky in the cases where it is actually an interface issue: "This data structure can contain null / should never contain null", "If this argument is absent, pass null / It is an error if this argument is null". In these cases the standard (implicit) treatment of nullability is error prone, and an explicit treatment is likely to improve correctness in the same way as other explicit interfaces. It is also difficult to address this kind of issue with anything else than an invariant which is stated explicitly, maintained at each relevant point, and hence globally valid; that is, for these cases it fits well to express nullability as a type.

For local variables the situation is somewhat different (because they may need to be "null for a while, but eventually non-null"). Definite assignment analysis was already mentioned several times as a helpful mechanism to handle that, but it is an interesting idea that locals might benefit from being treated less strictly.

eernstg on 26 Sep 2016

Software is too complex to expect to hit every permutation

Sure, most bugs are due to complexity. Obvious cure is to reduce complexity. You can't do it by chasing nulls under the lamppost. NPEs are bugs like any other bugs, have same origin.

This absolutely is not only an issue for sloppy/inexperienced devs that don't test.

It takes a lifetime of experience to realize that our abilities to handle complexity are very limited.
For superstars, it may take longer :)

How often did you hear developer saying: you know, this requirement is too complicated for me to implement - please find a smarter one for this. Very few can say this, but it's more effective than any bug-preventing tools combined. And in the end, those complicated, buggy features scare users away, too. Good idea is to set your complexity bar low - much lower than you think you can handle.

I'm not sure why you're so against non-null but you always have the option not to use it

I wrote 10 posts about it. In short: the medicine costs money and may have severe side effects, affecting complexity of both compiler and code. Not cost-effective. (The burden of proof is on those who propose the feature, not on those who are skeptical about it).

tatumizer on 26 Sep 2016

@eernstg but where these nulls passed as parameters come from? Aren't they uninitialized variables? Because they spread. You call constructor, pass (uninitialized) var, and the null gets stored in the object, so it looks like initialized field, but the origin of the problem is in another place, where it wasn't initialized. If you remove root cause, you have less problems like this. So the transitive closure of the problem of uninitialized vars can be more than 50% of cases.

Another source of nulls is: functions sometimes returning null, like map[]. This can be cured by another method that throws "not found" instead (as discussed earlier). Not only it throws, but includes the name of key in the message. Quite convenient. Maybe there's a couple of other cases. By targeting those individually, you solve big part of problem naturally.

Correction to the earlier post: No need for exclamation mark (last item in the list). If compiler prints false warning for "uninitialized" for declaration int x;, simply write int x=null; - it will silence the warning, b/c it's initialized already.

Consider a "game" - somebody offers you a choice: you get $1000 for free; OR you have to work hard and then you can get up to $2000, but no one tells you what "up to" means, so you can get $2000, or $1500, or $500, or nothing, or even owe money to the house after the game.

What variant would you choose?

Added later: maybe you remember our debate about different meanings of nulls? Null may have several meanings, but there's certainly a finite number of them. Most frequently used are "not initialized", "not found" and "not applicable". Completely different meanings, it's like there's a number of different "flavors" or nulls, which for historical reasons are merged under the same name "null". My point is that each flavor requires specific treatment. Because the flavors are completely different.

tatumizer on 26 Sep 2016

Sure, most bugs are due to complexity. Obvious cure is to reduce complexity.

All of your comments are about reducing complexity. Allowing you to "opt out" of nulls where they don't make sense is reducing (accidental) complexity.

I'm not sure why you're so against non-null but you always have the option not to use it

I wrote 10 posts about it. In short: the medicine costs money and may have severe side effects

If you don't like the side-effects, you don't have to take the medicine :)

I understand why you don't want these, but I don't understand why you think nobody should have them. Surely devs are able to make their own minds up and choose to use this feature where it makes sense? Many other languages have non-nulls and they're generally considered to be a great thing (you're the first person I've ever seen against the idea).

Anyway; I feel we're going in circles and I don't want to add more emails to people subscribed to this case. I'd love to have non-nulls (I really don't think forced nulls belong in any modern language), but I trust the Dart Team will do what makes the most sense. I don't think I can add anything more (and my opinion doesn't count for much, I've probably written the least Dart code of anyone here!).

DanTup on 26 Sep 2016

👍1

You can't "opt out". If you read the whole exchange, you will see that consistent implementation of "null-safety" entails complete redesign of the whole type system and will lead to cloning of swift, almost verbatim.

Dart still has an option of implementing it inconsistently, by targeting nulls specifically, but that would be really bizarre.

tatumizer on 26 Sep 2016

@eernstg but where these nulls passed as parameters come from? Aren't they uninitialized
variables? Because they spread. You call constructor, pass (uninitialized) var, and the null gets stored
in the object, so it looks like initialized field, but the origin of the problem is in another place, where it
wasn't initialized. If you remove root cause, you have less problems like this. So the transitive closure
of the problem of uninitialized vars can be more than 50% of cases.

The proposal where non-null types by default are used for interfaces (in particular, for function/method signatures and for field types) and local variables may be more flexible (maybe even nullable by default) will actually stop the propagation: The constructor invocation will reject the null as an argument and null won't be stored in the object.

Another matter is that we may prevent the propagation of null statically or dynamically. If we prevent it statically then local variables will be less convenient at usage points (because they will more often be nullable, and hence there will be a T | Null to T downcast for some T in more places), and if we prevent it dynamically (by accepting that downcast silently, but inserting a dynamic check) then the code may fail in more locations that are not obviously error prone.

In any case, the null won't propagate outside the block where that local variable is declared.

Another source of nulls is: functions sometimes returning null, like map[]. This can be cured by
another method that throws "not found" instead (as discussed earlier). Not only it throws, but includes
the name of key in the message. Quite convenient. Maybe there's a couple of other cases. By
targeting those individually, you solve big part of problem naturally.

Sure, the interface of Map.operator[] could have a nullable or non-null return type (if we use a non-null bound on the type arguments, such that we prevent null keys and values, that is!), and then we could communicate the "not found" outcome using an exception or by returning null. Both are useful designs, and we should be able to express them.

eernstg on 29 Sep 2016

and then we could communicate the "not found" outcome using an exception

This would be bad design from a performance point of view. I strongly advise against using exceptions for general control-flow.

fsc8000 on 29 Sep 2016

This would be bad design from a performance point of view. I strongly advise against using exceptions for general control-flow.

This is not for control flow. As we discussed before, there should be 2 getters, one can return null, and another throws on "not found". When you call the one that can throw, you implicitly make an assertion that element is supposed to be there. E.g.

var map=Map();
map["foo"]="hello";
var x= map.get("foo"); // I assert it's there, otherwise it's a bug - go ahead and throw

This works quite well with JSONObject in java. And you always have another method map.hasKey("foo") for explicit check, and you have map["foo"] if you are OK with returning null.

@eernstg : what if we had a special value for "uninitialized"? Then, on top of static analysis (which doesn't always work - e.g we can have uninitialized elements in the list) we could check inputs of every function (in checked mode) for "uninitialized" values.

I personally think static analysis would be good enough in practice, but there's so much scaremongering about nulls that someone would want another protection, as long as it's free. Dealing with "questioned" types is not free by any measure, people should be careful what they wish for. Or go to kotlin and try them, just to realize how much you liked java.

tatumizer on 29 Sep 2016

Hey kids following along at home! @eernstg noted that my original grammar suggestion would allow invalid things like:

class Foo extends Bar? {}

He fixed that by splitting out the type production into a separate one that does not include the optional "?". I updated the grammar here to follow that.

munificent on 29 Sep 2016

👍1

.. what if we had a special value for "uninitialized"?

You just gave me an idea there! ;-)

What if we treat null as the special value meaning "uninitialized" for variables which have a non-null type but no initializing expression?

This means that we can allow for a local variable C x; to be declared and initialized to null even though C is a non-null type. We can use standard type checking to ensure that x is not _made_ null by assignment, so an actual null value will unambiguously mean that x is being used before initialization, and that should be caught by checking dynamically at every evaluation of x where it is not yet statically known to have been initialized. For cases where definite assignment analysis proves that there is no way to encounter the uninitialized value it makes no difference (there are no locations where the check must be performed), but for cases where static analysis does not prove that initialization has taken place at a location where x is evaluated, there would be a dynamic check.

As usual, we can make the choice to insert the uninitialized-check (i.e., null-check) implicitly, or we may emit a diagnostic message, but that is a separate issue.

The point is that we can allow for "eventually non-null" behavior backed up by a combination of static and dynamic checks which is sound and convenient, which is quite Darty. ;)

eernstg on 30 Sep 2016

Oh, that _is_ an interesting idea!

That might really increase the usability of "eventually non-null" use cases, without punishing the performance of common cases where we can eliminate the runtime checks through definite assignment analysis.

It's also not entirely without precedence in Dart. We have runtime checks for accesses to top level variables to check for cyclic references:

var a = b;
var b = a;

main() {
  print(1);
  print(a);
  print(2);
}

This prints "1" before throwing a runtime error.

I'd like to think about this some more, but once we start playing with implementing non-nullable types, this is definitely an idea to put on the table.

munificent on 30 Sep 2016

This would eliminate a nasty case when you have to LIE about your type (pretending it's nullable), just because compiler can't prove the opposite.

tatumizer on 30 Sep 2016

What if we treat null as the special value meaning "uninitialized" for variables which have a non-null type but no initializing expression?

I haven't been following the entire thread, but I believe that what you are referring to _was_ part of the original proposal, see Section B.3.4.

chalin on 30 Sep 2016

@chalin : Maybe it's the same idea indeed, but could you please clarify one point:

For a non-null local variable, a static warning (and a dynamic type error) will result if there is a path from its declaration to an occurrence of the variable where its value is being read

The key word here is "path". Whether said path exists, becomes clear only in runtime. So dynamic type error is based on real path. But "static warning" is based on "imaginary path" that compiler thinks exists, but it really doesn't. And that's the problem.

As a part of corporate voodoo, we have Klocwork java tool running for every nightly build. It produces lots of complaints about nulls. How many of them are real? My personal experience: had just 1 (one) case within last 5 years where it found a real problem. The rest (lots of them) were false. But even this single case I already fixed by the time of report.

I think we are discussing the idea of not printing warnings at all. If compiler thinks there's a path, it will insert check for null. Otherwise it won't. If compiler can't see any assignment to the variable whatsoever, anywhere, it may produce a warning (then it's a real warning).

What do you think?

tatumizer on 30 Sep 2016

To make it more concrete:

class Foo {
  int _x;
  int get x => _x;
  void set x(int value) {_x = value;}
}

Is there a warning in getter? My understanding is that your proposal wants a warning here (if I'm wrong, I apologize). Suppose there's a warning. Programmer knows that in reality, he always calls a setter before getter, so the warning is false, and needs to be suppressed. How? Which leads to absurd code:

class Foo {
  int? _x; // false type
  int get x => _x!; // explicit non-null assertion
  void set x(int value) {_x = value;}
}

tatumizer on 1 Oct 2016

@eernstg : did you consider making nullability a property orthogonal to type (as opposed to "part of the type")? E.g. reading "String? x" not as "x or type nullable String", but as "x or type String, nullable".

There're some arguments for this. Among other things, it would greatly simplify generics.
Right now, there's a number of problems with generics. Even syntax problems. E.g. it's painful to even think of this:

class Sorter<T extends Ordered?<T>> {}
class Sorter<T extends Ordered<T?>> {}
class Sorter<T extends Ordered?<T?>> {}

(I don't know what it all could mean, if anything, but it seems the proposed syntax allows it)

If you make nullability an orthogonal property, generic definitions would never have to mention "?".
There will be the only valid form class Sorter<T extends Ordered<T>>, and every generic parameter will internally (inside implementation of Sorter) be interpreted as nullable(*) So all implementations of all generics will remain as they are now. But when we instantiate the class, we pass actual types, along with nullabillity property, and compiler can generate invocation code accordingly. E.g., if we have var map=Map<String,String?>,
we won't be able to use key "null". The only change to be made in implementation of Map is changing return type of [] to "T?" (otherwise compiler will automatically check for null, and throw when not found)

There are other arguments, which we can discuss if you think the idea is worth it.

(*) nullable is a default, and the only possible value, for generics. Because generics are "generic". For regular declarations, default is "non-nullable".

Added later:

if nullability is not part of type, maybe it's better to use different syntax, e.g. nullable String x; instead of String? x. Optional parameters of functions with no default value are automatically nullable.
If we eliminate "false nullables", then cases where we need explicit "nullable" will be rather rare IMO.

tatumizer on 2 Oct 2016

what you are referring to was part of the original proposal, see Section B.3.4.

Indeed, I hadn't discovered that -- sorry!

eernstg on 3 Oct 2016

I haven't been following the entire thread, but I believe that what you are referring to was part of the original proposal, see Section B.3.4.

Oh, sorry, thank you for pointing that out! I've read your spec several times, but it's a little too big to hold it all in my increasingly small brain space.

munificent on 3 Oct 2016

did you consider making nullability a property orthogonal to type (as opposed to "part of the type")? E.g. reading "String? x" not as "x or type nullable String", but as "x or type String, nullable".

Yes, we have discussed making nullability a property of declarations and not types.

I'd have to dig up our old notes, but my recollection was that it ends up feeling very limited in abitrary ways, especially around generics.

class Sorter<T extends Ordered?<T>> {}
class Sorter<T extends Ordered<T?>> {}
class Sorter<T extends Ordered?<T?>> {}

Only the second is valid according to the proposed syntax.

There are other arguments, which we can discuss if you think the idea is worth it.

I'll gently remind you that this issue is only for tracking _experimental_ non-nullability _syntax_. The semantics are, of course, vital, and we have thought about them a lot, but they are out of scope here. When we start working on them, we (probably me) will write up a more detailed proposal to send out before we start putting real effort into it.

This issue is deliberately vague on semantics because that comes in a later phase.

munificent on 3 Oct 2016

@munificent do you remember this article
http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/

Especially this phrase:

I don’t know about you, but nothing gets me going in the morning quite like a good old fashioned programming language rant

I have a good news for you: you have enough material for another old fashioned rant . This time, it's "What color is your type?"

The very fact that you had to introduce so many syntax rules to cover nullables is the admission that nullable types are not really types. Or, rather, not first-class types. It's another color of types at best.
Because had they been first-class types, you could use T and T? on equal footing in any context.

Now syntax is split into rules for red types and blue types. And the number of syntax rules will certainly continue to grow.

tatumizer on 4 Oct 2016

Here's my syntax proposal:

no syntax changes at all
introduce annotation @nullable, which is valid only in 2 contexts: declaration of variable/parameter and instantiation of generic type. Examples:

@nullable int x;
new Map<@nullable String, @nullable int>();

Please note that we are talking only about syntax here!

tatumizer on 4 Oct 2016

The very fact that you had to introduce so many syntax rules to cover nullables is the admission that nullable types are not really types.

No, all it shows is that before now, Dart had no syntactic distinction between types and classes. There are places in the language where you must refer to a _class_. When you're specifying a superclass, superinterface, etc., you need to refer to some class declaration.

Before this proposal, the syntax for types was indistinguishable from the syntax for referring to a class, because there were no type notations that didn't look like classes. As soon as you add ? for non-nullable types, explicit function type syntax, union types, tuples, or any of the other panoply of possible non-nominal kinds of types, we would end up having to split the grammar in this way.

It's just that up to now, we didn't need to, so we could have type in the grammar do double duty as a type and a reference to a class.

munificent on 4 Oct 2016

Before this proposal, the syntax for types was indistinguishable from the syntax for referring to a class

You can always keep it this way. Like you currently do for function types via typedef. It's a structural type, so tuple type can be introduced exactly like that - you always have a simple name. And this is true for the union types and any of the hypothetical panoply, too. This is just a very convenient syntactic device to keep syntax simple.

Your argument is not very convincing IMO because it lacks a massive use case that could justify it.
But it's your call.

tatumizer on 4 Oct 2016

@eernstg @munificent: no worries; the original proposal _is_ a bit lengthy, but nullity (as we all know) can be tricky to do "right" in the context of Dart.

All: thanks for keeping the discussions moving forward.

chalin on 5 Oct 2016

@munificent : is this a valid _syntax_:

var map=new Map<String?, String>();

Assuming the answer is "yes": suppose I want to implement my own map that only works with String keys, but I want null to be a valid key. How to write definition of this generic class:

class StringMap<K extends WHAT???, V> {...}

Is there a _syntax_ for that?

tatumizer on 6 Oct 2016

@munificent : is this a valid syntax:

var map=new Map();

Yup, that's fine.

Assuming the answer is "yes": suppose I want to implement my own map that only works with String keys, but I want null to be a valid key. How to write definition of this generic class:

class StringMap {...}

Well, in this case, the answer would be just class StringMap<V> { ... }. String is a sealed type, so there's no reason to make it generic on the key type. The only valid type argument would be String. :)

But let's say you want to define a number set that can be used with ints or doubles and you also want null to be a valid member. You would do:

class NumSet<T extends num> {
  void add(T? value) { ... }
  void remove(T? value) { ... }
  // etc...
}

munificent on 6 Oct 2016

String is a sealed type, so there's no reason to make it generic on the key type

I made it generic precisely to show that it can handle null keys - so that it can be instantiated with both String and String?

var map=new StringMap<String?,String>();
var map1=new StringMap<String,String>();

Anyway, you answered the same question with num example.

tatumizer on 6 Oct 2016

I made it generic precisely to show that it can handle null keys - so that it can be instantiated with both String and String?

Ah, sorry. I thought you meant you wanted it to _always_ support null keys, regardless of the type parameter type. In my NumSet example, you can _not_ do NumSet<int?> because the constraint is num, which is non-nullable. If you want to allow that, you'd do:

class NumSet<T extends num?> { ... }

This means that in the body of NumSet, T is now a nullable num type, so before you can call methods on it, you have to test for null first.

munificent on 6 Oct 2016

This was precisely my question, and now I understand that extends num? is allowed in this context

class NumSet<T extends num?> { ... }

Then let's come back to the earlier question:
Which of the following definitions are _syntactically_ correct:

class Sorter<T extends Ordered?<T>> {}
class Sorter<T extends Ordered<T?>> {}
class Sorter<T extends Ordered?<T?>> {}

You said only the second one. But why? Your example with num shows that it would be fine (syntactically) to say

class Sorter<T extends Ordered?> {}

Which makes all 3 definitions syntactically correct by implication, no?

tatumizer on 6 Oct 2016

But why?

The other two would be:

class Sorter<T extends Ordered<T>?> {}
class Sorter<T extends Ordered<T?>> {}
class Sorter<T extends Ordered<T?>?> {}

The ? was in the wrong place. (Also, I think when I first commented I may have forgotten that our tentative plan does allow ? in constraints.)

munificent on 6 Oct 2016

Thus, we established that both following definitions are valid:

class MyGenericClass<T extends Object> {...}
class MyGenericClass<T extends Object?> {...}

(The latter can be instantiated with any concrete type, but the former - only with non-nullable concrete types)

The question is: what is the default? When I write

class MyGenericClass<T> {...}

is it equivalent to the first, or to the second one?

tatumizer on 7 Oct 2016

class MyGenericClass<T extends Object> {...}
class MyGenericClass<T extends Object?> {...}

These would be equivalent since Null is a subtype of Object. Null|Object collapses to Object.

(The latter can be instantiated with any concrete type, but the former - only with non-nullable concrete types)

My current idea for the semantics does not give you a way to express "any type, but not nullable". If the only thing you know about the type parameter is that it's Object, it's not the end of the world to permit null—it supports all of the methods that Object does. It _is_ an object.

All of these questions would be answered by a proposal for the _semantics_, which I have not yet written down. This issue is just for the _syntax_.

Do you think we can table this discussion until I have a real proposal to go on? Right now, we're sort of doing a breadth-first traversal through the semantics one comment at a time, which isn't an efficient use of either of our time.

munificent on 7 Oct 2016

If we wish to support a distinction between Object with and without null we can remodel the type hierarchy such that Null is not a subtype of Object any more, but we haven't decided that this is a good idea.

eernstg on 7 Oct 2016

Do you think we can table this discussion until I have a real proposal to go on?

Sure. Thanks. Looking forward to semantics proposal :)

tatumizer on 7 Oct 2016

@munificent the link to the dart2js bug is wrong up top. Off by one error 😉

donny-dont on 14 Jan 2017

Is this not happening? :/ I find this to be a really useful concept for ensuring type safety of my code, and along with the lack of union types is making me wary of switching from TypeScript/React Native to Dart/Flutter in future projects.

osdiab on 27 Jun 2018

It's not happening yet. Moving Dart to strong mode is a necessary pre-condition for non-nullable types. We are doing that in Dart 2. We hoped to get non-nullable types in at the same time, but it proved to be too big of a change to fit into 2.0, so it's going to have to wait until a later version.

munificent on 27 Jun 2018

👍8 ❤2

So now as Dart 2.0 is released would be there any changes so it finally get happened?

vanesyan on 9 Sep 2018

Now all that's left is to design and implement non-nullable types, figure out a migration plan, add language features to make them more usable, etc. :) Basically, we have to do all the work. It's a giant feature.

munificent on 11 Sep 2018

👍9 ❤5 🎉1

If this year there was only one feature that dart shipped, this would be my choice. I really hope it becomes a high priority soon.

gbaldeck on 23 Jan 2019

👍2

@gbaldeck and that's exactly what is happening this year see https://github.com/dart-lang/language/issues/110 :)

@munificent Should we close this and all associated issues and indicate that people can follow along the new language process in the language repo?

mraleph on 24 Jan 2019

👍7 ❤3 🎉3 👀1 🚀1

Yeah, good call. Closing this in favor of https://github.com/dart-lang/language/issues/110 which is the main tracking issue for the current plan to add non-nullable types.

munificent on 24 Jan 2019

Sdk: Non-nullable type annotation syntax

Flag

Tracking issues

Most helpful comment

All 118 comments

Related issues