Language: Null Safety: Sound non-nullable types with incremental migration (NNBD)

[leafpetersen]

This is a solution proposal for #71. The proposal covers the intended final language design, and the language, tooling, and ecosystem support required for the migration.

• Roadmap
• Feature specification
  
  
  
  • NNBD related issues
  
  

Topic specific discussion issues for resolution:

• [ ] [Throw/catch static errors](https://github.com/dart-lang/language/issues/385)
• [x] [Null assertion operator](https://github.com/dart-lang/language/issues/196)
  
  
  
  • Resolved, yes, !.
  
  
• [x] [Should the NNBD opt in also remove implicit casts](https://github.com/dart-lang/language/issues/192)
  
  
  
  • Resolved, yes
  
  
• [x] [Is Object nullable?](https://github.com/dart-lang/language/issues/141)
  
  
  
  • Resolved, no
  
  
• [x] [Do we need T!?](https://github.com/dart-lang/language/issues/142)
  
  
  
  • Resolved, no
  
  
• [x] [Do we need generic constraints?](https://github.com/dart-lang/language/issues/143)
  
  
  
  • Resolved, no
  
  
• [ ] [Grammar issues](https://github.com/dart-lang/language/issues/144)
• [x] [Fields initialized after creation](https://github.com/dart-lang/language/issues/146)
  
  
  
  • Resolved, we will use late
  
  
• [x] [Chaining null-aware operators](https://github.com/dart-lang/language/issues/155)
  
  
  
  • Resolved, yes.
  
  
• [x] [Default parameter values](https://github.com/dart-lang/language/issues/156)
• [x] [Can object methods be called on nullable types](https://github.com/dart-lang/language/issues/175)
  
  
  
  • Resolved, yes.
  
  
• [x] [Precedence of await vs !](https://github.com/dart-lang/language/issues/319)
  
  
  
  • Resolved, ! binds more tightly.
  
  
• [x] [Querying the state of late variables](https://github.com/dart-lang/language/issues/324)
• [x] [Null short-circuiting spread operator](https://github.com/dart-lang/language/issues/330)

[leafpetersen]

I've added a draft of the roadmap for this proposal, linked from the header comment.

[yjbanov]

@jmesserly I'm not sure how to constructively move the NNBD discussion from https://github.com/dart-lang/language/issues/125 here, because the my suggestion is all about linking the two issues. But I'll try:

I suggest that, as a first step, we introduce sound non-nullable types as part of immutable types (https://github.com/dart-lang/language/issues/125). This has several advantages:

• It would be a non-breaking change: immutable types is a new feature so having them NNBD doesn't break any existing code.
• It could have the biggest performance impact: if non-nullability is implemented such that many highly-used types can be inlined, such as double, int, bool, enum, String, we could significantly improve memory layout efficiency.
• It would give us a chance to get the syntax and semantics right and educate users before rolling NNBD to the "normal" classes, which does require a bigger migration.

[jmesserly]

IMO, linking the two proposals would be counterproductive. We are working hard on NNBD, there is a lot of design & planning work going on. There aren't any major open questions on syntax or desired semantics. Linking the two would prevent us from pushing forward on NNBD, because immutable types has a lot of open design questions, and we'd have to delay work on NNBD until that's resolved.

(Also it could be confusing if different parts of the language treat int as having different non-nullability. It would be really nice if int always means non-null, and int? always means mean nullable. The NNBD proposal achieves that.)

[yjbanov]

That's fine too, it was just an idea. I don't have enough context to know how best to proceed, so if you believe that NNBD and immutables should proceed completely independently, then they should :)

[jmesserly]

Makes sense :). Yeah, we seem to be gathering a lot of momentum behind Leaf's proposal. It's really exciting! I think we'll be able to apply the many lessons of Strong Mode to NNBD, to make the migration as easy & rewarding as possible. (e.g. ideally it's non-breaking for code that isn't migrated. Until you turn on NNBD, you don't break. Once you turn it on, you have to fix some errors, but you gain some degree of null safety, even if everything else isn't migrated yet. Once everything in an app is migrated, then it has sound null safety. It's pretty neat.)

[leafpetersen]

I've updated the header with some sub-issues for discussing specific topics.

[leafpetersen]

Added links to two new discussion issues in the header comment.

[leafpetersen]

Added a discussion link to the header for removing implicit casts.

[leafpetersen]

Added a discussion link to the header about a non-null assertion operator.

[ollydixon]

Do you think we'll get an elegant way to assert** optionals with this? Much like Swift, TypeScript, Flow & many other languages?

Small example below. (TS)

class User {
    // We don't need to unwrap this, it's non-nullabe. Which effectively means nothing if you have optionals.
    uid: string,
    // Not chosen yet. The team knows this might be null from the '?' Might never be chosen.
    username?: string
    // Also union types in Dart would be v-nice.
    accountType: 'email'|'facebook' 
}

[leafpetersen]

@ollydixon I didn't really understand your question. Is 'asset optionals' a typo?

The goal of this proposal is that Dart will allow the equivalent of the code that you wrote less the union type (for now).

[ollydixon]

@leafpetersen yes it was a typo, sorry. That's good to hear. Optionals will be a welcomed addition.

[leafpetersen]

Filed an issue for discussion of relative precedence of await and ! here.

[leafpetersen]

Filed an issue for discussion of querying the state of late variables.

[leafpetersen]

Filed an issue for discussion of null-short-circuiting the spread operator.

[jodinathan]

as anxious as I can be, I tried some testing in WebStorm few days ago and the indentation does not work with NNBD. I've opened an issue so we can use the syntax in our common days along with the implementation: https://youtrack.jetbrains.com/issue/WEB-38930

[leafpetersen]

Filed a discussion issue on throw/catch static errors here

[rrousselGit]

Is there there any discussion around prototype overload for non nullable types caused by optional parameters?

There are a few objects that behaves differently based on whether an optional argument is present or not.

One example is StreamBuilder. If initialData is passed, then it's prototype is:

StreamBuilder<T>(
  initialData: T, 
  builder: (BuildContext, AsyncSnashot<T>) => Widget, 
)

But if omitted, the prototype is:

StreamBuilder<T>(
  initialData: null, 
  builder: (BuildContext, AsyncSnashot<T?>) => Widget, 
)

---

Similarly, there are a few asserts performed by widgets like Positioned (that accepts many parameters, but not all at once) which could be removed using a null safe prototype overload.

[leafpetersen]

Is there there any discussion around prototype overload for non nullable types caused by optional parameters?

I'm not entirely sure what you mean, but I think the answer is no. This might be more of a question for the flutter framework to answer - how they plan to migrate their API here. We don't have overloading in Dart, and won't get it any time soon.

For this API, it looks to me like either you have to either:

• split this into two different constructors
• make initialData required, which would force T to be a nullable type if you want to pass null for it
• make initialData have type T? and keep it optional. I think this requires AsyncSnapshot

[jodinathan]

Indentation for NNBBD now works with WebStorm 2019.2.

[derolf]

Is there there any discussion around prototype overload for non nullable types caused by optional parameters? ...

@leafpetersen I think the question is in general: what if you have two parameters where exactly one is null, but not the other.

Usually, you do assert((a == null) != (b == null)).

It would be nice to instead list all possible “variants” of the function signature and bind it to one implementation, like:

void func(int? a, int b),
void func(int a, int? b) {
...
}

[jodinathan]

I hear method overloading?

Don’t play with my feelings

Em 24 de ago de 2019, à(s) 17:14, derolf [e-mail] escreveu:

Is there there any discussion around prototype overload for non nullable types caused by optional parameters? ...

@leafpetersen https://github.com/leafpetersen I think the question is in general: what if you have two parameters where exactly one is null, but not the other.

Usually, you do assert((a == null) != (b == null)).

It would be nice to instead list all possible “variants” of the function signature and bind it to one implementation, like:

void func(int? a, int b),
void func(int a, int? b) {
...
}

[Levi-Lesches]

Hi, is there an ETA on non-nullables? Even a very loose one, like tomorrow, 5 months, or 2 years from now.

[rrousselGit]

Hi, is there an ETA on non-nullables? Even a very loose one, like tomorrow, 5 months, or 2 years from now.

There's a flag for a preview usually. Here's the flag for static extensions: https://github.com/dart-lang/sdk/issues/37893#issuecomment-522576983

I don't know the flag for nnbd, but it's likely that if you use the 2.5-dev and try to use nnbd syntax, the compiler will tell you to turn on the flag.

[derolf]

The flag is --enable-experiment=non-nullable. However, the main issue is that the SDK itself has not migrated yet. For example, trying to use Map[] currently returns a non-nullable value and your != null or ?? checks will be flagged.

[munificent]

However, the main issue is that the SDK itself has not migrated yet.

We're working on that. :)

[Levi-Lesches]

If you use the 2.5-dev and try to use nnbd syntax, the compiler will tell you to turn on the flag.

Yes, thanks, I see that now.

We're working on that. :)

So does anyone have a (relative) idea of when this will be ready, or at least usable?

[leafpetersen]

It would be nice to instead list all possible “variants” of the function signature and bind it to one implementation, like:

Yeah, it would be nice. Unfortunately, not at the top of our list of things to tackle right now, sorry!

[leafpetersen]

So does anyone have a (relative) idea of when this will be ready, or at least usable?

We really don't want to promise dates yet. Everybody is working full out on this and extension methods right now, it will be ready when it's ready! You can search for NNBD in the sdk issue tracker to get a sense of where things are, and I'll try to update now and then when notable milestones are passed.

[Levi-Lesches]

Ok, thanks!

[derolf]

Wow, extensions functions are coming as well? That’s really cool. What about let/apply/run-constructs from Kotlin?

[derolf]

For early adopters, is it possible to already run code that utilises the extensions?

See https://stackoverflow.com/questions/57934341/error-this-requires-the-non-nullable-experiment-to-be-enabled

[munificent]

Wow, extensions functions are coming as well?

Yes, extensions let you define all kinds of members: methods, getters, setters, index operators, operators, etc.

For early adopters, is it possible to already run code that utilises the extensions?

Not yet. We're pretty far along but there's still implementation work left to do. Stay tuned. :)

[derolf]

https://github.com/dart-lang/sdk/issues/38623

Is ‘Null’ nullable and is ‘Null’ the same as ‘Never?’.

[lrhn]

Null is nullable. It's equivalent to Never? (they are mutual subtypes), but not necessarily the same ... for some definition of "same".

[ollydixon]

Hey, we converted an entire project over the last 3 days to use optionals since without them there's so much possiblity for crashes. Seems we can't compile it to release. Any solutions for this? Some setting to bypass it?

[lrhn]

@ollydixon
No, not yet. This is still a feature in development. None of our production compilers support it yet (and even if they did, the feature would probably still have enough bugs that it should not be used in production).

[Levi-Lesches]

Any timeline? Even if it's vague

[IchordeDionysos]

Just out of interest is it planned to support something like a "conditional" cast?

Example in Swift

var optionalString = dict["SomeKey"] as? String

This would be super helpful if one has to deal a lot with Map types from JSON or Firestore documents as SomeKey would resolve to the String value if it's a string and to null if not.

Also to resolve strict linting rules you could avoid those errors:

The initializer type 'dynamic' can't be assigned to the field type 'String'

I believe that this feature would be super helpful to have :)

[leafpetersen]

Just out of interest is it planned to support something like a "conditional" cast?

There are no current plans for this, though I think it has come up as a desirable feature before. You can do this with extension methods somewhat reasonably in the meantime:

extension ConditionalCast on Object {
  T tryCast<T>() {
    T self = this; 
    return self is T ? self : null;
  }
} 

void test(Map<String, dynamic> dic) {
  var optionalString = dict["SomeKey"].tryCast<String>();
}

[tatumizer]

Indeed, it is a good use case for extension methods. But right now, there's a caveat: dic["SomeKey"] has a type of dynamic, which doesn't match any extension method. So the attempt to call test({"someKey": 1}); fails in runtime with the message
"Uncaught TypeError: Cannot read property 'tryCast$0' of nullError: TypeError: Cannot read property 'tryCast$0' of null".
To fix that, you either have to change the type of the parameter to Map<String, Object>:

void test(Map<String, Object> dic) {
  var optionalString = dic["SomeKey"].tryCast<String>();
}

or cast dic["SomeKey"] as Object before calling tryCast.
Maybe we can revisit the decision of special-casing dynamic WRT extension methods?

[lrhn]

It's not really that dynamic is special cased. It's treated exactly like it is everywhere else in the static semantics: As if it has members with all names.

The problem with allowing extension methods on dynamic is that it's impossible to check statically whether the object has a member of the same name already. In all other situations, the existing instance member wins over an extension member. For dynamic, we have to pick either the instance member or the static extension member, and we prefer to do so at compile-time.

It would be possible to delay the choice until run-time: Determine at compile-time which static extension method would win if extension methods applied, then defer to a run-time check whether the object has a member of the same name already, and if so call that. If not, call the pre-determined extension method instead.

That would still not follow the model I generally use for deciding how dynamic works: Dispatch at run-time to whatever operation would have happened if the expression had had its dynamic type as static type. For that to work, the run-time dispatch would have to know all static extensions in scope and choose between them at run-time. And that would happen for each dynamic invocation which in scope of any static extension (which will likely be all code the moment we add any extension declaration to dart:core).

That would be very expensive, and with that kind of expensiveness comes code-size for ahead-of-time compiled programs. We have already made dynamic dispatch somewhat expensive with extra dynamic type checks. I'm not sure it would be prudent to heap more work onto the operation.

Even the "one possible static extension alternative" way will add overhead when a dynamic invocation happens in the scope of an extension on Object?. It's less expensive, but still there.

[tatumizer]

The problem with allowing extension methods on dynamic is that it's impossible to check statically whether the object has a member of the same name already.

Here's a counter-argument to this. Sure, dynamic has all methods potentially defined. But the compiler can see that the extension method (say, foo) is actually defined. In a common-sense logic (which operates in the head of a user), we normally tend to give higher weight to the actual facts. But it's not only that.

The user might know for a fact that his dynamic does not have method foo because he is sure it's either an int or a String. (the user almost always knows all the actual possibilities hiding behind the term dynamic in a given context). In which case, the behavior of the compiler can be (and is!) quite surprising (that's why we are having this debate - the issue pops up repeatedly).

Granted, if dynamic was a rarely used type, no one would care. But currently, we use Map<String, dynamic> as Json objects all over the place, and every user will be wondering why his/her extension method doesn't work for json["x"].foo() - which looks perfectly fine, it doesn't even mention dynamic.

So my point is: the dilemma of what consideration has a higher weight (potential vs actual) could be decided either way. But for practical/pedagogical reasons, I think it would be better to decide it in favor of the extension method - statistically, it would be less surprising for the user IMO.

[derolf]

I would even go one step further and remove calling “random” methods directly on dynamic, but would propose to introduce an invoke method on dynamic to make it explicit:

dynamic foo = ...
foo.invoke(“bar”, 42);

Then the only callable methods of dynamic would be invoke and static extension functions.

The reasons is that I already saw bugs where the compiler widened the type to dynamic (because of a missing union type in dart) and then there was a spelling error in the method call — uncaught by the compiler and failing at runtime.

Aww also: please bring union types to dart!

[tatumizer]

@derolf : making it more verbose won't help. You have the same probability of misspelling "bar" in
foo.invoke("bar") as you do in foo.bar(). The current dart's rules for dynamic are very convenient.
You can write json["foo"][0].bar whereas java would require json.getJsonArray("foo").getJsonObject(0).getJsonObject("bar"), which is incomprehensible and doesn't make it "safer" at all compared with dart's notation.

It's just we hit the corner case on the intersection of "dynamic" and extension methods, and in this corner case, the language has a number of possible choices. It might be that the current choice is not the best compromise.

[derolf]

@tatumizer I think you didn’t get my point. I wrote that “accidentally” the compiler inferred a type to dynamic (due to the lack of union types) and thus didn’t do any checks on the method name. By demanding “invoke” for calling methods on dynamic this would not have happened.

Typescript has a way superior typesystem than Dart and I would really love the Dart engineers to adopt some of their features. They have union and intersection types and instead of just “dynamic” they have “any” and “unknown“.

Flutter is such a great framework, but Dart feels like sitting in a way-back-machine to the 90s.

[nex3]

@derolf Although I agree union types in particular are a notable omission from Dart's type system, I think your frustration may be mostly due to a lack of familiarity. For example, TypeScript's any and unknown are analogous to Dart's dynamic and Object. And just like TypeScript has a --noImplicitAny flag that produces an error when your code accidentally ends up typed as any, Dart has a --no-implicit-dynamic flag to the static analyzer that produces a similar error for dynamic*.

When comparing Dart's type system to TypeScript's, it's also important to bear in mind that Dart provides a much stronger guarantee that colors the way its whole type system works: Dart has sound types, which means that it guarantees that the runtime type of a variable will match its static type. TypeScript has no such guarantee, and the added constraint means that any additions to Dart's type system must be very carefully designed and implemented to preserve soundness. In exchange though, you get much more confidence in the runtime behavior of your code, and there are many more opportunities for optimization and dead code elimination that helps Dart run faster and slimmer.

*: This flag is currently somewhat over-zealous, so in practice I use the narrower strict-inference and strict-raw-types flags.

[mit-mit]

This has now launched in tech preview 🚀. More details in https://medium.com/dartlang/announcing-sound-null-safety-defd2216a6f3.

[pedromassango]

Why the Dart team decided to use: nullable!.call() intead of nullable?.call()?

[marcelgarus]

Both these code snippets are supported and have different semantics:

• ! means that you guarantee to the compiler that the value can never be null. That means, calling nullable!.call() if nullable == null throws a NoSuchMethodError.
• ? means that you only want the call to execute if the value is non-null. In the case of nullable == null, nothing will happen.

By the way, you can also try out these examples at nullsafety.dartpad.dev.

[Levi-Lesches]

I'm not sure if this is the right issue, but dartdoc doesn't properly document the return type for Map.[] as V?. However, it does document int.tryParse properly as int?, so maybe it's just an operator thing?

[mateusfccp]

I'm not sure if this is the right issue, but dartdoc doesn't properly document the return type for Map.[] as V?. However, it does document int.tryParse properly as int?, so maybe it's just an operator thing?

[Levi-Lesches]

Correct, that's the code snippet. Yet the return type according to the docs is still V. See int.tryParse to see what I mean.

That return value gets mentioned elsewhere, such as in the method list for classes, function list for libraries, etc.

[mateusfccp]

Correct, that's the code snippet. Yet the return type according to the docs is still V. See int.tryParse to see what I mean.

That return value gets mentioned elsewhere, such as in the method list for classes, function list for libraries, etc.

Maybe a bug on dartdoc for parameter types?

[Levi-Lesches]

I left a note on https://github.com/dart-lang/dartdoc/issues/2166 as well just in case, but I don't know if that's the right issue for it. I can open a new one if it's irrelevant here.

[Levi-Lesches]

I opened a bug on https://github.com/dart-lang/dartdoc/issues/2232 for it.

[ganeshrvel]

I have opened a bug/feature request https://github.com/dart-lang/sdk/issues/42723