Language: Per Library Language Version Selection

[lrhn]

Proposed solution for #93.

See feature specification.

Discussion Issues:

• Should breaking changes be opt in?
• Should we have per library upgrading?

Summary:

Some new language features may be enabled on a per-library basis during a migration phase.
The feature is enabled for a package by it depending on an SDK version which has that feature. The version opt-in is stored in .packages.
An individual library can opt-out of the new feature by adding a tag at the beginning of the library (syntax pending).
Tools can take an --default-package=foo flag which then treats all unpackaged libraries as belonging to package foo. This can be used for testing, so tests get the package-wide opt-in even if they are not imported using package:foo/....

[natebosch]

This is another place where having a language-level understanding of a "package" could be useful. I'd expect a package to be a more likely unit to upgrade at once than a library.

[lrhn]

It's easier to require updates to be per package (for some definition of "package"), but I believe it is useful to allow a one-library-at-a-time migration, especially for large packages.

We can drop that and say that migration is always one package at a time. It would remove the need for new syntax, we just need a way to attach metadata to package names (and likely, some way to include file-URI source, like tests, in the package).

[jcollins-g]

"The syntax can be bike-shedded a lot more. The important point is that versioning uses version only. You cannot opt-in to one feature of a new version, but not the rest. You either migrate or not."

It is probably no surprise to anyone who has heard me on the experimental feature discussion that I'd advocate for a somewhat more flexible solution. :-) I'd suggest listing features explicitly that the library requires.

There's some prior art for it in other languages: PEP 236, for example. While both this and the strawman proposal do prevent users from going "back in time" to run code without certain features while allowing to pull in experimental bits from the future, I think putting the feature list in the source code has advantages in tooling.

First, abstracting away the individual features with a version number in the source code loses information (What exactly does this library require about 2.2? Different dev versions of 2.2 may have a different idea of what features 2.2 supports if for no other reason than implementation schedules.).

Second, users may be tempted to use that declaration for reasons besides language features, like dodging VM bugs or even indicating what they've tested the library with. With an explicit feature list, tooling can then easily tighten that down by throwing errors if the user is declaring something that isn't available yet with their SDK, or linting/warning if the declaration is unnecessary (if the feature is enabled by default in their SDK version, for example).

Third, dropping use of version numbers I think will eliminate some of the headaches listed around package versioning. Either the SDK you are using has the feature or not, and if it does your package works. Users are used to adjusting the SDK version constraints based on whether their package is functional at that version. Using explicit features there's no need to try to infer versioning based on .packages files and you can punt on the whole "language definition of a package" problem.

[lrhn]

One important distinction here is that this feature is not for experimental features.
It is intended for shipped features that would be breaking, and where we want to give users a migration path. It's not conditional compilation based on SDK version, and it's not a way to handle buggy implementations. That would be a much different feature.

We may allow the feature to be used prior to shipping it, but that is done using an "experiment" flag and carries no promise of working the same way tomorrow. You should not be shipping code that depends on an unshipped feature.

The "language versioning" here is verisoning of released language versions in such a way that you can opt-in/out of a breaking change. We don't have to use language versions as keywords, we could give them code-names, but it is important that it's a progression, not independent flags. I don't want someone to opt-in to "newstrings" released in Dart 2.8 and skip "nnbd" released in Dart 2.6 and "nosemicolons" released in Dart 2.4. That way lies combinatorial madness.

The intent is that code should be migrated to the new language, just that it doesn't have to happen all at once. Staying back means not getting the new features. That's an incentive to migrate. Then we can add another incentive by eventually removing the support for the old version.

We are not plannig to add user-selectable versioning for all changes. A non-breaking change-only version would just be added to the most recent selectable versions. You get it automatically if you are at the "head" of the language design.
I would prefer if even non-breaking new features are blocked for unmigrated code (it's fine to parse it and understand it, and then write "You try to use feature foo, but your language version is too low" if the library is not at a sufficient language version).

That will make the opt-in/out a way to select between strata of versions, milestone releases.

[jcollins-g]

I am hearing that we have a slightly different understanding of the definition of "experimental" and I likely shouldn't have used that word in "experimental bits from future versions", instead saying "features from future versions".

I do believe I understand the proposal, and that you are not planning to add user-selectable versioning for all changes. It's not an unreasonable approach. I'm only saying I think it might be a good idea to add user-selectable features for some changes, and laying out some reasons (and prior art in other languages) as evidence for why.

WRT: I would prefer if even non-breaking new features are blocked for unmigrated code (it's fine to parse it and understand it, and then write "You try to use feature foo, but your language version is too low" if the library is not at a sufficient language version).

This seems like a way to make things fragile for end users unnecessarily, but I suppose if our migration tools are really excellent and easy to use maybe this doesn't matter too much.

[eernstg]

@lrhn wrote

One important distinction here is that this feature is not for experimental features.
It is intended for shipped features that would be breaking, and where
we want to give users a migration path.

Maybe something like --enable-new-feature foo would be more self-explanatory than --enable-experiment foo?

[lrhn]

The --enable-experiment flag is not for shipped features. It's for features being developed.

The language version choice is for delaying migration to the newest released version.
The experiments flag is for testing new features being developed.

The moment a new feature is released, the experiments flag stops having an effect, and you will have to depend on language versioning to avoid enabling the new feature for your code.

New features are being developed at the tip-of-tree, and I don't want the developers to want to consider exponentially many combinations of features and migration versions so you can either stay back on a not-most-recent language version and delay migration, or you can test new features that are increments on the most recent version, but you can't do both. There is no skipping a feature, only a monotonically progressing sequence of features.

[leafpetersen]

So it seems like active questions are:

• Syntax
  
  
  
  • Part of the language: library with nnbd
  
  
  • Text in a comment: ` // #use nnbd
  
  
  • Pragmas: @pragma('nnbd')
  
  
• Do we allow opt out as well as opt in
  
  
  
  • Opt out by negating the opt in name?
  
  
  • Opt out by opting in to a special name (e.g. "library use legacy")
  
  
• How do we name versions?
  
  
  
  • Numeric based on Major.minor version of SDK in which the feature is released?
    
    
    
    
    
    • If so, can you specify any arbitrary Major.minor, or only specific ones
    
    
    
  
  
  • Arbitrary tag: "nnbd" describes the nnbd release
  
  
  • Something else?
  
  

Does that cover it?

[leafpetersen]

A concern about @pragma is that it presumably requires resolution before you can decide whether you've opted in or not, which affects both clients which don't do resolution, and also makes it hard to opt into features that change how resolution happens.

[munificent]

A concern about @pragma is that it presumably requires resolution before you can decide whether you've opted in or not

Right. Early on, Kotlin used, effectively, metadata annotations for many modifiers on members like "protected". They eventually switched that to real keywords specifically because of the confusion around resolution. No one wants to have to import what they think of as a keyword.

also makes it hard to opt into features that change how resolution happens.

In particular, a pragma that you have to import makes it really hard to opt in to better import syntax, which has been a desired thing literally since the say "package:" was introduced.

Also, pragma annotations have annoying soft failures where if you accidentally put the annotation on the wrong thing, the compiler simply doesn't see instead of telling you something went wrong.

I like the idea of having a linear list of "versions", but using codenames to refer to them. Each codename would be the main breaking feature shipped in that version, like "nnbd", or "optional_semicolons".

This naturally extends to a syntax for opting a library out of features when you've opted the entire package in. We just pick a codename like "legacy" that refers to Dart 2.1 and then users can use that to opt the library out.

[bwilkerson]

There's one more question we talked about in PDX: Is the mechanism per library or per file?

The only corner case I can think of is if there were a feature related to parts that impacted how we decide whether a file is a library or a part. If we needed to find the library before we could know how to decide whether it's a library then analyzer would be stuck. Such a feature seems unlikely, and it might be that this would only impact analyzer and its clients, so it probably isn't a major concern, but I thought I'd mention it.

[leafpetersen]

I added a discussion issue on the subject of whether non-breaking changes should be opt-in.

[leafpetersen]

I added a discussion issue on the subject of whether we should allow per library upgrading or just downgrading.

[lrhn]

• Syntax
  
  
  
  • Part of the language: library with nnbd
  
  
  • Text in a comment: ` // #use nnbd
  
  
  • Pragmas: @pragma('nnbd')
  
  

As stated, @pragma is probably too far down the parsing path to be useful as a way to trigger syntax changes. It requires parsing the entire library it's used in to see if it declares a pragma name overriding the one in dart:core.
The library with ... is very early in the file, and the comment can be even earlier.
I'd just go for the comment for now, and require it to precede all declarations in the library.

• Do we allow opt out as well as opt in
  
  
  
  • Opt out by negating the opt in name?
  
  
  • Opt out by opting in to a special name (e.g. "library use legacy")
  
  

We only allow opt-out (or rather, opt-down to a lower version).
If we get more than one feature to opt out of, a single "legacy" name is not sufficient.

Also, opting out really means picking a specific language version and acting like that. Whatever you write must corresspond to one version, so a negative choice is not the most precise way of saying that.
That said, a -nnbd would be usable as the version prior to introducing nnbd.

• How do we name versions?
  
  
  
  • Numeric based on Major.minor version of SDK in which the feature is released?
    
    
    
    
    
    • If so, can you specify any arbitrary Major.minor, or only specific ones
    
    
    
  
  
  • Arbitrary tag: "nnbd" describes the nnbd release
  
  
  • Something else?
  
  

I've generally kept to numeric version numbers because it's easier to handle. We can add textual aliases, so nnbd is an alias for the version introducing nnbd. The way opt-out is currently define, you likely want to opt out of NNBD, so we might need //# -nnbd meaning the version before nnbd.

I would say that you can use any major/minor version combination prior to the package's SDK dependency version, except that we'll have a cut-off point before which we no longer support the version. That version is currently 2.0, anything before that is unsupported.

[lrhn]

The opt-out is definitely per library. We do not want a single part to be treated differently from the rest of the same library.

If we ever change syntax sufficiently that we can't see if something is a part or a library (or the same file can be both), then we might have to adapt the opt-out syntax somehow.
I can't imagine that change now, so I also can't predict how to prepare for it.
So, I'd still say it's library only. The opt-out syntax must be placed before any other declaration, including part declarations.

[mit-mit]

@lrhn can this be closed as done?

[lrhn]

The feature has landed.