Language: Should per library versioning allow upgrading or only downgrading from package defaults?

[leafpetersen]

The library versioning strawman doc says:

Individual libraries can opt-in to a lower version than the package default, but not a higher one.

This means that one cannot incrementally migrate a package by marking individual libraries as opted in: instead, one must mark the package as opted in, and then opt out all of the libraries (removing the opt out one at a time).

Is this the right choice? Why do we make this restriction?

cc @lrhn @munificent @eernstg

[lrhn]

I believe this is the right choice.

Depending on a feature, even in only one library, is not safe if the package does not require an SDK version which supports the feature. So, if you want to use feature X, you need to increment the SDK dependency version to a version supporting X. That will automatically opt you in to the new behavior, and you will need to opt other libraries back out.

If we allow a library to opt-in to a feature which is not supported by the lowest supporiting SDK for the package, then things can just crash and burn if the package is compiled against that SDK.

If we separate the SDK version from the feature opt-in, then we won't automatically opt in on new packages. We'll forever have to keep the extra opt-in information around. Using just the SDK version as opt-in will provide a strong incentive to actually opt-in. You will then have to make an effort to make some libraries stay back on the earlier version of the language.

So, the design here goes for a design which encourages upgrading, by making it easy, even automatic, when you change the SDK requirement, and by gating all new features on upgrading the SDK requirement. If you want the new feature, you only need to do one thing: Update the pubspec.yaml in one place (and then migrate code as neccessary). If you want to stay back, you can also just not update the pubspec.yaml, but you will not get any new features at all (and you can't use experiments flags either, those are always considered incremental to the newest SDK versions). Or you can update it, and keep a few libraries back, but that feature is only intended for allowing gradual migration, not for long-term opt out. Even though it can obviously be used that way, but keeping a single package on multiple versions for a longer time is just not convenient for anybody.

[leafpetersen]

So, the design here goes for a design which encourages upgrading, by making it easy, even automatic, when you change the SDK requirement

An implication here is that you can't bump your min SDK constraint without opting into any of the "opt-in" breaking changes in that release. This may be the correct choice, but... it's a fairly aggressive interpretation of "opt-in".

and you can't use experiments flags either, those are always considered incremental to the newest SDK versions

I'm not 100% sure what this means. Certainly you need to be able to use experiments flags even if you have packages in your build that haven't incremented to the newest versions.

[munificent]

I believe this is the right choice.

Me too.

An implication here is that you can't bump your min SDK constraint without opting into any of the "opt-in" breaking changes in that release. This may be the correct choice, but... it's a fairly aggressive interpretation of "opt-in".

Sure, but the other way to look at is is why would you bother to raise the minimum constraint if you don't want to opt in? If you aren't using any of the new stuff, just leave it alone and your package will work on a wider range of SDK versions, which can be nice for your users.

At the point that you do want some of the goodies, yes, you have to take all of them, even the ones that require some migration. This does put some pressure on us to ship tooling to make the migration as easy and automatic as possible. If you, say, just want some new core library method, and you have to manually migrate a bunch of stuff for some syntax sugar you don't care about, you won't be happy.

But putting pressure on us to ship good tooling is probably a good thing. The ultimate goal is we want most users on the latest Dart most of the time, and good tooling is an important part of that.

[leafpetersen]

Sure, but the other way to look at is is why would you bother to raise the minimum constraint if you _don't_ want to opt in?

Bug fixes? New APIs? I don't disagree with our goal, just feeling a little cautious about this. Users right now don't think of raising their min SDK constraint to a higher minor version as opting into breaking changes. We're saying that they will now have to do so. We're also saying that they will have to raise their min SDK constraint in order to get new features. This feels a little like taking the radio control dial on the car and saying "now this is how you start the car too". It may be a reasonable choice, but folks might be a wee bit confused for a while. We'll definitely need to do some educating about this, or we're going to get a lot of confusion of the form "I downloaded the newest Dart SDK but none of the new features you're talking about work", or "I bumped my min SDK to get a bug fix and all of my code broke!".

[lrhn]

The alternative option I can see is to add an extra field in pubspec.yaml which restricts the language to an earlier version than the SDK constraint. It's basically setting the default opt-down language level for every library, but a library can still opt for any language level up to the actual SDK constraint.

That way, you can update the SDK constraint to ^2.5.0 while keeping all libraries at language level 2.4, without having to do an opt-down on every library individually. This allows you to use APIs included in 2.5, and you can even upgrade individual libraries to language level 2.5.

The main issue with such an approach is that you need to clean up your pubspec.yaml file at some point. If you upgrade every library to 2.5, you seem to be fine, but there is an unused "default-language-level: 2.4" line in the pubspec.yaml, and if you forget that when adding a new library, it might bite you.
Again, a good tool could detect that every library (even the ones in test/bin/whatnot) has a version override, and tell you to clean up pubspec.yaml.

By forcing you to add the library downgrade to every library, you get a visible and searchable reminder for each library which is not up-to-date, and when you are done migrating libraries, you are done.
I'd prefer having a tool to add those markers on every library when you upgrade, to having a tool for detecting and removing them when they are no longer necessary.
Imagine that pub sdk-upgrade 2.5 would increment the SDK requirement in pubspec.yaml, and if SDK 2.5 introduces a breaking change, it will also modify every library file with //@sdk(2.4) (or whichever syntax we use) and tell you that you need to migrate (and which migration tool to use). Or maybe the migration tool is the one inserting the version comment in its --safe mode, and you can run it in optimistic mode to do conversions that might need hand-tuning afterwards.

[munificent]

This allows you to use APIs included in 2.5, and you can even upgrade individual libraries to language level 2.5.

I'm worried this may cause problems with some future language/API changes where this separation doesn't make sense. Examples:

• Say we add extension methods in 2.4 and also define a bunch of new extension methods in the core library. What does it mean to use the 2.4 core lib with the 2.3 language? Are those extension methods accessible? How?

• Say we add tuples in 2.5 and add some new core lib methods that return them. If you set the SDK constraint to 2.5 but the language level to 2.4, what happens when you call those methods?

• Say we add extension methods in 2.3, and tuples in 2.4. In 2.4, we add an extension method to a corelib that returns a tuple. What happens if I try to make a library at language level 2.2 and lib level 2.4?

We have a well-defined answer for this around NNDB (legacy types), but that answer took a lot of effort design and will have a high implementation cost. Are we signing up for doing that kind of additional work for each language change?

I'm hesitant to say that the version selection in a library only applies to the language and not the core libraries because that basically gets us back to having to support the composition of each language/corelib pair.

[lrhn]

I do believe we are signing up to make every new feature backwards compatible in some way. We do not have library API versioning. There is no "library level". I wish we did have that, but that's a completely different feature than language syntax/semantics versioning, and I haven't found any good approach.

• Extension methods may or may not work in libraries using pre-extension method language. We can choose either, it's not important to the feature. You can't declare any because the syntax is not available. You should be able to export them through a non-opted in library. (But it's a good point that we should address in the extension methods design document).

• Tuples will likely be treated as Object in non-opted-in libraries, if tuples are objects at all. Otherwise we have a bigger type system issue. You can't create them, you can't destructure them, but you can pass them through if needed, and can likely accept tuple types as type arguments.

So, either a language level 2.2 library cannot call extension methods at all (in which case we're done), or it calls the core library extension method which returns something that it only knows to be an Object.

More interesting is what happens if it tries to call a method accepting a tuple with an object that is a tuple, but which only has static type Object. With NNBD, there won't be a down-cast. Prior to that, will there be one? Probably. Or post-NNBD, you can cast it to dynamic and get an implicit downcast to a type that you cannot write explicitly.

That is the kind of integration that we need to do in order to have different language level libraries interact. This is not about platform libraries, any two libraries with different versions have the same issue, so we need to solve the issue. That solution should work for platform libraries too.

[lrhn]

Decision: There will be no way to upgrade a library above the "default level" (which would also need a way to set a default level below the minimum available level of the SDK constraint).

Migration tools should make it easy to update the SDK dependency and add downgrade markers to an entire package (or just migrate the package for you).
Removing downgrade markers will then be all you need to do to finish migration.

If you use a default-level declaration and upgrade markers, then at the end of migration, you would have to clean those up. That extra step is an extra risk wrt. getting packages upgraded. A staged migration that happens over multiple SDK releases might leave files that are fully compatible with the newest version at the previous version, just because they had to be marked to get up to that version.

[leafpetersen]

I'd like to re-open this discussion in the context of the code generation discussion.

@jakemac53 points out that allowing this would solve a lot of the issues that we have right now with code generators. Essentially, the problem we have now is that if a code generator generates code marked with (for example) @dart=2.5.3, it has no way of ensuring that libraries so marked won't get injected into a package which still has not opted in. Consequently, the pub version solver may upgrade you to a new version of a code generator which doesn't actually work with one of your libraries.

If we allow opting forward, then this would just work. The code generator would force the SDK you're running on to be at least 2.5.3, and the code generator is then free to inject 2.5.3 opted in libraries into packages that have otherwise not opted in.

This also helps with dealing with code generators inside of google, where we only have one version: instead of having to stay on an old version until all target libraries are upgraded, a code generator could start generating opted in code at any time.

As discussed above, this potentially re-introduces the problem of being able to publish code that doesn't actually work with your min-sdk constraint. There are three possible resolutions of this:

• The analyzer might already sufficiently understand code generation that we could only allow opting forward in generated code. This code is not published. So we could still disallow opting forward in actual user code.
• Alternatively, we could implement a pub check (client side or server side) to validate that you are not publishing opted forward code
• Or we could just not try to check for this. We're then allowing people to go wrong if they go out of their way to do so, but only in a way that they could have gone wrong anyway before we added language versioning.

cc @srawlins @davidmorgan @jakemac53 @grouma @matanlurey

[lrhn]

If we allow opting forward, then this would just work.

This seems to assume that the SDK running the code generator is also the SDK running the generated code. (Or I guess that is what the second item would ensure).

It seems there are at least two cases: Dev-time code generation and build/link-time code generation.

With dev-time code generation, the code generator is run by the developer of the library, and the generated code is published normally. The clients and end users are oblivious to the code generation.

With build-time code generation, the code generator is run by the library client while building their own application. If they build files into other packages, then they are breaking the pub package abstraction (the package is no longer uniquely defined by its version number). I don't know if this happens, but if it does, it's going to be messy. If it just generates files in its own library, then it's basically a dev-time code generation.

Is there a third option, where the code generator is run by the end-user (JIT-code generation)? That one can know for certain which SDK will run the program.

For a dev-time code generator, the code generator will likely be a dev-dependency. Then it would meant that it would force your SDK to be >2.5.3 when developing the package, but when you publish the package with a 2.1.0 SDK dependency, nothing prevents someone using a 2.1.0 SDK from using the package.

If you publish //@dart=2.5 code, you really, really need to set your min-SDK to at leat 2.5.0. I don't see any way around that.
Adding a verification step here would prevent publishing that without updating the minimum SDK version to 2.5.0.

[jakemac53]

This seems to assume that the SDK running the code generator is also the SDK running the generated code. (Or I guess that is what the second item would ensure).

For all build-to-cache builders externally this is the case (the build is specific to the root package being built, and the entire thing is invalidated if the sdk version changes). These files would not be published on pub.

Technically somebody could do a build and then try to run the outputs of that with a different SDK but that would already not be expected to work since snapshots/kernel are not compatible across versions.

Is there a third option, where the code generator is run by the end-user (JIT-code generation)? That one can know for certain which SDK will run the program.

I believe this is the case I am talking about - I don't fully understand the 2nd case you listed though.

For a dev-time code generator, the code generator will likely be a dev-dependency. Then it would meant that it would force your SDK to be >2.5.3 when _developing_ the package, but when you publish the package with a 2.1.0 SDK dependency, nothing prevents someone using a 2.1.0 SDK from _using_ the package.

Correct - fwiw this is already the case today and I haven't seen evidence it is causing issues in practice. The analyzer does already have hints about not having a high enough lower bound on the sdk when using some language features which helps.

It can also be mitigated with a check on publish as you/leaf suggest, which I think would be sufficient (and in fact put us in a better place than we are today).

[lrhn]

OK, so we are saying here that from the run-time SDK's perspective, there is no problem having some libraries being ahead of the package's default language level. The SDK will either be able to run those, or it won't. It's all just code, and it either works, or it doesn't.
If the code generator is applied by the run-time SDK, and the code generator itself requires the SDK it generates code for, then it cannot possibly generate code that cannot be executed.

The moment you separate code generation from code running, you need to have some sort of contract between the two, something which specifies the minimum and maximum required language levels that the running SDK must support in order to run the code. That doesn't have to be in the code, it can be specified in metadata (maybe in Kernel files).

So, if we allow individual library language version numbers above the Dart package default level at run-time, but ensure that pub does not allow any library version number above the Pub package minimum required language level, then we should be good.

Our tools will use the Pub package minimum language level as the Dart package default language level, but that's just a (convenient) convention. The two are technically separate concepts, one specified in pubspec.yaml and used for constraint solving, and the other specified in .packages and only used for assigning a language level to individual libraries when compiling them. The latter doesn't have to satisfy any constraints

[leafpetersen]

This will be allowed, but pub will complain if you publish the code. Closing the issue again as resolved.

[mit-mit]

This will be allowed, but pub will complain if you publish the code.

The issue tracking this is https://github.com/dart-lang/pub/issues/2316