Language: Proposal: Allow downgrading of the language version (below the min sdk) with a field in the pubspec

[jakemac53]

Packages may depend on non-language related functionality in a newer sdk (compiler features, new core library apis, etc), but be blocked from updating to the latest language version for some reason (your transitive deps may need to be migrated first for example).

The way language versions work today (based on min sdk constraint alone) you would be required to manually downgrade your language version in all files in your package, as there is no central way to define it outside of the sdk constraint.

I would like to propose a new optional language field in the pubspec.yaml, under the environment section:

environment:
  sdk: ">=2.9.x <3.0.0"
  language: "2.8"

If present this would be the language version that is copied into the .dart_tool/package_config.json file instead of deriving the language version from the min sdk constraint.

I propose three constraints the values of this field:

• It may only contain a concrete language version (no ranges etc)
• You will only be able to downgrade your language version in this manner (it must be <= the min sdk constraint)
• You cannot downgrade the major language version, only the minor language version

cc @leafpetersen @jonasfj @munificent @lrhn @natebosch

[natebosch]

I don't think this solves the root of the problem which is that there are 2 meanings for language version 2.9 and that upstream deps may not even be aware that a consumer can change out the interpretation of their code between the 2 meanings without their control. Adding this control allows an all-knowing package maintainer to pick a language version like 2.8 which does have only one meaning - but they have to be aware enough of both meanings of 2.9 to know that they need to do that. It's also a roundabout way to express their intent - using language version 2.8 isn't _really_ their goal. Their goal is just to avoid a language version with 2 meanings.

In other cases where a language version has a single meaning I think we generally prefer that a package should be forced to migrate before they can use new features - that was one of the ways we planned on motivating the ecosystem to move forward.

[jakemac53]

The issue you describe @natebosch is specific to experiments. This proposal does provide an escape hatch for that scenario for sufficiently informed package authors, but it is not the goal of this proposal to solve that problem.

In other cases where a language version has a single meaning I think we generally prefer that a package should be forced to migrate before they can use new features - that was one of the ways we planned on motivating the ecosystem to move forward.

There is some motivation there for sure, but this is an overly large hammer to use. The ability to migrate is not always in the control of the packages (they may be blocked by transitive deps).

We still have the motiviation in terms of not allowing you to use new language features until you opt in, as well as a linear path in terms of what language features you can use (ie: you must adopt null safety before you can use variance).

[jonasfj]

You can do this by just adding a _language version marker_ //@dart=2.8 to the start of all files, right...
Hence. users can script their way out of this problem...

So the question becomes if this is a common scenario?

[lrhn]

There is no technical issue with this approach.
It's a pub-only change where it has an override for the language version it writes to the package_config.json file. It's backwards compatible because no-one uses the override yet.

The real question, as Jonas puts it, is whether it's a common enough need that it's worth the extra complexity for Pub.
There is a workaround. It's currently cumbersome, but I think we should have an option to dart migrate which just updates the SDK dependency and adds //@dart=old.version to every .dart file in the package. (Then you can upgrade individual libraries afterwards).

The current language marker design is deliberately ugly and annoying. You get reminded, every time you look, that this file is being held back. We want it to be ugly and visible if some files are not on the newest language version.

The one thing I do not want is someone doing sdk: >=2.9.0 and languageVersion: 2.8 in the pubspec and then writing //@dart=2.9 in individual files.
That would be allowed by this feature because using //@dart=2.9 is perfectly valid and compatible with the min SDK version of the package, but it breaks the design goal that when you are done migrating, there is nothing more to remove.

For that reason, I'm somewhat opposed to allowing a blanket downgrade, and would prefer to give people an easy way to mark every Dart file instead.

[jakemac53]

The one thing I _do not want_ is someone doing sdk: >=2.9.0 and languageVersion: 2.8 in the pubspec and then writing //@dart=2.9 in individual files.

We could easily block this in the same way that we discussed not allowing you to increase the language version beyond that of the min SDK constraint. I am not sure if there is really any value in explicitly blocking it though, it feels arbitrarily restrictive to me.

For very large packages it would be easier to add // @dart=2.9 to files one at a time instead of removing // @dart=2.8 (at least in the current state where there is no tool to add a language version comment to all files)?

The real question, as Jonas puts it, is whether it's a common enough need that it's worth the extra complexity for Pub.

While there is some extra complexity in Pub, I would argue it is quite limited in scope. Less than a day of work including validation, tests, etc.

I think that the overall ROI on this quite good.

There is a workaround. It's currently cumbersome, but I think we should have an option to dart migrate which just updates the SDK dependency and adds //@dart=old.version to every .dart file in the package. (Then you can upgrade individual libraries afterwards).

I am more concerned about the long term technical debt this incurs in packages such as these. Every time a new file is created the developer must remember to add this comment to it, and that will be easy to miss in code review. Forgetting to do so has long-lasting impacts that quite likely require a breaking change to revert (downgrading the language version is quite likely breaking).

I would argue that even a single package being in this state incurs significantly more technical debt than the Pub feature does.

[lrhn]

For very large packages it would be easier to add // @dart=2.9 to files one at a time instead of removing // @dart=2.8 (at least in the current state where there is no tool to add a language version comment to all files)?

Agree, so let's add that tool instead of supporting an undesired workaround.

I'm not too concerned about someone accidentally getting the newest language version for a library. If it's not intentional, then it's unlikely to work at all. Doing major reworking of your package in the middle of a language migration is also not such a great idea.

The goal is to get everyone to the newest version. Allowing you to stay back is intended to be possible, but not necessarily comfortable.

[jonasfj]

The real question, as Jonas puts it, is whether it's a common enough need that it's worth the extra complexity for Pub.

While there is some extra complexity in Pub, I would argue it is quite limited in scope. Less than a day of work including validation, tests, etc.

It's worth noting that:
(A) Adding this feature will continue to incur cost after we've all migrated to null-safety.
(B) This feature is mostly relevant for null-safety, as future language version bumps probably won't be as hard to upgrade to.
(C) Publishing of mixed-mode packages on pub.dev is _not recommended_ (afaik).

I don't really have strong opinion on this.

Packages may depend on non-language related functionality in a newer sdk (compiler features, new core library apis, etc)

I suppose you can use them, but not declare dependency on the sdk-version in pubspec.yaml. So users of your package with old SDKs would be broken when upgrading.
This could be worked around by having a single dependency upon some dummy package with a higher lower-bound SDK constraint (not saying that this is ideal :see_no_evil:, but mixed null-safety packages is already _not recommended_).

[jakemac53]

(A) Adding this feature will continue to incur cost after we've all migrated to null-safety.

All code has some ongoing cost, sure. The ongoing cost of this should be very low (travis cycles for tests, etc). Is there something in particular you are worried about?

(B) This feature is mostly relevant for null-safety, as future language version bumps probably won't be as hard to upgrade to.

I am not so sure about this assumption. I could very easily see us using language versioning in the future for syntax related changes. As an example if we needed to make some breaking syntax changes to allow for optional semicolons.

We have put a huge investment into language versioning and might as well use it in the future.

(C) Publishing of mixed-mode packages on pub.dev is not recommended (afaik).

That isn't the primary goal here - I actually want this feature to help me avoid _accidentally_ doing exactly this. The only real alternative is to add dart language comments in every single file, which means people will forget to do that at some point, and we end up with some random libraries that are opted in.

[jakemac53]

Agree, so let's add that tool instead of supporting an undesired workaround.

The tool is the undesired workaround, imo. This is a longer term solution to the problem that is far less hacky than adding language comments in every single file in your package.

I'm not too concerned about someone accidentally getting the newest language version for a library. If it's not intentional, then it's unlikely to work at all.

It might work just fine (plenty of code, especially newer code, needs zero nnbd changes). And then you accidentally end up with a mixed mode package.

Doing major reworking of your package in the middle of a language migration is also not such a great idea.

In most of these cases you haven't even started the migration yet.

The goal is to get everyone to the newest version. Allowing you to stay back is intended to be possible, but not necessarily comfortable.

There is going to be an extended period of time where its not even possible to migrate in a sane manner for a lot of packages. We should not be punishing those packages for things that are not their fault.

[natebosch]

In another thread (that I can't currently find), I think @mit-mit was most concerned about the ambiguity and risk over changing how we derive a language version from a pubspec.yaml file, and that it was too late at this point to consider such changes. We are now considering such a change by switching how we interpret a missing lower SDK bound in a pubspec, so that particular ambiguity and risk is going to be resolved either way.

If I understand correctly, the remaining concerns we have are:

• Ongoing implementation complexity in pub.
• Increased complexity for users to understand how to interpret the language version for some code they are looking at.
  
  
  
  • The language_version: config is strictly easier to interpret than language markers where they need to be used. Users are already expecting to look at the pubspec to understand the language version in the vast majority of cases, seeing an explicit config is not any harder than interpreting the lower bound SDK constraint.
  
  
• Risk from having to rethink things like what validation happens. Additional complexity for validating the new config, or whether that change impacts how we validate language version markers in file.
  
  
  
  • IMO this is not complex and is limited to pub. The validation for the config is the same as the language markers, and we only need to check that we validate against the lower bound SDK, and not against the overall derived language version, for language markers. Other validation already happens against package_config.json, not against pubspec.yaml.
  
  
• Making it "too easy" to downgrade language versions will remove pressure to migrate, and we want to keep pressure high so that the ecosystem migrates quickly.
  
  
  
  • IMO this isn't an effective approach to that pressure. The folks impacted by this are more likely to be the ones who would migrate if they could, but are held back by dependencies. The folks that won't migrate but can are also unlikely to be actively maintaining the package where they would need to bump the min SDK for some new feature or bug fix.
  
  

@munificent @leafpetersen - did I miss anything?

Some benefits that we get.

• Less risk around maintaining packages that need new SDK features or bug fixes but are held back from migration by dependencies.
• Easier for users to understand when one of their dependencies is not null safe. They are used to looking at the pubspec to check null safety, and an explicit config (for the packages that use it) will be easier for them to understand than to know they need to look at the library files for language version markers.

[jakemac53]

In an attempt to quantify some things, here are the number of packages depending on stable versions 2.7 through 2.10:

2.10.x: 78
2.9.x: 105
2.8.x: 429
2.7.x: 3836

I can't presume exactly _why_ they chose to increase their sdk constraints, but there are a fair number of packages who _did_ increase them to the latest current stable. Here are all of the reasons I can think of that users _might_ set the latest stable as their sdk constraint, in no particular order:

• [ ] It is a new package, and they just set it to the latest one.
• [x] They want to run their tests on their oldest supported SDK, and one of their deps requires 2.10
• [x] They require some new feature or functionality from the latest sdk.
• [ ] They just find it easier to reason about versioning that way

For 2 of these 4 reasons that I listed (the checked ones), they do _need_ to downgrade their entire package in some way, and so this feature would apply to them (this assumes they are not ready to migrate themselves for whatever reason).

I think probably the most common one will be the 2nd, and I think it is a pretty compelling use case which will potentially become quite prevalent (much more so than any release post 2.7). The reason I think this will become prevalent is that many packages will be publishing breaking change releases, which require the 2.12 sdk.

All packages that depend on those packages will need to either support multiple major versions of the package (highly not recommended in this case given one is opted in), or also increase their min sdk constraint to 2.12 to be able to test on their min sdk version. However, if they are blocked from migrating themselves then they will have to opt out their entire package. This proposal should significantly improve the situation for any package in that situation.

[leafpetersen]

ackages will need to either support multiple major versions of the package (highly not recommended in this case given one is opted in), or also increase their min sdk constraint to 2.12 to be able to test on their min sdk version.

I don't understand this. Why do I have to increase my min sdk constraint to 2.12 to be able to test on 2.12? That makes no sense to me, what am I missing?

[jakemac53]

I don't understand this. Why do I have to increase my min sdk constraint to 2.12 to be able to test on 2.12? That makes no sense to me, what am I missing?

The problem comes in as soon as you depend on a package whose min sdk is 2.12 (which your packages will do when they migrate).

So, lets say you have two packages a and b, where b depends on a. The a package migrates to null safety, does a breaking version bump, and sets its min sdk to >=2.12.0.

Now package b, wants to upgrade to the next version of a, so it changes its constraint from say ^1.0.0 to ^2.0.0.

At this point, the b package can only get a version solve on the 2.12 sdk, since one of its deps requires that. It is common in this case to increase your own min sdk constraint to the same version, since you can no longer test your package on what your min sdk used to be.

You could, as the author of the b package, use a constraint like >=1.0.0 <3.0.0 on the a package, but I would argue that is a pretty risky thing to do.

[leafpetersen]

since one of its deps requires that. It is common in this case to increase your own min sdk constraint to the same version, since you can no longer test your package on what your min sdk used to be.

Ok, but... just don't? The claim was that you can't test on a 2.12 sdk unless you update your min sdk constraint, but I don't see why that's relevant. Leave your min sdk constraint where it is, and test on 2.12, what's the problem?

[natebosch]

Leave your min sdk constraint where it is, and test on 2.12, what's the problem?

Our best practice is to have tests that run against the min SDK. We have caught a number of mistakes because of this practice, but we don't follow it 100%.

[leafpetersen]

Following up on a couple of concrete details. The original proposal specified three constraints on the language version field:

• It may only contain a concrete language version (no ranges etc)
• You will only be able to downgrade your language version in this manner (it must be <= the min sdk constraint)
• You cannot downgrade the major language version, only the minor language version

In addition the proposal is that language version comments in files be required to be less than or equal to the min sdk constraint (not the language version). It's not 100% clear to me that this is really feasible without also changing the format of package_config.json to include the sdk constraint (or making tools that don't currently read pubspec.yaml do so). If not, as an alternative, we could simply require that language version comments be less than or equal to the language version written into package_config.json.

Tools that we believe would need to be updated to support this change are:

• The sdk script that updates package_config.json
• The corresponding flutter script
• pub
• the analyzer

[munificent]

• Users are already expecting to look at the pubspec to understand the language version in the vast majority of cases, seeing an explicit config is not any harder than interpreting the lower bound SDK constraint.

But they won't be interpreting an explicit config. They'll be interpreting an explicit config and also an SDK constraint. And in most cases, the explicit config will be absent. So they need to learn to read the SDK constraint, but also remember to disregard it if the other config is present.

My experience from pub is that users don't allocate a lot of brainpower to thinking about versions so our complexity budget here is low. No one wakes up and says, "Man, I can't wait to solve some semver constraints in my head today!"

@munificent @leafpetersen - did I miss anything?

Nope, I think you captured my concerns.

[jakemac53]

My experience from pub is that users don't allocate a lot of brainpower to thinking about versions so our complexity budget here is low. No one wakes up and says, "Man, I can't wait to solve some semver constraints in my head today!"

I disagree that this actually adds any meaningful complexity, it has nothing to do with semver, and it is strictly more obvious and easier to understand than the current default (min sdk version).

Consider an A/B study where you give users one of the following two pubspecs and asked them what the language version of the package is:

environment:
  sdk: ">=2.8.0 <3.0.0"

and

environment:
  sdk: ">=2.8.0 <3.0.0"
  language_version: "2.7"

I strongly believe that they would be _less_ confused by the latter pubspec. It is clear and obvious, regardless of your understanding of language versioning, what its version is. It is _less_ taxing on their brain, because it says it right there explicitly.

It also adds no additional tax when not present, because well its not present :).

[natebosch]

They'll be interpreting an explicit config _and also_ an SDK constraint. And in most cases, the explicit config will be absent. So they need to learn to read the SDK constraint, but also remember to disregard it if the other config is present.

This seems easier for users than to learn to read the SDK constraint, but also remember to disregard it after scanning every file in lib/ to look for an opt out comment.

I think we are approaching this discussion from different angles.

The question for me is _not_ "do we want users to opt out of null safety using this config?" The question is "Given that some users _will_ need to opt out of null safety due to their dependencies, would we rather those users were doing it with a single line of config in the same file other users will look at to know the language version, or would we rather those users opt out all libraries in their package one by one and have it hidden away from users looking at the pubspec."

I do no understand the idea that making a _necessary_ task less painful also makes it more necessary.

[munificent]

it is strictly more obvious and easier to understand than the current default (min sdk version).

But your proposal does not remove their need to also understand the min SDK version.

Consider an A/B study where you give users one of the following two pubspecs and asked them what the language version of the package is:

That's not the right study, though. A user will encounter many pubspecs and will likely end up needing to understand all of the combinations of uses. If you want to see that they can reason about the behavior the two approaches take, an A/B would look like:

A

package a

environment:
  sdk: ">=2.8.0 <3.0.0"

package b

environment:
  language_version: "2.7"
  sdk: ">=2.8.0 <3.0.0"

package c

environment:
  language_version: "2.7"

package d

environment:
  language_version: "2.9"
  sdk: ">=2.8.0 <3.0.0"

B

package a

environment:
  sdk: ">=2.8.0 <3.0.0"

...there are no packages b, c, d

This seems easier for users than to learn to read the SDK constraint, but also remember to disregard it after scanning every file in lib/ to look for an opt out comment.

As I understand it, the two options we're considering are:

Current behavior

1. If the file has a language version comment, use that version.
2. Else if there is a pubspec with an SDK constraint, infer from its lower bound.
3. Else use the latest version.

With explicit field

1. If the file has a language version comment, use that version.
2. Else if there is a pubspec with a language_version field, use that.
3. Else if there is a pubspec with an SDK constraint, infer from its lower bound.
4. Else use the latest version.

I'm having trouble understanding how the second is supposed to be simpler than the first. It contains all of the existing logic of the first and adds one more rule for users to reason about. It doesn't remove any of the existing (already fairly high) complexity.

[jakemac53]

I'm having trouble understanding how the second is supposed to be simpler than the first. It contains all of the existing logic of the first and adds one more rule for users to reason about. It doesn't _remove_ any of the existing (already fairly high) complexity.

When present, I think it does remove ambiguity/complexity, because it explicitly tells the user the information they are seeking. When not present it doesn't change anything from what we have today.

I don't agree that the _possibility_ of it being present adds meaningful complexity, since it is configured right next to the sdk constraint, so you don't have to look in any extra places. Even if you are sufficiently informed to know that it _could_ be put there it doesn't make it more complicated except in a pedantic sense that there is technically one additional thing at play.

I agree there is a _small_ implementation and documentation overhead here, but I don't think it goes beyond that to something that could confuse users.

[leafpetersen]

When present, I think it does remove ambiguity/complexity, because it explicitly tells the user the information they are seeking. When not present it doesn't change anything from what we have today.

I think overall I agree with this. I think seeing this in the pubspec will never be confusing, and not seeing it will be no more confusing than it is now. And it if does discourage some uses of // @dart = 2.9 then overall it will be a win in comprehensibility.

[lrhn]

My main worry is that people will use language-version; 2.9 in the pubspec and then migrate incrementally by adding // @dart=2.12 to migrated libraries. When they are done, they'll then have to do a big sweep to remove the language-version entry and all the //@dart=2.12 markers. They can easily forget a marker, and they have no easy way to know that they are done migrating (it's easier to grep for //@dart=2.9 than search for a .dart file without //@dart=2.12).

If they just migrate by running dart migrate, and that command is language-version aware, then there shouldn't be a problem.

We could prohibit publishing with a language-version and a later //@dart=... marker, but it's not unsound as long as the marker is not above the SDK constraint, so prohibiting it is needlessly restrictive.

Other than that worry, which breaks the migration design where there is nothing to remove when you are done, the feature itself is simple and safe.

[jakemac53]

My main worry is that people will use language-version; 2.9 in the pubspec and then migrate incrementally by _adding_ // @dart=2.12 to migrated libraries.

You could do exactly this today with the min sdk constraint, you just can't _publish_ that way.

@leafpetersen has brought up that there is actually a decision to be made here that I have not yet explicitly called out, which is whether we would block upgrading the language constraint beyond the _min sdk_ or the _explicit language version field_ when both are present. Either is safe, and my inclination would be to still use the min sdk since that is the least restrictive. But we _could_ instead say you can't upgrade beyond the _default language version_ for the package and publish, which would make the feature behave identically to how the min sdk behaves.

[munificent]

When present, I think it does remove ambiguity/complexity, because it explicitly tells the user the information they are seeking. When not present it doesn't change anything from what we have today.

I don't agree that the _possibility_ of it being present adds meaningful complexity, since it is configured right next to the sdk constraint, so you don't have to look in any extra places. Even if you are sufficiently informed to know that it _could_ be put there it doesn't make it more complicated except in a pedantic sense that there is technically one additional thing at play.

I would agree with this if every Dart users only ever had to read a single pubspec. But Dart users end up interacting with a wide variety of packages, and they need to be able to understand all of them. So the total cognitive load is the union of what all of those packages do, not just what any particular package does.

The argument that "It's OK to add feature X because users who don't use X won't need to learn X" is how you get C++. It is true that you can be a proficient C++ programmer in any given C++ codebase while only knowing a fraction of the whole language. But look at how tiny the C++ code reuse ecosystem is compared to other languages, despite being an older language and fantastically popular. That's because there is no "C++". There's a thousand different subsets, all mutually incomprehensible to each others' users.

Obviously, this one feature won't turn Dart into C++. But I believe very strongly that we have to always be disciplined about adding complexity because it does pile up. There is no single feature of C++ that turned it into what we think of today. It's a death of a thousand cuts.

[jonasfj]

Our best practice is to have tests that run against the min SDK.

I can see this is sensible :D

But are we going to have packages using language_version: 2.7 for long?
While we wait for packages to migrate or be _discontinued_ -- will it matter much that we deviate from _best practices_?

I agree, adding _language-version-markers_ everywhere is a bad idea, especially now that we've simplified our null-safety analysis. And if we think this will be necessary in more than a few obscure packages I'm inclined to agree that this is worthwhile feature.

---

• Ongoing implementation complexity in pub.

I'm not particularly worried about the implementation of this in:

• dart pub get
• dart pub upgrade
• dart pub publish (which I think will cover most important warnings about upgrading beyond minimum SDK constraint)

But changes will also be required in:

• analyzer quick fixes for bumping _minimum SDK constraint_,
• documentation for language versioning,