Pub: Allow multiple versions of the same package to be used by an app

[amirh]

Pub currently forces resolving to a single version of each package in the dependency graph. This can be over restrictive for second-level dependencies.

The failure scenario (which I have hit multiple times) is:

1. An app depends on package_a: ^1.0.0 and package_b: ^1.0.0.
2. Packages package_a v1.0.0 and package_b v1.0.0 both depend on package_second_level: ^1.0.0.
3. package_second_level gets a breaking change and the major version is bumped to 2.
4. package_a v1.0.1 is updated to depend on package_second_level: ^2.0.0, package_b was not updated to use the new package_second_level yet.
5. Something forces package_second_level: >=2.0.0 for the app (this may be a result of e.g flutter upgrade updating pinned dependencies, or of the app developer needing a new feature available in the newer versions of package_a).
6. pub cannot solve the dependency constraints of the app 😢 .

If package_second_level is only used as an internal implementation detail in package_a and package_b I would say it is reasonable to build the app with 2 copies of package_second_level's code at different versions (binary size is the cost, but I'd argue it's preferred over not being able to build the app).

We still need to have the kind of dependency that allows only a single version for an app; this is for second level dependencies that are exposed as part of a package's public API.

cc @jonasfj @mit-mit @collinjackson @ianh

[mit-mit]

I'd much rather work on better tools and bots for ensuring that package_a and package_b quickly get onto package_second_level: ^2.0.0.

cc @munificent who can share more details about the historical considerations that lead to the current design.

[amirh]

I'd take quick updates of all packages over this thing any day 😄
Though it is worth considering that updating both packages may require non-trivial changes and may take awhile regardless of how good the incentives and tooling are.

I'd love to use this issue to get a good understanding of the pros and cons of this suggestions, and if it seems like an overall good idea we I'd suggest we understand the amount of effort required to make this happen before deciding on a priority.

[jakemac53]

Dart library identity is defined by the uri the library is imported with. So two imports of package:foo/foo.dart always necessarily must refer to the same library. This isn't a pub thing but a Dart language thing, and its pretty fundamental.

Even if that somehow changed - now you would get very confusing behavior where Package A and Package B leak some instance/type from package_second_level which look equivalent but aren't actually - leading to confusing a Foo is not a Foo errors.

Additionally there is only a single .packages (or soon package_config.json) file per application - individual libraries cannot pick and choose their own versions of packages.

[amirh]

Dart library identity is defined by the uri the library is imported with. So two imports of package:foo/foo.dart always necessarily must refer to the same library. This isn't a pub thing but a Dart language thing, and its pretty fundamental.

Even if that somehow changed - now you would get very confusing behavior where Package A and Package B leak some instance/type from package_second_level which look equivalent but aren't actually - leading to confusing a Foo is not a Foo errors.

Additionally there is only a single .packages (or soon package_config.json) file per application - individual libraries cannot pick and choose their own versions of packages.

What if we add something like internal_dep: package_second_level ^1.0.0 to pubspec.yaml which will rename the package (and the Dart code of package_a will actually be importing the internal name of the package).

(probably there are many edge cases I'm not considering right now, but just throwing an idea)

I'd guess we could potentially do some enforcement that "internal dependencies" are not leaked through public APIs.

[jakemac53]

What if we add something like internal_dep: package_second_level ^1.0.0 to pubspec.yaml which will rename the package (and the Dart code of package_a will actually be importing the internal name of the package).

All imports in the package itself would still have the wrong name :(

I'd guess we could potentially do some enforcement that "internal dependencies" are not leaked through public APIs.

This starts getting really complicated and restrictive though - types leak all over the place.

[jakemac53]

Note that this would also quite likely not just mean copying package_second_level but also possibly all of its transitive dependencies as well.

[jakemac53]

See also https://dart.dev/tools/pub/versioning#shared-dependencies-and-unshared-libraries which has some discussion on the philosophy here :)

[jakemac53]

Fwiw it took me a while to find but this issue has come up before and was closed with wont fix - https://github.com/dart-lang/pub/issues/462. That was a long time ago though.

[amirh]

Thanks for all the feedback!

My gut feeling (based on anecdotal experience) is that this problem is painful enough (for Flutter apps, where it's probably amplified as Flutter is pinning some dependencies) to justify considering remedies (even radical ones).

What if we add something like internal_dep: package_second_level ^1.0.0 to pubspec.yaml which will rename the package (and the Dart code of package_a will actually be importing the internal name of the package).

All imports in the package itself would still have the wrong name :(

I was thinking that all imports in the package will use the internal name for an "internal dependency".

This starts getting really complicated and restrictive though - types leak all over the place.

It just requires that a package author is explicit about leaking a type (by making sure the dependency is not an "internal dependency").

Note that this would also quite likely not just mean copying package_second_level but also possibly all of its transitive dependencies as well.

My guess is that it will be common to prefer a binary size hit over not being able to build your app. Ideally there could be a solution that only takes the binary size hit when it's required.

See also https://dart.dev/tools/pub/versioning#shared-dependencies-and-unshared-libraries which has some discussion on the philosophy here :)

Thanks! I read it again. It's a good read, the argument there against "unshared libraries" is totally valid IMO when you apply this approach to all dependencies, always. I guess what I'm suggesting here is allowing "unshared libraries" for "internal only" dependencies, where that problem does not apply.

Fwiw it took me a while to find but this issue has come up before and was closed with wont fix - #462. That was a long time ago though.

Thanks for the reference, IIUC that issue is talking about applying some sort of the "unshared libraries" approach to all dependencies, which isn't what I'm suggesting to consider here.

FWIW I find this discussion productive - if the decision is that this isn't something we'd want to do, I think it's useful to have the reasons against it documented (which is what this issue seems to be producing already).

[mit-mit]

where it's probably amplified as Flutter is pinning some dependencies)

Perhaps that is where we start then...

[jakemac53]

FWIW I find this discussion productive - if the decision is that this isn't something we'd want to do, I think it's useful to have the reasons against it documented (which is what this issue seems to be producing already).

Ya - I was surprised at my inability to find in depth similar discussions already, as I known it has come up in the past but maybe just more anecdotally.

There are valid arguments on either side of the debate to be sure - to be clear my goal hasn't been to shut down debate just highlight the reasons I am aware of that would make it difficult :).

where it's probably amplified as Flutter is pinning some dependencies)

Perhaps that is where we start then...

I do think flutter pinning dependencies is likely contributing negatively to the problem. It does allow them to be more confident about their releases but at the expense of holding their users back to old versions of packages :man_shrugging:.

[natebosch]

Another downside to this is that it is fundamentally at odds with the philosophy of our internal third party package management.

https://opensource.google/docs/thirdparty/oneversion/

Anything we do to make it easier for the package ecosystem to have long-standing discrepancies in version support for their dependencies will increase the burden of resolving these discrepancies while importing and updating packages internally.

[amirh]

Anything we do to make it easier for the package ecosystem to have long-standing discrepancies in version support for their dependencies will increase the burden of resolving these discrepancies while importing and updating packages internally.

Good point. IMO this is the strongest argument against this proposal. Thanks.

[jonasfj]

Maybe allowing packages versions to be retracted or flagged broken would alleviate some of the concerns around version pinning in Flutter.

Personally, I find the cargo model very attractive. But authors can also just publish mypkg2 when breaking compatibility, in those cases where the author suspects downstreamers would prefer multiple versions of mypkg.

[jupanubv92]

Because flutter_simple_shopify >=0.0.25-alpha depends on graphql ^3.1.0 and no versions of graphql match >3.1.0 <4.0.0, flutter_simple_shopify >=0.0.25-alpha requires graphql 3.1.0.
And because graphql 3.1.0 depends on http_parser ^3.1.3, flutter_simple_shopify >=0.0.25-alpha requires http_parser ^3.1.3.
And because firebase_auth >=1.0.0 depends on firebase_auth_web ^1.0.0 which depends on http_parser ^4.0.0, flutter_simple_shopify >=0.0.25-alpha is incompatible with firebase_auth >=1.0.0.
So, because flutter_store depends on both firebase_auth ^1.0.0 and flutter_simple_shopify ^0.0.25-alpha, version solving failed.

How do you fix these kind of issues when pub does not allow multiple versions of the same library?

[jonasfj]

Dependency overrides is an option:
https://dart.dev/tools/pub/dependencies#dependency-overrides

It's also possible that you need to downgrade other packages..

[jakemac53]

@jupanubv92 The ideal solution here is to file issues on your dependencies that don't allow the latest versions asking them to update. These messages can be a lot to decipher (usually because its actually a complex interaction they are trying to describe), but in your case the issue looks to be that the version of graphql that flutter_simple_shopify uses doesn't allow the latest http_parser, but the latest version of that is required by the latest flutter_auth_web. So I would file an issue on flutter_simple_shopify to update to the latest version of graphql.

You could try downgrading firebase_auth until flutter_simple_shopify does update, or using a dependency override on http_parser in the meantime (set it to the latest).

[jupanubv92]

@jonasfj, @jakemac53 thanks for your response. I gave up upgrading to Flutter 2.0 because as you can see I have package dependencies that haven't upgraded to the latest pub packages optimized for Flutter 2.0.

The only thing that I can do is to ask about the upgrade on these packages that haven't upgraded yet. However, the reason for my message here is to ask again the question to see if it is worth considering supporting multiple versions of the same library.
This will help with major Flutter release in future such as Flutter 2.0 where people have to wait weeks/months until all packages upgraded so they can benefit from the latest features.

[jakemac53]

@jupanubv92 yes I understand the use case. However I believe quite strongly myself that the cons of that approach would strongly outweigh the benefits. See my above comments to describe what I mean. The language itself really doesn't support this either (also described above).

[munificent]

if it is worth considering supporting multiple versions of the same library.

It's not really a question of "cost", at least in terms of Dart team cost to implement the support. It's not like if we had more time we could just implement it and be done with it.

Package sharing is fundamental to how pub and the Dart language itself works. Changing it would require significant change not just in our implementations, but in how users write code and all of the workflows and best practices around code reuse. It would significantly increase the complexity of reusing code and likely increase the size of shipped applications.