Pub: [Proposal/Discussion] "Symbolic packages" support

[amirh]

This is a proposal for allowing a single package to specify multiple version numbers.

Problem - a breaking change that doesn't affect all dependents

A breaking change to a package doesn't necessarily impacts all dependent packages, this is mostly relevant for big packages, for example Flutter:

Flutter plugins currently don't specify an upper bound for the Flutter versions they are known to work with. This currently makes sense as most Flutter breaking changes aren't breaking plugins. Most plugins only care about Flutter's services layer and the embedder's plugin API where breaking changes are less common.

However this leaves plugins unprotected from breaking changes to the services layer which they do depend on.

(A similar issue happens in a federated plugins model where breaking changes to the app-facing API are likely to not affect the platform-implementation facing API.)

Potential workaround

We can express these constraints by publishing dummy packages (e.g an empty flutter_services package) that's there just for affecting the constraint resolution.
A major downside of doing so is polluting the search results on pub.dev. This will also add complexity to the publishing process .

Proposal - symbolic packages

Allowing pub packages to specify "symbolic"/"virtual" packages may be a useful tool to address the issues above.

e.g package foo: 1.0.0 specifies that it provides the "symbolic" package bar versioned 1.0.0. This adds a ('bar', '1.0.0', deps: {'foo': 'any'}) node to the dependency graph (the node is only used for constraint resolution). Now another package, baz can depend on bar: ^1.0.0 which is only bumped when the bar component of foo gets a breaking change.

Additional potentially related problem - expressing constraints that are external to the pub-verse

For example the Android's migration to AndroidX requires that AndroidX-migrated plugins are only used by applications that have migrated to AndroidX. However pub is completely unaware of this requirement, and may resolve to using AndroidX-migrated plugins in applications that were not migrated.

"Symbolic" packages could potentially have been used to mitigate the AndroidX issue by having apps that migrated to AndroidX add a virtual androidx: 1.0.0 node. And have plugins that migrate to AndroidX depend on androidx: ^1.0.0.

@Hixie @mit-mit @jonasfj

[jakemac53]

One problem with this approach (as currently proposed at least) is that it defeats pubs validation that you depend on all the packages you use, which is based on import/export usage in the package. The user would simply have to know that they need to add this magic constraint because they are using some special api/feature of a package, but there would be no enforcement to keep them on the rails.

I think this feature could provide some value in some contexts but it could also easily lead to a false sense of safety leading to a lot of unintended breakage if there isn't any enforcement mechanism.

[amirh]

I guess explicit dependencies could still be enforced if the "symbolic" node won't depend on the package that generates it. e.g in the example above bar won't have any dependency, and baz will depend on bar: ^1.0.0 and foo: any.

[Hixie]

You could also require that baz depend on bar: ^1.0.0 _and_ foo: any or foo: >=1.0.0 or whatnot.

[jonasfj]

I'm not entirely sure I understand what a "symbolic package" is. Or what problem they are solving.

But if you have a package with two interfaces that should be versioned independently, it might be better to have two packages.

Example, package:logging has two sides to it:
(A) an interface used by packages that produces log messages, and,
(B) an interface used by packages that can read generated log messages and send them to a logging service.

Breaking (A) would affect a LOT of packages, breaking (B) would only affect packages like appengine that knows how to send log messages. In the case of package:logging both of these interface exist in a single package covered by a single version number. But it might be better if (A) and (B) were two different packages. In that case (A) might be a package that depends on (B) and exports a small subset of the surface of (B).

In the case of flutter_services, I suppose you could create a library in the Flutter SDK that can only be imported from the flutter_services package. Maybe that way the services API could be versioned independently of the Flutter SDK.

---

I think there is a lot of value in keeping these concepts simple: A package is name and a version number.

[jonasfj]

I had a chat with @amirh about this, from what I understand the goal is to:
Give a single package two (or more) version numbers.

Such that you can depend on either one of those version numbers. Depending on which interface from the package you are depending on.

---

A workaround would be to publish
a) flutter_webview containing a Flutter widget, and,
b) flutter_webview_platform_interface to be used by packages implementing the platform interface on a specific platform.

This allows the Flutter widget in the flutter_webview package to be updated and broken without requiring all the platform implementations to be updated.

The downside of this approach is that:
i) it splits the package in two, and publishing is then a two step process,
ii) it pollutes the search results on pub.dev with unnecessary results as (b) would be uninteresting to most users.

I propose in https://github.com/dart-lang/pub/issues/2184 that packages starting with an underscore are hidden on pub.dev, meaning they don't show up in search results unless there is an exact name match. This would alleviate (ii), leaving (i) which could be mitigated with custom tooling similar to mono_repo.

---

Understanding what symbolic packages is I personally see that the workaround above isn't ideal. However, keeping a dependency to be a a name and a number is rather attractive, as it's very easy to understand. Furthermore, symbolic dependencies are likely to be hard to understand when reading the pubspec.yaml (depending on the syntax of course).

[amirh]

Just to note that (i) is not just about publishing pains, I believe there will be an overhead for developing a single codebase across 2 packages, things like IDE support, and moving/refactoring/sharing code under lib/src are going to be harder.

[jakemac53]

In general it is true that managing more packages causes some non-zero amount of extra work. Ultimately though I think it leads to a better end result for the user because the pub validation checks just work as expected, its more discoverable/obvious what they should be doing, and in general its just a simpler and already existing model.

[amirh]

Ultimately though I think it leads to a better end result for the user because the pub validation checks just work as expected, its more discoverable/obvious what they should be doing,

Would you mind elaborating on this a bit?
Which validation checks won't work as expected? and what gets less discoverable/obvious for end users?

[jakemac53]

Which validation checks won't work as expected?

Specifically today the check that you depend on the packages you directly use (import/export). By having separate packages this just works, if the imports all look like they are coming from the "primary" package then it won't know to tell you to use the "symbolic" package dependency based on what features you are using.

and what gets less discoverable/obvious for end users?

Similar to the previous answer - I think there is a discoverability problem with these symbolic deps.

How do you enforce that users use them when they are expected to?

The concrete real world implication is some user would depend on only the main package, but use some feature that is supposed to require a version on the "symbolic" package. However they won't add it (because how would they know to and pub wouldn't tell them to like with other packages), and then they will get broken by a "non-breaking" release in the main package.

Essentially, i think it is likely this feature would lead to bad practices by package owners, some of which _will_ abuse this feature very egregiously (create a symbolic package for every single api they expose etc), and then this will end up causing a lot of pain when people get broken even though they didn't really do anything wrong.

[isoos]

This currently makes sense as most Flutter breaking changes aren't breaking plugins. Most plugins only care about Flutter's services layer and the embedder's plugin API where breaking changes are less common.

Is the problem the non-semantic versioning of Flutter (or its internal APIs)?

Give a single package two (or more) version numbers. Such that you can depend on either one of those version numbers. Depending on which interface from the package you are depending on.

On a cursory look, this seems to become a source of potential problems and confusion.

As a hypothesis test: if we are allowing more than one version, why limit it as two, why not three, four, or any number of arbitrary versions? If it is arbitrary, we may need to name them, and at that point, the version identifications are hard to distinguish from the package names we already have. I think we should not introduce a second, very similar concept to the already existing one, and let's just use packages (or SDKs) and their constraints.

[blasten]

I'd like to raise this issue.

One common problem I see is that a combination of pub packages can result in build failures because one of the platform dependency resulted in conflict. The error messages can be hard to trace back to a pub package since they are generated by platform-specific build systems that most Flutter/Dart developers are unfamiliar with.

This issue could be prevented if pub had visibility into the platform dependencies. Perhaps, pub could provide better UX affordances and help app developers detect the problems earlier on.
When publishing a new pub package, we could add a phase to the Flutter tool that automatically scans the platform dependency graph and updates the constraints in pubspec.yaml.

I bring this up is because I'm working on the migration to AndroidX. AndroidX uses semver and it will likely increase the number of build failures if one popular dependency is updated.

cc @mit-mit @jonasfj @timsneath