Pub: Can we accommodate more use-cases for dependency_overrides

[jonasfj]

Use-cases for dependency_overrides:

A) App development, when you know better than the version constraint solver and you wish to override the dependencies of your application it makes sense to add dependency_overrides in pubspec.yaml.

B) mono repository setup, when developing in a mono repository it makes sense to override dependencies for the packages in the same repository, such that you are testing with and against these. These overrides also makes sense in CI, in fact they make sense in any context where dev_dependencies is installed.

C) poly repository setup, when working on package A that is used by package B in two different repositories, it makes sense to temporary override the dependency on A in B and run tests for B to ensure that these work, before submitting a patch for A. This is often the case when fixing a bug in package A. Overrides are not useful in CI, and overrides are generally specific to the local development setup.

---

Are we missing any use-cases? @sigurdm, @lemz90, @StorminGorman,

[jonasfj]

Options currently proposed:

1) Add an pubspec.override.yaml file (#2150)
2) Add an --ignore-overrides option (#2151)
3) Add an --overrides=my-overrides.yaml option (https://github.com/dart-lang/pub/pull/2151#issuecomment-506667067)
4) Add a dev_dependency_overrides to pubspec.yaml which is only used when installing locally.

---

It seems to me that dependency_overrides as we already have solves (A) fairly well.
And perhaps using dependency_overrides in the pubspec.yaml to solve (B) and (C) is a misappropriation.

This suggests that maybe (2) isn't a good option. Similarly, (4) is not great for (C).

The me the downside with (1) is that introduces a new configuration file that we will have to support going forward (which feels a bit magic). Removing it later would probably be controversial, even if there is just a few people using it -- moreover it would be hard to know if it has few or many users.

---

While (3) isn't easy to use, it could be used by other tools.. it could also be a --pubspec=my-pubspec.yaml option. An option to simply specify a different file, instead of pubspec.yaml.

In this scenario, the mono_repo pub get tool could read the pubspec.yaml file, add overrides matching the other packages in the repository, and write the file to /tmp/<random>-pubspec.yaml and run pub get --pubspec=/tmp/<random>-pubspec.yaml.

Then the mono_repo tool could have an --omit-overrides options to allow testing without these overrides. This would allow testing and developing with dependency-overrides for the other packages in the same repository, without modifying the pubspec.yaml that is checked into the repository, so long as the call to pub get goes through mono_repo pub get.

Similarly, we could make a poly_repo tool that supports reading a local pubspec.override.yaml file, write a temporary /tmp/<random>-pubspec.yaml file, and call pub get --pubspec=/tmp/<random>-pubspec.yaml. Maybe this command could be called pub_overrides or some other better name than poly_repo, hehe :)

These tools can be developed independently, distributed on pub, installed with pub global activate. By versioning them breaking changes would be less painful. Most importantly this would avoid polluting pub commands with a lot of options that most developers won't need.
I would imagine that the documentation for dependency_overrides could reference tools for solving (B) and (C).

This is not fully thought through, but there could supposedly be other use-cases where a tool calling pub get would want to rewrite the pubspec.yaml first, this would give tool authors such an option.

[StorminGorman]

The me the downside with (1) is that introduces a new configuration file that we will have to support going forward (which feels a bit magic). Removing it later would probably be controversial, even if there is just a few people using it -- moreover it would be hard to know if it has few or many users.

Many projects I've worked on have had a local.properties or a similar config file that is used for this purpose. Usually its checked in as local.properties.example and the developer edits it if they want to override anything or set environments specific to their local build setup. One of the most famous and used examples is Gradle and Android Studio https://developer.android.com/studio/build .. Gradle automatically uses local.properties _if present_ ..

These tools [mono_repo, poly_repo] can be developed independently, distributed on pub, installed with pub global activate. By versioning them breaking changes would be less painful. Most importantly this would avoid polluting pub commands with a lot of options that most developers won't need.
I would imagine that the documentation for dependency_overrides could reference tools for solving (B) and (C).

I do not support developing separate tools to facilitate these functions. One of the major reasons is alluded to by @sigurdm https://github.com/dart-lang/pub/pull/2151#issuecomment-506667067

pub get _should_ do everything "magically" as you say ... it makes integration with CI and IDE's less brittle and keeps the dependency resolution process simple for the developer who may be less concerned with how the development environment is setup.

Upon further thinking about #2150, I think that is a good way to go for both mono and poly repos. Having a simple pubspec.override.yaml file solves A, B & C in my mind because the folks who want consistent overriding throughout the devops process can just check in that file into version control (A & B) where (C) folks can just ignore it. You could also add your --ignore-overrides flag to ignore the checked in pubspec.override.yaml if desired.

I totally understand and support the hesitance of adding a feature which may or may not be wildly used and then having to support that minority feature for years to come. That being said, we need to find a balance between that and adapting the functionality of the base tool to be most widely adoptable.

[StorminGorman]

Options currently proposed:

1. Add an pubspec.override.yaml file (#2150)
2. Add an --ignore-overrides option (#2151)
3. Add an --overrides=my-overrides.yaml option (#2151 (comment))
4. Add a dev_dependency_overrides to pubspec.yaml which is only used when installing locally.

I would also add a another to this list.

1. change dev_overrides to work more similarly to npm's optionalDependencies

In this case, it uses the dependencies if they are there .. but if they cannot be found then it keeps on trucking using the normal dependencies. So in other words its more of a override-if-present option...

[lemz90]

I like option 1 (pubspec.override.yaml) personally for the main mechanism for overriding.

However, I think adding some of the other aspects on top of the override file could be useful. I think having pub get by default use something like pubspec.override.yaml would be good. From there, we need to decide if we want to have flags to ignore the override file (--ignore-overrides) or change the pubspec file (--pubspec,-p)

[sigurdm]

A circumvention for local overrides not checked in might be to use a filter like:
https://stackoverflow.com/questions/16244969/how-to-tell-git-to-ignore-individual-lines-i-e-gitignore-for-specific-lines-of

[StorminGorman]

Thats a cool trick ... but I'm afraid it may end up being brittle and/or resulting in a shotgun surgery like situation (every time you add/remove your pubspec overrides you have to edit .gitattributes). Since .gitattributes is a "hidden" file it makes it even harder to figure out what is going on. (even more "magical" than the proposed solutions) . Also, and probably more importantly, it's not available for SVN.

[lts1610]

@jonasfj : Which option was chosen? And it was release?

[jonasfj]

We haven't adopted any of these yet.

And as one can worked around it with comments it's probably less urgent.

I suspect maybe we're trying to accommodate too many use cases, in practice dependency_overrides should rarely be necessary.

[StorminGorman]

the "comment" work around is a constant headache for our team. People accidentally checking in pubspec.yaml's with their local dependency overridden ... Always having to go in and comment out your override when you make a change to the pubspec before you check in ... its annoying. To say that using it is "rarely necessary" I think is a bit short sighted tbh... sure if you have a mono repo you should rarely use it but for those who are sharing external dependencies and working on them all at the same time its annoying and I don't think thats a "rare" situation... but maybe I'm wrong?

[lts1610]

@StorminGorman : I have same issue, and i think op1 and op2 are enough to resolve it

[lemz90]

I suspect maybe we're trying to accommodate too many use cases, in practice dependency_overrides should rarely be necessary.

My team uses dependency_overrides every day in a 'polyrepo' setup (which I would assume is a very common configuration - albeit not trendy). We have a pure Dart business library package that is depended on by our Flutter and AngularDart front ends. Most of our developers are working in the business package and a client package at the same time, therefore overriding the business logic package to a local path.

I'm honestly not sure I understand the 'monorepo' use case of overriding. If there is something I'm missing about 'monorepo' that requires the default (and only) behavior of the command line to override (other than pre-existing projects using, in my opinion, a subpar override mechanism), then please enlighten.

My main question is: if the dependency_overrides make sense in CI for 'monorepo' then how is it an override?
Why would you ever want overrides checked in?
If you do want the overrides checked in, then just check in my proposed pubspec.override.yaml (#2150) :P

---

Then the mono_repo tool could have an --omit-overrides options to allow testing without these overrides. This would allow testing and developing with dependency-overrides for the other packages in the same repository, without modifying the pubspec.yaml that is checked into the repository, so long as the call to pub get goes through mono_repo pub get.

Similarly, we could make a poly_repo tool that supports reading a local pubspec.override.yaml file, write a temporary /tmp/<random>-pubspec.yaml file, and call pub get --pubspec=/tmp/<random>-pubspec.yaml. Maybe this command could be called pub_overrides or some other better name than poly_repo, hehe :)

Do you really think this would play more nicely with IDEs, CI, other tools, etc. than pubspec.override.yaml? What is the advantage to this approach?

---

My vote still goes to the pubspec.override.yaml (#2150). However, I would settle for the --ignore-overrides (#2151) flag considering many projects already use the dependency_overrides key. However, most of our CI scripts would --ignore-overrides, and we would still have the issue of developers accidentally checking in their local path overrides.

[tiholic]

Waiting for this, to have better CI and dev environment management.

It is best to have an override yaml file so we can have flexibility to override any other configuration in future!
Also, the IDE plugins should be updated to auto-detect and install from appropriate yaml file _(ex: "pub get" action in AndroidStudio)_.

[TimWhiting]

I use the polyrepo setup, with git dependencies as default, I would like to override them to path dependencies for local development, but that makes CI a real pain if I have to clone all the repositories into the right relative paths. I personally vote for the --ignore-overrides so that we can have the path overrides for local development, but use the git dependencies for CI. I don't see why this is a problem for the other use cases, and it seems like the most straightforward.

[blaugold]

It might be worth looking at cargo's approach for an established example of how to solve for these use cases. cargo is the package manager for rust. In a cargo package, dependencies can have multiple locations:

It is possible to specify both a registry version and a git or path location. The git or path dependency will be used locally (in which case the version is checked against the local copy), and when published to a registry like crates.io, it will use the registry version. Other combinations are not allowed.