Poetry: Update to major versions of packages

[hemanta212]

• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] I have searched the documentation and believe that my question is not covered.

Question

When you first install a package, a caret requirement is used by default. I.e if when installing poetry finds that latest version of package A is 1.2.3. In pyproject.toml file '^1.2.3' will be used.

In the long run when version of package A will reach to 1.3.0 poetry won't update it since it is against the caret requirements. So am I supposed to know the (major and minor) releases of each packages and manually edit the pyproject.toml file to something like '^1.3.0' (eg for package A) ?

[David-OConnor]

That's correct.

[hemanta212]

so if I have 20/30 dependencies then? How can I manually follow release a new version?

Can't poetry scan and tell you what the lastest version of package is vs what I have?

[sdispater]

Running poetry show -l will display the current and the latest version available of the packages required by your project.

If you use the latest 1.0.0b1 release, you can use poetry add pendulum@latest to automatically update your package to the latest version. See https://github.com/sdispater/poetry/pull/1221 for more information about the improvements to the add command.

[jabbalaci]

I'm using Poetry version 1.0.10. In pyproject.toml I have this line:

mypy = "^0.761"

When I want to upgrade it with the command poetry add mypy@latest, I get this error:

[SolverProblemError]
Because ... depends on both mypy (^0.790) and mypy (^0.761), version solving failed.

[hemanta212]

@jabbalaci you should try upgrading to current latest version of poetry and if the issue still persists, create new issue with detailed logs.

[jabbalaci]

@hemanta212 Thanks! I installed it from my OS's repo but it was an old version. Thanks, I will try with the latest version.

[finswimmer]

Hello @jabbalaci,

I guess you have mypy defined in the [tool.poetry.dev-dependencies] section. A poetry add mypy@latest will add mypy to the [tool.poetry.dependencies]. Having the same dependency in both section is supported by poetry.

If I'm right with my assumption above, you would end up with two dependency definitions for mypy. One with the version constraint ^0.761 and one with ^0.790. According to the docs this would translate to >=0.761,<0.762 and >=0.790,<0.791, which conflicts.

To summarize: If you have mypy defined in the development section, you want poetry add --dev mypy@latest.

fin swimmer

[jabbalaci]

@finswimmer : Thanks, you are absolutely right. mypy was originally in the dev. section. I didn't add the --dev option.

Well, in this case a more informative error message would be better. If the error message mentioned that the normal dependencies conflict the dev. dependencies, then it would be easy to figure out what's wrong.

[JosXa]

so if I have 20/30 dependencies then?

This thread did not address the case where you have tons of outdated dependencies or would like to run a CI pipeline that upgrades your packages and executes your test suite on the outcome (and then files a PR, like dependabot does for instance). I would really appreciate something like a poetry upgrade --all command that will handle this use-case automatically so that I can upgrade locally and smoke test if everything still works alright.

This would boil down to a builtin workflow that...

1. Runs poetry show -l -o:

1. For each outdated package executes poetry add {name}:latest:

However, I can imagine the following problems and cases which make such a script hard to write and generalize yourself:

• It seems that poetry show also gives you sub-dependencies of the packages you have added. Typically you don't want to mess with or install those, instead you want the diff for only the packages you've specified in pyproject.toml, but there doesn't seem to be an option for that in the show command.
• This does not preserve special cases such as packages added via git repository or from file, aswell as dev dependencies and extras. With the naive approach from the 2nd screenshot above, I am running into the same problem as @jabbalaci:

Updating dependencies
Resolving dependencies... (14.4s)

[SolverProblemError]
Because josxabot depends on both mypy (^0.790) and mypy (^0.770), version solving failed.

Thanks, you are absolutely right. mypy was originally in the dev. section. I didn't add the --dev option.

• That means that the add package:latest approach cannot be run in a simple command, because calculating all permutation groups of e.g. --dev and --extra would be a nightmare.

Can you ease my mind and I should just write this myself, or would you agree that these use cases are valid (or does something like this maybe already exist)? Theoretically, if you don't feel like maintaining this yourselves, maybe this could also be released as an extra package like it is in npm:

https://nodejs.dev/learn/update-all-the-nodejs-dependencies-to-their-latest-version

[hemanta212]

if you don't feel like maintaining this yourselves, maybe this could also be released as an extra package like it is in npm

Yes this should be one usecase for upcoming poetry's plugin api.

[sinoroc]

@JosXa In such cases I tend to reach for the hammer and simply recreate the virtual environment from scratch. It is not ideal, but maybe that would be good enough for you (or others with similar use cases). A command such as poetry upgrade --all sounds good though (but there might be too many edge cases, you pointed some yourself).
@hemanta212 Yes, that sounds like a good candidate for a plugin to me.

[JosXa]

@sinoroc That's a comparable amount of effort though since you still have to be careful with extras, dev, etc., am I right?
I'm also wondering this: If I add a star (*) dependency for every version in toml, is it possible to let poetry resolve and pin the latest versions that make a valid dependency graph somehow? Write them back in pyproject.toml?

[JosXa]

@hemanta212

Yes this should be one usecase for upcoming poetry's plugin api.

Is there maybe a collection issue, roadmap or ideas board where issues can be linked to and be prettied up for the community to pick projects from?

[sinoroc]

Yes this should be one usecase for upcoming poetry's plugin api.

Is there maybe a collection issue, roadmap or ideas board where issues can be linked to and be prettied up for the community to pick projects from?

You mean some kind of "_wishlist for poetry plugins_"? That seems like a rather good idea.

That's a comparable amount of effort though since you still have to be careful with extras, dev, etc., am I right?

No. Unless your pyproject.toml is missing some information, it should be straightforward. You might need to remove the lockfile, though.

I'm also wondering this: If I add a star (*) dependency for every version in toml, is it possible to let poetry resolve and pin the latest versions that make a valid dependency graph somehow?

I think, yes. I am not sure why you would want that, but fair enough, to each their own. Just give it a try. And I think I might need to come back on what I said earlier: no need for a poetry upgrade --all command after all.

Write them back in pyproject.toml?

This is not automated as far as I know, you would have to do it by hand.

[hemanta212]

Is there maybe a collection issue, roadmap or ideas board where issues can be linked to and be prettied up for the community to pick projects from?

Yes there is, you have to search issues with "plugins" label. There aren't much though, as of now. People have been asking for interconvertability between pipfiles requirements and other formats.
I can see poetry being extended to cover remaining development workflow like running custom scripts, cookiecutter for projects, tests runner etc. Features provided by dephell are good candidates too.

I'm also wondering this: If I add a star (*) dependency for every version in toml, is it possible to let poetry resolve and pin the latest versions that make a valid dependency graph somehow?

I dont think such workaround exists but for a plugin it should be pretty easy to pipe the new versions to poetry since poetry show -l already does most of the work

[sinoroc]

For easy clicking: this is the list of tickets labelled "_plugins_".

@JosXa

If I add a star (*) dependency for every version in toml, is it possible to let poetry resolve and pin the latest versions that make a valid dependency graph somehow? Write them back in pyproject.toml?

As I wrote earlier, I believe this would work. And the resulting pinned version numbers would be written in the lockfile. What would be the point of writing the pinned versions in pyproject.toml (if that is really what you're asking for)?

[JosXa]

Okay let me lay out that use-case once more:

There is a project you haven't touched in a year with 80 dependent PyPI packages. You decide it's time to make some changes and install potentially security-critical updates. You want to upgrade all packages to their latest versions while maintaining a valid dependency graph but also leave the versions halfways pinned in pyproject.toml and running some smoke tests. This way the next person checking out the project can run poetry update without upgrading major versions and having to go through the whole testing process again.

Plus the automated approach from above. Not sure if it makes sense or if leaving * versions in .toml is actually not as bad as it feels to be

[finswimmer]

https://github.com/python-poetry/poetry/issues/461 ?

[timkofu]

Are there plans to add a switch to upgrade to upgrade to the very latest release versions? For example:

poetry update --latest

[sinoroc]

Are there plans to add a switch to upgrade to upgrade to the very latest release versions?

@timkofu I am not sure I understand how it is different than everything that has been asked and answered in this discussion and in https://github.com/python-poetry/poetry/issues/461. Would you mind expanding on what you are asking for has not been already covered?

[timkofu]

@sinoroc An official confirmation that this is now a planned or currently being worked on feature.

@finswimmer only confirmed the request is valid back in June.

[sinoroc]

@timkofu Ah, ok, understood. On the other ticket, you can see there was some movement 3 days ago.

finswimmer added this to To do in Features via automation 3 days ago

So it is here in the project roadmap now.