botocore 1.14.4 blocks users from updating docutils to 0.16

[moseb]

Hi!

When botocore is used in a virtualenv next to other packages or in context of a whole GNU/Linux distribution, this line in setup.py
https://github.com/boto/botocore/blob/70c7767374c42bd86a896391dff55fc25ebfac27/setup.py#L27

blocks users from updating to docutils 0.16 which is not ideal. Can the upper boundary be dropped or be turned into semver-like <1 at least to be more friendly to other packages and user freedom?

Thanks and best, Sebastian

PS: I have seen #900 and #933 (but I don't consider this the right direction while docutils is imported from public botocore API).

[swetashre]

Thank you for your post. Marking this as feature request.

[prometheanfire]

seems like we have to go through this issue every few weeks

[neg3ntropy]

This is not an enhancement, It's a bug.

[roll]

Hi, our stack users run into:

ERROR: botocore 1.15.36 has requirement docutils<0.16,>=0.10, but you'll have docutils 0.16 which is incompatible.

[yan12125]

@swetashre: Could you have a look into https://github.com/boto/botocore/pull/2011? It's an updated fix for this issue.

[edrozenberg]

Same issue here. Thanks to the very useful utility pipdeptree for making it easy to dig up issues (https://pypi.org/project/pipdeptree/)

Warning!!! Possibly conflicting dependencies found:
* botocore==1.16.20
 - docutils [required: >=0.10,<0.16, installed: 0.16]

This version of docutils comes standard with my OS (slackware64-current) so would much prefer not having to downgrade this package and possibly breaking other things just to satisfy this dependency. python-docutils-0.16-x86_64-3

[katzdm]

Would be nice to have this constraint removed; docutils==0.16.0 was released nearly six months ago.

[yan12125]

@edrozenberg: since you are using system packages from Slackware, a workaround might be adding the patch at #2011 to the botocore package on Slackware, like what I did for Arch Linux [1].

[1] https://git.archlinux.org/svntogit/community.git/commit/trunk?h=packages/python-botocore&id=4f4eecd0bf7719bdc77856c1f40e3466e94ad8db

[edrozenberg]

@yan12125 Thanks! That would certainly do it and I'll consider applying that sometime. For now I'll just ignore the issue since I don't care about building boto docs.

[prometheanfire]

not sure that helps those that want to install from pip

[yan12125]

I rebased my branch on top of botocore 1.6.23. Before my branch is merged, you can use the following command to install my branch:

pip install git+https://github.com/yan12125/botocore@issue-1942-allow-use-with-docutils-0-16

[prometheanfire]

that does help some, but others have to use pip (openstack uses only released packages for instance).

[arnaudsjs]

It would be nice to have this dependency removed. This issue is going to break a lot of projects in October when PIP starts using its new dependency resolver in PIP version 20.3 (https://pip.pypa.io/en/stable/user_guide/#changes-to-the-pip-dependency-resolver-in-20-2-2020). This version of PIP will be more strict with respect to pinned dependencies.

[erdseb]

@swetashre can you label this a bug rather than a feature request? Playing nice with other packages in the environment and not blocking updates does not seem a feature request to me. Also, everyone who uses Dependabot needs to blacklist docutils. Thank you.

[nateprewitt]

Hey @erdseb, as we said in #2121 this is pinned because it's a requirement for the project. 0.16 has breaking changes to the docutils API and until that's resolved we cannot upgrade. We currently have work underway to get everyone unblocked.

For reference, could anyone in this thread share some examples of libraries that are requiring 0.16 explicitly?

[erdseb]

@nateprewitt my sole point above is that it's a bug, not a feature request.

[nateprewitt]

@erdseb, sure, we definitely agree the issue has been problematic for users. To be clear though, libraries should be able to select appropriate dependencies and you'll find this is done in many major Python libraries. Not supporting the latest release of a dependency isn't a bug in a library, it's a feature support issue, which is why this has the chosen categorization.

In regards to pip's dependency resolver, this issue should actually become less problematic after that's been released due to a more intelligent dependency resolution scheme.

[erdseb]

@erdseb, sure, we definitely agree the issue has been problematic for users. To be clear though, libraries should be able to select appropriate dependencies and you'll find this is done in many major Python libraries. Not supporting the latest release of a dependency isn't a bug in a library, it's a feature support issue, which is why this has the chosen categorization.

@nateprewitt you can pick a dependency but if your code depended e.g on Windows 95 and would not work with Windows 10, in 2020 that's a bug in my book. I'm exaggerating to make a point, software does not live in a vacuum. If it's a feature rather than a bug in your book, we can agree to disagree.

In regards to pip's dependency resolver, this issue should actually become less problematic after that's been released due to a more intelligent dependency resolution scheme.

Could you elaborate how the new resolver will be able to improve the situation with this very problem?

[nateprewitt]

Could you elaborate how the new resolver will be able to improve the situation with this very problem?

0.16 is a seldomly used version of docutils, specifically because it's both new and has an inappropriately specified breakage. Current adoption 0.16 is at ~12% compared to 0.15.2 which accounts for 82% of all docutils downloads [source].

Most libraries support a range of values for a given dependency. The way pip currently works is it treats projects as being in a vacuum. It will install the latest version of a dependency for one project and then move on to install the next projects dependencies, which is why you're hitting these conflicts. With the new resolver, it will evaluate the dependency closure holistically, and realize that 0.15.2 satisfies the requirements of all projects and elect to use that. You can demo this pretty quickly by creating a project that requires botocore and docutils <= 0.17, then installing it with and without the resolver.

The only case this wouldn't work is if a dependency specifies docutils >= 0.16, which I'm unable to find any major projects currently doing this.

[prometheanfire]

I will say, openstack (which includes docutils just about everywhere) is blocked on updating docutils due to botocore (and only botocore).

[yan12125]

Great to see docutils is finally removed as a dependency in #2154. Looking forward to a new release of botocore.

[nateprewitt]

This should now be resolved in botocore 1.18.0.