Botocore: time to update python-dateutil again

[Tecktron]

Looks like it's that time to update again.

ERROR: botocore 1.13.9 has requirement python-dateutil<2.8.1,>=2.1; python_version >= "2.7", but you'll have python-dateutil 2.8.1 which is incompatible.

Why not pin the upper bound to a major version instead of minor patch version?

[jamesls]

Why not pin the upper bound to a major version instead of minor patch version?

This is what we normally have, but this was change we had to make today (https://github.com/boto/botocore/pull/1871) due to a recent dateutil release. This should be a temporary fix until we can fix our installers and revert back to a <3.0.0 version range for dateutils.

This would only be an issue if you're installing something else that is pulling in the latest version of dateutils. If I just use pip install botocore, it pulls in the right version of dateutils:

$ pip install botocore
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting botocore
  Downloading https://files.pythonhosted.org/packages/bd/45/042a6db7d025be8b04bd47528bc8db414de02e2e5802142e9fb45610d620/botocore-1.13.9-py2.py3-none-any.whl (5.3MB)
     |████████████████████████████████| 5.3MB 5.1MB/s
Collecting jmespath<1.0.0,>=0.7.1
  Using cached https://files.pythonhosted.org/packages/83/94/7179c3832a6d45b266ddb2aac329e101367fbdb11f425f13771d27f225bb/jmespath-0.9.4-py2.py3-none-any.whl
Collecting urllib3<1.26,>=1.20; python_version == "2.7"
  Using cached https://files.pythonhosted.org/packages/e0/da/55f51ea951e1b7c63a579c09dd7db825bb730ec1fe9c0180fc77bfb31448/urllib3-1.25.6-py2.py3-none-any.whl
Collecting python-dateutil<2.8.1,>=2.1; python_version >= "2.7"
  Using cached https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl
Collecting docutils<0.16,>=0.10
  Using cached https://files.pythonhosted.org/packages/3a/dc/bf2b15d1fa15a6f7a9e77a61b74ecbbae7258558fcda8ffc9a6638a6b327/docutils-0.15.2-py2-none-any.whl
Collecting six>=1.5
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Installing collected packages: jmespath, urllib3, six, python-dateutil, docutils, botocore
Successfully installed botocore-1.13.9 docutils-0.15.2 jmespath-0.9.4 python-dateutil-2.8.0 six-1.12.0 urllib3-1.25.6

Here's the background on this if you're curious:

There was a changed merge into dateutils a while back that moved constants out from setup.py to setup.cfg (https://github.com/dateutil/dateutil/pull/880).

The consequence of this is that you need a modern version of setuptools for this to work properly. Our bundled installer is using a version of setuptools that doesn't support this and as a result won't install correctly (the linked botocore PR has shows the specific error).

In order to unblock today's release, we set an upper bound of <2.8.1 for the time being until we can get our installers upgraded. I'm looking into this now, but don't have a concrete ETA just yet.
I should know more soon.

[Tecktron]

Thanks! Appreciate the background, I was a bit curious. I'm also a bit curious about your use of py2.7 (lol) given it's EOL in less than 2 months (Jan 2020 is gonna be a good month, py2 & win7 FINALLY!).

I'm concerned about how you could be using such an old version of setuptools. It has handled cfg files since 30.3.0 (released 8 Dec 2016). That's about 3 years old right now is I hope all I need to say about that.

So I guess the solution for now is that one has to manually add python-dateutil==2.8.0 to the top of all their requirement files that use boto3/botocore until either you get your installer up to date and make a new release, or pip releases fixes for dependency loading. In the meantime, we'll all just be here hoping that nothing else being installed is dependent on using dateutil's latest features and/or bug fixes. :persevere:

For those reading, here's the commit showing how laziness has ruined your day: https://github.com/boto/botocore/commit/e87e7a745fd972815b235a9ee685232745aa94f9

[prometheanfire]

watching this, capping is bad, especially on 'core' libraries. Blocking all update on boto til this is fixed :( https://review.opendev.org/692951

[TomAugspurger]

@jamesls were you able to come up with an ETA?

[bwesen]

FWIW, 1.13.14 still has this issue (which I got pulled in from the latest boto3 update):
ERROR: botocore 1.13.14 has requirement python-dateutil<2.8.1,>=2.1; python_version >= "2.7", but you'll have python-dateutil 2.8.1 which is incompatible.

[robvdl]

This is preventing me from installing erpnext now, when installing erpnext I am running into this and it's failing. I tried to install an old version of python-dateutil before installing erpnext but it still runs into this error:

ERROR: botocore 1.13.14 has requirement python-dateutil<2.8.1,>=2.1; python_version >= "2.7", but you'll have python-dateutil 2.8.1 which is incompatible.

Looks like a fix is needed sooner rather than later as people are affected by this.

[lapphan]

Even botocore 1.13.23 has this bug. Please fix it asap.

ERROR: botocore 1.13.23 has requirement python-dateutil<2.8.1,>=2.1; python_version >= "2.7", but you'll have python-dateutil 2.8.1 which is incompatible.

[JasperHG90]

Encountered this issue today when trying to install gensim 3.7.1 in a docker container. Added the following two dependencies to 'requirements.txt' which solved the problem for now

boto3==1.10.6
botocore==1.13.9

[abitrolly]

@jamesls is there an open bug for fixing installers you're speaking about?

[essandess]

+1 This also affects MacPorts and creates dependency hell.

[Vermorxt]

Botocore 1.13.28 still has this issue:

ERROR: botocore 1.13.28 has requirement python-dateutil<2.8.1,>=2.1; python_version >= "2.7", but you'll have python-dateutil 2.8.1 which is incompatible.

[MartinFalatic]

@jamesls, you wrote:

The consequence of this is that you need a modern version of setuptools for this to work properly. Our bundled installer is using a version of setuptools that doesn't support this and as a result won't install correctly (the linked botocore PR has shows the specific error).
In order to unblock today's release, we set an upper bound of <2.8.1 for the time being until we can get our installers upgraded. I'm looking into this now, but don't have a concrete ETA just yet.
I should know more soon.

This is causing more and more problems and conflicts as other packages update to the newer python-dateutil (with the https://github.com/dateutil/dateutil/issues/871 changes).

• What "bundled installer" is affected by this?

• How can one reproduce the problem you're currently working around with this pin?

• Is this related to ongoing support for (soon-to-be-obsolete) Python 2?

[MartinFalatic]

Is this all because OSX's native Python 2.7 (which eventually won't even be included with OSX at all) is failing to run setup? Per the link @Tecktron shared earlier, that seems to be at least part of the problem.

Beyond 2020-01-01, there should be no reason to hold this back for the sake of a distribution that will be EOL and soon thereafter defunct as a default package on OSX.

[abitrolly]

If OSX is an officially supported package, I'd expect it to have at least build job on Travis to propose PRs with fixes.

[MartinFalatic]

As far as I can tell the issue didn't appear in Travis - it appeared somewhere else. My guess is it's OS X, but without more information I can't be sure what broke that led to them pinning this in the first place (let alone what they think would solve it).

[abitrolly]

I am trying to tell that Travis supports OS X and build job for OS X installers where could help pinpoint the problem. If sources for OS X installers is here, of course.

[MartinFalatic]

I understand. I want to hear from the person who pinned this dependency what installers they plan to upgrade to fix this properly - and when.

[jamesls]

I'm working on this today. I'll update this with more details once I get a fix implemented and tested, but I plan to have an update by the end of the day.

[MartinFalatic]

Awesome! This will make a lot of people quite happy. :-)

[jamesls]

EOD update: So I have a fix that I've been experimenting with that I think fixes our issues. I've verified the changes on py26/py27 but there seems to be issues on py3. I'm trying to track down if these are related to my changes or not. Once I can confirm it's not due to my changes I'll go ahead with the PR. Should finish all this testing tomorrow. I realize this is affecting quite a few people so this is my main priority right now.

[MartinFalatic]

Should finish all this testing tomorrow. I realize this is affecting quite a few people so this is my main priority right now.

Any update? (I see the PR above (and the one it's connected to) but it's not clear if these are pending review or test.)