Requests: Looks like dependency mismatch for chardet lib.

[synchr-darnold]

Version: requests-2.17.3

Exception: Requests dependency 'chardet' must be version >= 3.0.2, < 3.1.0

cd /usr/local/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/_vendor/requests/packages/
[root@ip-10-0-11-136 packages]# python
Python 2.7.12 (default, Sep  1 2016, 22:14:00) 
[GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import chardet
>>> import os
>>> print os.path.abspath(chardet.__file__)
/usr/local/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/_vendor/requests/packages/chardet/__init__.pyc
>>> chardet.__version__
'2.3.0'

[kennethreitz42]

looks like you need to update your chardet to the versions specified :)

[sigmavirus24]

Exactly what @kennethreitz. pip install -U 'chardet >= 3.0.2, < 3.1.0' will fix this up for you.

[synchr-darnold]

The chardet that is included with requests is broken.

[Lukasa]

No chardet is included with the new versions of Requests: they depend on it via pip.

[synchr-darnold]

If requests depends on it, and running pip install requests is broken because the version is not pinned for requests dependencies, shouldn't the version be pinned? Having to fix the installation of requests seems to mean that it's broken.

[Lukasa]

The version is pinned. This is a known problem due to pip's lack of a dependency solver, which we're hoping will get resolved this summer when pip gets a proper dependency solver.

[synchr-darnold]

The automation that I've been using to deploy services using Ansible was working last Friday, ceased to work on Monday. Are we sure that nothing else is changed? Seems like the pip problem is a bit older than last Friday. This was on a new Amazon Linux ec2 instance.

There were a few releases that went out over this last weekend.

[Lukasa]

No, to be clear, Requests stopped vendoring chardet on Friday and started requiring it as an explicit dependency. That is what changed. See the 2.16.0 release notes.

[synchr-darnold]

Ah, didn't quite catch that... Unvendor ALL the things! so, basically just wait until pip is fixed. Thanks for your patience @Lukasa

[Lukasa]

You shouldn't need to wait: you can just make sure you install Requests with all its dependencies by doing a pip install -U requests, which should bring in anything new it needs.

[joshma]

(Let me know if this should be a new issue - thought it was related enough.) @Lukasa @kennethreitz based on https://github.com/requests/requests/issues/4069#issuecomment-304432367, it seems like the rationale behind unvendoring isn't super clear. Is it just to conform better with how Python packages tend to operate?

Is there any chance we can consider reverting back to vendoring?

In our specific case, chardet>=3.0.2 (which requests now requires) is causing problems with our usage of UnicodeDammit (https://www.crummy.com/software/BeautifulSoup/bs4/doc/#unicode-dammit), which relies on chardet. (This is a whole other nasty rabbit hole, but basically it seems like chardet 3 incorrectly detects some UTF-8 strings as ISO-8859-1, which chardet 2 was correct on.)

My current workaround is to downgrade to requests==2.15.1, before the great unvendoring, so we can keep chardet 2.

[Lukasa]

I'm afraid it's not very likely: the reality is that the great unvendoring helped many more users than it inconvenienced. I appreciate that it has caused problems here, but the better solution may be to see if we can widen the pin to support the older chardet version.

[joshma]

@Lukasa widening would work great, if possible!

(out of curiosity, how did unvendoring help? was it to lower package sizes?)

[kennethreitz42]

@joshma it was a decision I made, for a multitude of reasons, including security concerns.

[Lukasa]

Heh, there are lots of issues in the back catalogue caused by vendoring. The core issue is that for various reasons the vendored packages were unpopular and removed, which led to subtle difficult bugs depending on install source.

[kennethreitz42]

e.g. the less code we ship, the better. especially certifi.

[joshma]

Cool, thanks all for the context. Is there anything I can do to help determine if we can widen the chardet pin? :crossed_fingers:

[sigmavirus24]

I don't think we can widen the chardet pin. For most every other user, the newer version is better. I know Dan is working on retraining the models for some of the newer bugs, but chardet 3.x should be faster than 2.x and more accurate on most any other encoding.