Gallery-dl: pkg_resources.ContextualVersionConflict: urllib3, requests

[Lin-Buo-Ren]

The edge channel snap crashes when launching with the following message:

$ gallery-dl
Traceback (most recent call last):
  File "/snap/gallery-dl/x2/lib/python3.6/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/snap/gallery-dl/x2/lib/python3.6/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/snap/gallery-dl/x2/lib/python3.6/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (urllib3 1.25.1 (/snap/gallery-dl/x2/lib/python3.6/site-packages), Requirement.parse('urllib3<1.25,>=1.21.1'), {'requests'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/snap/gallery-dl/x2/bin/gallery-dl", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/snap/gallery-dl/x2/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3241, in <module>
    @_call_aside
  File "/snap/gallery-dl/x2/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3225, in _call_aside
    f(*args, **kwargs)
  File "/snap/gallery-dl/x2/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3254, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/snap/gallery-dl/x2/lib/python3.6/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/snap/gallery-dl/x2/lib/python3.6/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/snap/gallery-dl/x2/lib/python3.6/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (urllib3 1.25.1 (/snap/gallery-dl/x2/lib/python3.6/site-packages), Requirement.parse('urllib3<1.25,>=1.21.1'), {'requests'})

I suspect the c35217e commit causes this error, can you check it out?
I'll implement a CI test to catch this kind of runtime error more promptly.

[Hrxn]

I think it's because the latest urllib3 version 1.25.1 (as indicated in your log) is not working with requests at the moment. They haven't updated it yet. But an attempt is, uh, underway.

https://github.com/kennethreitz/requests/issues/5067
https://github.com/kennethreitz/requests/pull/5063

[mikf]

Setting a maximum version for urllib3 (and requests) fixed it. Would be nice if pip could respect gallery-dl's and request's version requirements at the same time.

Also: why does a Python snap do version checks at run-time, but not during build-time? (rhetorical question)

[iamleot]

Hello Mike,

Mike Fährmann writes:

[...]
Setting a maximum version for urllib3 (and requests) fixed it. Would be nice if pip could respect gallery-dl's and request's version requirements at the same time.
[...]

Can you please elaborate the rationale behind that? Why does urllib3
1.24.[12] fails to download on Windows?

AFAICT gallery-dl 1.8.2 works fine with urllib 1.24.2 and in my
experience most of the times setting an upper version is harmful
for packaging.

Thank you!

[mikf]

urllib3 1.24.1 & 1.24.2 cause some issues with artstation (#227) and when bypassing some Cloudflare challenge pages, but with 1.24 and the new 1.25.x it works ... for whatever reason. I already looked at the diff between 1.24 and 1.24.1 and there is really nothing that should cause these issues, but they happen nonetheless.

[mike tmp ]$ python -m pip install --user urllib3==1.24.1
...
Successfully installed urllib3-1.24.1
[mike tmp ]$ gallery-dl https://www.artstation.com/nshade
[artstation][error] HTTP request failed:  403: Forbidden for url: https://www.artstation.com/users/nshade/quick.json

[mike tmp ]$ python -m pip install --user urllib3==1.24
...
Successfully installed urllib3-1.24
[mike tmp ]$ gallery-dl https://www.artstation.com/nshade
/tmp/artstation/nshade/artstation_3365702_11700926_Lightforged Draenei.jpg
^C
KeyboardInterrupt

So I added some version specification to setup,py and requirements.txt to exclude those faulty versions (c35217e9a301e215dbe27d24872fad7fb76ab153), but that causes pip to ignore the version specification from requests ("urllib3 < 1.25"), and this in turn installs and uses urllib3 1.25.x when building the gallery-dl snap, which brings us to this here issue.

I'm going to wait until requests supports the new urllib3 version and remove those version specs before releasing the next version of gallery-dl , but until then - to keep the Snap from breaking - I'll keep it this way.

[mikf]

I've managed to figure out why urllib3 1.24.1 and 1.24.2 (as well as older versions) caused some issues and built a workaround for it (https://github.com/mikf/gallery-dl/commit/35f343206c59c9556ff801f0aad443387f7987b6).

Version requirements for urllib3 have been removed and incompatibilities between requests and urllib3 in Snaps should no longer happen.

[iamleot]

Hello Mike,

Mike Fährmann writes:

I've managed to figure out why urllib3 1.24.1 and 1.24.2 (as well as older versions) caused some issues and built a workaround for it (https://github.com/mikf/gallery-dl/commit/35f343206c59c9556ff801f0aad443387f7987b6).

Version requirements for urllib3 have been removed and incompatibilities between requests and urllib3 in Snaps should no longer happen.

Wow! Great catch!

Thank you very much!