pipenv gets wrongly dependency function32

[fujiaoliu]

Issue description

Seems pipenv install gets an unnecessary dependency function32 which supports python2.7 only.

Expected result

in this case, pipenv should not include function32 as a dependency since it is not needed(pip doesn't need it)

Actual result

pipenv uses it as a secondary dependency and it will fail because function32 doesn't work with python3

Steps to replicate

pipenv install tensorflow==2.0.0-rc0 --three --pre --verbose

---

[fgervais]

Yes I'm getting this exact issue also.

I'm on:

pipenv run python --version
Python 3.6.8

pipenv --version
pipenv, version 2018.11.15.dev0
# I installed pipenv as editable to debug

For some reason, pipenv considers the python 2.7 version of tensorflow a candidate and not only that but the "best_candidate"

DEBUG:pipenv.patched.notpip._internal.index:best_candidate
DEBUG:pipenv.patched.notpip._internal.index:<InstallationCandidate('tensorflow', <Version('2.0.0rc0')>, <Link https://files.pythonhosted.org/packages/41/59/7260f522aade1f3f692a456f29952ee6b8ec47df4266a18f8b1e37ae4497/tensorflow-2.0.0rc0-cp27-cp27m-macosx_10_11_x86_64.whl#sha256=566394792b8156b000f45227b29335a0d493a95b40ed8f3fba21a712cfa5adc7 (from https://pypi.org/simple/tensorflow/)>)>
DEBUG:pipenv.patched.notpip._internal.index:Using version 2.0.0rc0 (newest of versions: 2.0.0rc0)

From here is gets the 2.7 dependencies, tries to use functools32 and fails.

I'm not familiar enough with python packages installation to know who is wrong at this point.

See this is "applicable_candidates":

DEBUG:pipenv.patched.notpip._internal.index:[<InstallationCandidate('tensorflow', <Version('2.0.0rc0')>, <Link https://files.pythonhosted.org/packages/41/59/7260f522aade1f3f692a456f29952ee6b8ec47df4266a18f8b1e37ae4497/tensorflow-2.0.0rc0-cp27-cp27m-macosx_10_11_x86_64.whl#sha256=566394792b8156b000f45227b29335a0d493a95b40ed8f3fba21a712cfa5adc7 (from https://pypi.org/simple/tensorflow/)>)>, <InstallationCandidate('tensorflow', <Version('2.0.0rc0')>, <Link https://files.pythonhosted.org/packages/37/91/bac5970057b77aea5a97343cc61e480e2e9930b652c0870cc51bf4d4a4c4/tensorflow-2.0.0rc0-cp27-cp27mu-manylinux2010_x86_64.whl#sha256=e563fef77cd4ae9d0b4e8ac9709467e2225f4e8ad59bc2bc7b7d3b4e2348bb55 (from https://pypi.org/simple/tensorflow/)>)>, <InstallationCandidate('tensorflow', <Version('2.0.0rc0')>, <Link https://files.pythonhosted.org/packages/ca/7d/65230ab363f7aeac0345a6c8f46d87aa05589c2e177427deaaf7718e1cd5/tensorflow-2.0.0rc0-cp35-cp35m-macosx_10_11_x86_64.whl#sha256=306500d13b5774c3b62e1141c261526106620f549abe05f2a02f368e47e66423 (from https://pypi.org/simple/tensorflow/)>)>, <InstallationCandidate('tensorflow', <Version('2.0.0rc0')>, <Link https://files.pythonhosted.org/packages/11/ea/3a9917b2f6862ba5db774bbbe0d03602aa76dfa64331753936097353f126/tensorflow-2.0.0rc0-cp35-cp35m-manylinux2010_x86_64.whl#sha256=1cfc662da9b8e7abd0aa8f49477450363bd9a413cd6e6bd1bcfcdeab1f0ed50a (from https://pypi.org/simple/tensorflow/)>)>, <InstallationCandidate('tensorflow', <Version('2.0.0rc0')>, <Link https://files.pythonhosted.org/packages/f0/6b/32059f5ed92041bacc6ef7aaa3cceab4ffcfefac4ae6140775d9208e2fb1/tensorflow-2.0.0rc0-cp35-cp35m-win_amd64.whl#sha256=393010ffa7b0bdbd28cff33abdc665cd1f4b82e957d759c529f2393f0adf8bee (from https://pypi.org/simple/tensorflow/)>)>, <InstallationCandidate('tensorflow', <Version('2.0.0rc0')>, <Link https://files.pythonhosted.org/packages/ba/9f/5f5870278e991c213edc08ad40ce5d7869935842bc4ad46a2ebc7cdda2c1/tensorflow-2.0.0rc0-cp36-cp36m-macosx_10_11_x86_64.whl#sha256=31f9b2fc23ac21c7532c5d75c6193ac1a0c2a57ecaabdee1184f126a4c652898 (from https://pypi.org/simple/tensorflow/)>)>, <InstallationCandidate('tensorflow', <Version('2.0.0rc0')>, <Link https://files.pythonhosted.org/packages/fb/4b/77f0965ec7e8a76d3dcd6a22ca8bbd2b934cd92c4ded43fef6bea5ff3258/tensorflow-2.0.0rc0-cp36-cp36m-manylinux2010_x86_64.whl#sha256=d6555cd775284ab3d59f31f3cea3b8d3e2f91cf212940fd2e1ec836ab958a6d6 (from https://pypi.org/simple/tensorflow/)>)>, <InstallationCandidate('tensorflow', <Version('2.0.0rc0')>, <Link https://files.pythonhosted.org/packages/08/99/c5966073fd069f8a2e78219e511b480237828d5697dbca317e3008b8ce3f/tensorflow-2.0.0rc0-cp36-cp36m-win_amd64.whl#sha256=27abf4e46683ad4c8369f7fa704e3e7fba87aa5add72e9b2d969b46808ee8b49 (from https://pypi.org/simple/tensorflow/)>)>, <InstallationCandidate('tensorflow', <Version('2.0.0rc0')>, <Link https://files.pythonhosted.org/packages/50/ad/3b0a96b85071860a8e1a9ecf2bf11d36e349b996bed9293445ca38487821/tensorflow-2.0.0rc0-cp37-cp37m-macosx_10_11_x86_64.whl#sha256=89925b73310f14a80fcff28fa23c28f77772b7418e8d170d56d5cad79ed51a2b (from https://pypi.org/simple/tensorflow/)>)>, <InstallationCandidate('tensorflow', <Version('2.0.0rc0')>, <Link https://files.pythonhosted.org/packages/0d/9b/29c187df5563fb29fc66eea7f6115a33c5f2e580ea78c899fde72cebf624/tensorflow-2.0.0rc0-cp37-cp37m-manylinux2010_x86_64.whl#sha256=44c83c9b6e3a3ad3dd3e456341595e350f16c49158ef9cc059962c1425f23797 (from https://pypi.org/simple/tensorflow/)>)>, <InstallationCandidate('tensorflow', <Version('2.0.0rc0')>, <Link https://files.pythonhosted.org/packages/67/39/c5d4179917f0b548756d4c1ee91230855ddfbde4f6ffa956e63c6f8d68e4/tensorflow-2.0.0rc0-cp37-cp37m-win_amd64.whl#sha256=b96951be8ff2090752b559d01a9e3c2e83e3bf416fd56bc0736bfd32364139ac (from https://pypi.org/simple/tensorflow/)>)>]

Is it normal that the 2.7 and the macosx versions are still a candidate at that point?

[fgervais]

I have a bit more info on this.

The version of tensorflow in play here is "2.0.0-rc0". This version is the first to use the manylinux2010 PEP571 tags.

I'm not really familiar with these tags so I'm not sure what is going on but on my system:

 lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.2 LTS
Release:    18.04
Codename:   bionic

these are the tags I have:

[('cp36', 'cp36m', 'manylinux1_x86_64'),
 ('cp36', 'cp36m', 'linux_x86_64'),
 ('cp36', 'abi3', 'manylinux1_x86_64'),
 ('cp36', 'abi3', 'linux_x86_64'),
 ('cp36', 'none', 'manylinux1_x86_64'),
 ('cp36', 'none', 'linux_x86_64'),
 ('cp35', 'abi3', 'manylinux1_x86_64'),
 ('cp35', 'abi3', 'linux_x86_64'),
 ('cp34', 'abi3', 'manylinux1_x86_64'),
 ('cp34', 'abi3', 'linux_x86_64'),
 ('cp33', 'abi3', 'manylinux1_x86_64'),
 ('cp33', 'abi3', 'linux_x86_64'),
 ('cp32', 'abi3', 'manylinux1_x86_64'),
 ('cp32', 'abi3', 'linux_x86_64'),
 ('py3', 'none', 'manylinux1_x86_64'),
 ('py3', 'none', 'linux_x86_64'),
 ('cp36', 'none', 'any'),
 ('cp3', 'none', 'any'),
 ('py36', 'none', 'any'),
 ('py3', 'none', 'any'),
 ('py35', 'none', 'any'),
 ('py34', 'none', 'any'),
 ('py33', 'none', 'any'),
 ('py32', 'none', 'any'),
 ('py31', 'none', 'any'),
 ('py30', 'none', 'any')]

The web tells me this is the older linux magic tag defined in PEP513.

So when in pipenv.patched.notpip._internal.index._candidate_sort_key() all packages gets the same priority (-26 in my case) and so the python 2.7 just happen to be the picked one.

What would be the way to go from here?
Is it just a matter of adding manylinux2010 somewhere?

Also it there a way to print this tag list other than adding debug in pipenv like I did? @fujiaoliu you could then verify that this is in fact your problem too and that I'm not just hijacking your thread.

[fgervais]

Well it seems the issue was described here all along and I missed it.

I updated pipenv to master and verified that 2.0.0-rc0 can be installed.

@fujiaoliu if you want to test on your side you can install pipenv in pipenv, it's quite easy and comes at no risk for your system. Also you'll get that nice inception feel :)

git clone https://github.com/pypa/pipenv.git
cd pipenv
git checkout master
cd ../project
pipenv install -e $(pwd)/../pipenv/
pipenv shell
pipenv install --pre tensorflow==2.0.0-rc0

[frostming]

@fgervais So is the problem solved on master branch?

[fgervais]

@frostming yes

[fujiaoliu]

thanks for your help @fgervais. will git it a shot 😉

[mmartinortiz]

I have the same issue, and @fgervais comment put me on the right track. Since this seems to be a problem of pipenv, I'm installing tensorflow using

pipenv run pip install tensorflow

The difference is that if we install pipenv from the repository and later tensorflow using the updated pipenv, the Pipfile is updated and any future new update of the virtual environment will cause the same problem we had at the beginning (as it is the system's pipenv the one updating the environment).

[pacospace]

Any update on this issue?

[quantverse]

Yeah, this is very annoying. I will need to dismiss Pipenv entirely in order to use Tensorflow 2, or I need to install Pipenv from master branch. I really wonder why there was no new release in last year...

[pacospace]

I don't see much changes in pipenv source code for few months now...

[nithin-vijayan]

Same issue here. :(

`Finding secondary dependencies:
mock==3.0.5 requires six
wrapt==1.11.2 requires -
gast==0.2.2 requires -
keras-applications==1.0.8 requires h5py, numpy>=1.9.1
scipy==1.3.3 requires numpy>=1.13.3
pdfminer==20191125 requires pycryptodome
pycryptodome==3.9.4 requires -
numpy==1.17.4 requires -
opt-einsum==3.1.0 requires numpy>=1.7
tensorflow==2.0.0 requires absl-py>=0.7.0, astor>=0.6.0, functools32>=3.2.3, gast==0.2.2, google-pasta>=0.1.6, grpcio>=1.8.6, keras-applications>=1.0.8, keras-preprocessing>=1.0.5, mock>=2.0.0, numpy<2.0,>=1.16.0, opt-einsum>=2.3.2, protobuf>=3.6.1, six>=1.10.0, tensorboard<2.1.0,>=2.0.0, tensorflow-estimator<2.1.0,>=2.0.0, termcolor>=1.1.0, wheel, wrapt>=1.11.1
pyyaml==5.2 requires -
six==1.13.0 requires -
keras==2.3.1 requires h5py, keras-applications>=1.0.6, keras-preprocessing>=1.0.5, numpy>=1.9.1, pyyaml, scipy>=0.14, six>=1.9.0
tensorflow-estimator==2.0.1 requires -
pymupdf==1.16.9 requires -
functools32==3.2.3.post2 not in cache, need to check index
Traceback (most recent call last):
File "/home/nk/.local/lib/python3.7/site-packages/pipenv/resolver.py", line 126, in
main()
File "/home/nk/.local/lib/python3.7/site-packages/pipenv/resolver.py", line 119, in main
parsed.requirements_dir, parsed.packages)
File "/home/nk/.local/lib/python3.7/site-packages/pipenv/resolver.py", line 85, in _main
requirements_dir=requirements_dir,
File "/home/nk/.local/lib/python3.7/site-packages/pipenv/resolver.py", line 69, in resolve
req_dir=requirements_dir
File "/home/nk/.local/lib/python3.7/site-packages/pipenv/utils.py", line 726, in resolve_deps
req_dir=req_dir,
File "/home/nk/.local/lib/python3.7/site-packages/pipenv/utils.py", line 480, in actually_resolve_deps
resolved_tree = resolver.resolve()
File "/home/nk/.local/lib/python3.7/site-packages/pipenv/utils.py", line 385, in resolve
results = self.resolver.resolve(max_rounds=environments.PIPENV_MAX_ROUNDS)
File "/home/nk/.local/lib/python3.7/site-packages/pipenv/patched/piptools/resolver.py", line 102, in resolve
has_changed, best_matches = self._resolve_one_round()
File "/home/nk/.local/lib/python3.7/site-packages/pipenv/patched/piptools/resolver.py", line 206, in _resolve_one_round
for dep in self._iter_dependencies(best_match):
File "/home/nk/.local/lib/python3.7/site-packages/pipenv/patched/piptools/resolver.py", line 301, in _iter_dependencies
dependencies = self.repository.get_dependencies(ireq)
File "/home/nk/.local/lib/python3.7/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 234, in get_dependencies
legacy_results = self.get_legacy_dependencies(ireq)
File "/home/nk/.local/lib/python3.7/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 426, in get_legacy_dependencies
results, ireq = self.resolve_reqs(download_dir, ireq, wheel_cache)
File "/home/nk/.local/lib/python3.7/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 297, in resolve_reqs
results = resolver._resolve_one(reqset, ireq)
File "/home/nk/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/resolve.py", line 260, in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "/home/nk/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/resolve.py", line 213, in _get_abstract_dist_for
self.require_hashes
File "/home/nk/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 294, in prepare_linked_requirement
abstract_dist.prep_for_dist(finder, self.build_isolation)
File "/home/nk/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 127, in prep_for_dist
self.req.run_egg_info()
File "/home/nk/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/req/req_install.py", line 474, in run_egg_info
command_desc='python setup.py egg_info')
File "/home/nk/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/utils/misc.py", line 705, in call_subprocess
% (command_desc, proc.returncode, cwd))
pipenv.patched.notpip._internal.exceptions.InstallationError: Command "python setup.py egg_info" failed with error code 1 in /tmp/tmpibww1vyabuild/functools32/

`

[shamoons]

Any resolution?

[mmartinortiz]

Some days ago, @techalchemy commented that the project is not dead (thanks, techalchemy for the update) and a new release will come soon.

@shamoons , if you check fgervais' reaction you can see that master branch has a patch for this issue, and it is a matter of time that it gets released.