Pipenv: cannot install google-cloud-datastore

Created on 5 Mar 2018  ·  18Comments  ·  Source: pypa/pipenv

Be sure to check the existing issues, both open and closed.

Describe the issue briefly here.

Describe your environment
  1. OS Type ubuntu 16.0.4
  2. Python version: $ python -V 3.5.2 (python of virtualenv will be different)
  3. Pipenv version: $ pipenv --version 11.0.2
Expected result

Pipfile.lock should be created

Actual result

When possible, provide the verbose output (--verbose), especially for locking and dependencies resolving issues.

pipenv install --python=3.6.4 --verbose
Creating a virtualenv for this project…
Using /home/david/.pyenv/versions/3.6.4/bin/python3.6m to create virtualenv…
⠋Running virtualenv with interpreter /home/david/.pyenv/versions/3.6.4/bin/python3.6m
Using base prefix '/home/david/.pyenv/versions/3.6.4'
New python executable in /home/david/.local/share/virtualenvs/test-FTV5fq3F/bin/python3.6m
Also creating executable in /home/david/.local/share/virtualenvs/test-FTV5fq3F/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /home/david/.local/share/virtualenvs/test-FTV5fq3F
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Could not find a version that matches futures>=3.2.0
Tried: 0.2.python3, 0.1, 0.2, 1.0, 2.0, 2.1, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.1.5, 2.1.5, 2.1.6, 2.1.6, 2.2.0, 2.2.0, 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.0.4, 3.0.5, 3.1.0, 3.1.1
Steps to replicate

here is my pipfile

[[source]]

verify_ssl = true
url = "https://pypi.python.org/simple"
name = "pypi"


[dev-packages]

yapf = "*"
pylint = "*"
mypy = "*"
rope = "*"
coverage = "*"
responses = "*"


[packages]

requests = "*"
google-cloud-datastore = "*"
google-cloud-storage = "*"
falcon = "*"
geopy = "*"
googlemaps = "*"
gunicorn = "*"
networkx = "*"
pymysql = "*"
python-dateutil = "*"
sqlalchemy = "*"


[pipenv]

allow_prereleases = true

Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

picture david-gang
👍1

Most helpful comment

Thanks for all the hard work. This package manager is really awesome. For the first time i know exactly what my dependencies are which i installed and what the transitive dependencies are. Combined with the fact that installs are deterministic, makes this package manager first class, like cargo

picture david-gang on 8 Mar 2018
🎉3 ❤2 😄2 👍2

All 18 comments

try pipenv lock --clear

kennethreitz42 picture kennethreitz42 on 5 Mar 2018

Seems that the culprit is google-cloud-datastore but i don't know how to fix this:

Here are the exact steps

pipenv install --python=3.6.4 --verbose
Virtualenv already exists!
Removing existing virtualenv…
Creating a virtualenv for this project…
Using /home/david/.pyenv/versions/3.6.4/bin/python3.6m to create virtualenv…
⠋Running virtualenv with interpreter /home/david/.pyenv/versions/3.6.4/bin/python3.6m
Using base prefix '/home/david/.pyenv/versions/3.6.4'
New python executable in /home/david/.local/share/virtualenvs/test-FTV5fq3F/bin/python3.6m
Also creating executable in /home/david/.local/share/virtualenvs/test-FTV5fq3F/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /home/david/.local/share/virtualenvs/test-FTV5fq3F
Creating a Pipfile for this project…
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (625834)!
Installing dependencies from Pipfile.lock (625834)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00
To activate this project's virtualenv, run the following:
 $ pipenv shell

Now i run

pipenv install google-cloud-datastore --verbose >> log.txt 2>&1

The log is at: log.txt

when trying pipenv lock --clear
```
pipenv lock --clear --verbose
Locking [dev-packages] dependencies…
Using pip: -i https://pypi.python.org/simple

                      ROUND 1

Current constraints:

Finding the best candidates:

Finding secondary dependencies:

Result of round 1: stable, done

Locking [packages] dependencies…

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Could not find a version that matches futures>=3.2.0
Tried: 0.2.python3, 0.1, 0.2, 1.0, 2.0, 2.1, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.1.5, 2.1.5, 2.1.6, 2.1.6, 2.2.0, 2.2.0, 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.0.4, 3.0.5, 3.1.0, 3.1.1

david-gang picture david-gang on 5 Mar 2018

Continuing to debug the issue. It looks like pipenv is ignoring the python version in extra dependencies:

/home/david/.local/share/virtualenvs/python-EYYLSEUr/bin/python  /usr/local/lib/python3.5/dist-packages/pipenv/resolver.py google-cloud-datastore --verbose





Using pip: -i https://pypi.python.org/simple

                          ROUND 1                           
Current constraints:
  google-cloud-datastore

Finding the best candidates:
  found candidate google-cloud-datastore==1.6.0 (constraint was <any>)

Finding secondary dependencies:
  google-cloud-datastore==1.6.0 requires google-api-core[grpc]<2.0.0dev,>=1.0.0, google-api-core[grpc]<2.0.0dev,>=1.0.0, google-cloud-core<0.29dev,>=0.28.0, google-cloud-core<0.29dev,>=0.28.0

New dependencies found in this round:
  adding ['google-api-core', '<2.0.0dev,>=1.0.0', "['grpc']"]
  adding ['google-cloud-core', '<0.29dev,>=0.28.0', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2                           
Current constraints:
  google-api-core[grpc]<2.0.0dev,>=1.0.0
  google-cloud-core<0.29dev,>=0.28.0
  google-cloud-datastore

Finding the best candidates:
  found candidate google-api-core[grpc]==1.0.0 (constraint was >=1.0.0,<2.0.0dev)
  found candidate google-cloud-core==0.28.1 (constraint was >=0.28.0,<0.29dev)
  found candidate google-cloud-datastore==1.6.0 (constraint was <any>)

Finding secondary dependencies:
  google-cloud-datastore==1.6.0 requires google-api-core[grpc]<2.0.0dev,>=1.0.0, google-api-core[grpc]<2.0.0dev,>=1.0.0, google-cloud-core<0.29dev,>=0.28.0, google-cloud-core<0.29dev,>=0.28.0
  google-cloud-core==0.28.1 requires google-api-core<2.0.0dev,>=0.1.1, google-api-core<2.0.0dev,>=0.1.1

New dependencies found in this round:
  adding ['futures', '>=3.2.0', '[]']
  adding ['google-api-core', '<2.0.0dev,>=0.1.1,>=1.0.0', "['grpc']"]
  adding ['google-auth', '<2.0.0dev,>=0.4.0', '[]']
  adding ['googleapis-common-protos', '<2.0dev,>=1.5.3', '[]']
  adding ['grpcio', '>=1.8.2', '[]']
  adding ['protobuf', '>=3.0.0', '[]']
  adding ['pytz', '', '[]']
  adding ['requests', '<3.0.0dev,>=2.18.0', '[]']
  adding ['six', '>=1.10.0', '[]']
Removed dependencies in this round:
  removing ['google-api-core', '<2.0.0dev,>=1.0.0', "['grpc']"]
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 2: not stable

                          ROUND 3                           
Current constraints:
  futures>=3.2.0
  google-api-core[grpc]<2.0.0dev,>=0.1.1,>=1.0.0
  google-auth<2.0.0dev,>=0.4.0
  google-cloud-core<0.29dev,>=0.28.0
  google-cloud-datastore
  googleapis-common-protos<2.0dev,>=1.5.3
  grpcio>=1.8.2; extra == "grpc"
  protobuf>=3.0.0
  pytz
  requests<3.0.0dev,>=2.18.0
  six>=1.10.0

Finding the best candidates:
Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Could not find a version that matches futures>=3.2.0
Tried: 0.2.python3, 0.1, 0.2, 1.0, 2.0, 2.1, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.1.5, 2.1.5, 2.1.6, 2.1.6, 2.2.0, 2.2.0, 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.0.4, 3.0.5, 3.1.0, 3.1.1
Using pip: -i https://pypi.python.org/simple

                          ROUND 1                           
Current constraints:
  google-cloud-datastore

Finding the best candidates:
  found candidate google-cloud-datastore==1.6.0 (constraint was <any>)

Finding secondary dependencies:
  google-cloud-datastore==1.6.0 requires google-api-core[grpc]<2.0.0dev,>=1.0.0, google-api-core[grpc]<2.0.0dev,>=1.0.0, google-cloud-core<0.29dev,>=0.28.0, google-cloud-core<0.29dev,>=0.28.0

New dependencies found in this round:
  adding ['google-api-core', '<2.0.0dev,>=1.0.0', "['grpc']"]
  adding ['google-cloud-core', '<0.29dev,>=0.28.0', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2                           
Current constraints:
  google-api-core[grpc]<2.0.0dev,>=1.0.0
  google-cloud-core<0.29dev,>=0.28.0
  google-cloud-datastore

Finding the best candidates:
  found candidate google-api-core[grpc]==1.0.0 (constraint was >=1.0.0,<2.0.0dev)
  found candidate google-cloud-core==0.28.1 (constraint was >=0.28.0,<0.29dev)
  found candidate google-cloud-datastore==1.6.0 (constraint was <any>)

Finding secondary dependencies:
  google-cloud-datastore==1.6.0 requires google-api-core[grpc]<2.0.0dev,>=1.0.0, google-api-core[grpc]<2.0.0dev,>=1.0.0, google-cloud-core<0.29dev,>=0.28.0, google-cloud-core<0.29dev,>=0.28.0
  google-cloud-core==0.28.1 requires google-api-core<2.0.0dev,>=0.1.1, google-api-core<2.0.0dev,>=0.1.1

New dependencies found in this round:
  adding ['futures', '>=3.2.0', '[]']
  adding ['google-api-core', '<2.0.0dev,>=0.1.1,>=1.0.0', "['grpc']"]
  adding ['google-auth', '<2.0.0dev,>=0.4.0', '[]']
  adding ['googleapis-common-protos', '<2.0dev,>=1.5.3', '[]']
  adding ['grpcio', '>=1.8.2', '[]']
  adding ['protobuf', '>=3.0.0', '[]']
  adding ['pytz', '', '[]']
  adding ['requests', '<3.0.0dev,>=2.18.0', '[]']
  adding ['six', '>=1.10.0', '[]']
Removed dependencies in this round:
  removing ['google-api-core', '<2.0.0dev,>=1.0.0', "['grpc']"]
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 2: not stable

                          ROUND 3                           
Current constraints:
  futures>=3.2.0
  google-api-core[grpc]<2.0.0dev,>=0.1.1,>=1.0.0
  google-auth<2.0.0dev,>=0.4.0
  google-cloud-core<0.29dev,>=0.28.0
  google-cloud-datastore
  googleapis-common-protos<2.0dev,>=1.5.3
  grpcio>=1.8.2; extra == "grpc"
  protobuf>=3.0.0
  pytz
  requests<3.0.0dev,>=2.18.0
  six>=1.10.0

Finding the best candidates:
Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Could not find a version that matches futures>=3.2.0
Tried: 0.2.python3, 0.1, 0.2, 1.0, 2.0, 2.1, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.1.5, 2.1.5, 2.1.6, 2.1.6, 2.2.0, 2.2.0, 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.0.4, 3.0.5, 3.1.0, 3.1.1

But look at setup.py

extras = {
    'grpc': 'grpcio>=1.8.2',
    ':python_version < "3.2"': 'futures>=3.2.0',
}

I am using a bigger version that python 3.2

Now i am stuck.

david-gang picture david-gang on 5 Mar 2018
👍1

I think i finally found the bug:
https://github.com/pypa/pipenv/blob/13fe0abccbf8fca9283078636f0272aac3710127/pipenv/patched/piptools/repositories/pypi.py#L149
has to be changed to 

if 'extra' not in repr(i.markers) and i.markers.evaluate()

I am very new to this project and this is a huge one but essentially we have to evaluate the dependency as we make it in other places.
Here is the json where the parsing fails:
https://pypi.org/pypi/google-api-core/json

@kennethreitz Please guide me if this is a correct direction

david-gang picture david-gang on 5 Mar 2018

@david-gang the extras_require format of conditional dependency parsing for python versions isn’t supported by pip tools or setuptools anymore: http://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-platform-specific-dependencies - it is recommended that you use the format specified there. We have been encouraging people to submit patches upstream to correct this

techalchemy picture techalchemy on 6 Mar 2018
👍1

hi @techalchemy , I opened an upstream bug, but theoretically pipenv should handle this. It won't always be possible to convince the maintainer of the python package to change the setup.py, Some packages may not be maintained actively.

BR

david-gang picture david-gang on 6 Mar 2018

Or indeed, you could be actively trying to pin an old version using the old syntax.

intrinseca picture intrinseca on 6 Mar 2018

I agree that it would be nice to support this but it is not high priority since the resolver stack is quite complex

techalchemy picture techalchemy on 6 Mar 2018

you have no idea how tall of an order y'all are putting in here

kennethreitz42 picture kennethreitz42 on 6 Mar 2018

the amount of work we've put into making what works not work is tremendous

kennethreitz42 picture kennethreitz42 on 6 Mar 2018

FYI: Here is the PR for google-cloud-python https://github.com/GoogleCloudPlatform/google-cloud-python/pull/4991

david-gang picture david-gang on 7 Mar 2018

HI @kennethreitz ,

As a workaround what would you suggest?
Using a pipenv with a version previous of 0.11 or just install with the --skip-lock option?
Is there another workaround? I need the certain library.

Thanks for all your excellent work :+1:

david-gang picture david-gang on 8 Mar 2018

Considering they seems willing to fix it, both you mentioned should serve as a nice stopgap. Another is to fork it and install from Git until they release the fix.

uranusjr picture uranusjr on 8 Mar 2018

looks like version 11.1.5 fixed this issue

david-gang picture david-gang on 8 Mar 2018
👍1

\o/

kennethreitz42 picture kennethreitz42 on 8 Mar 2018
👍1

that wasn't easy :)

kennethreitz42 picture kennethreitz42 on 8 Mar 2018
👍1

Thanks for all the hard work. This package manager is really awesome. For the first time i know exactly what my dependencies are which i installed and what the transitive dependencies are. Combined with the fact that installs are deterministic, makes this package manager first class, like cargo

david-gang picture david-gang on 8 Mar 2018
🎉3 ❤2 😄2 👍2

@david-gang https://docs.pipenv.org/#user-testimonials

kennethreitz42 picture kennethreitz42 on 8 Mar 2018
👍2
Was this page helpful?
0 / 5 - 0 ratings

Related issues

randName picture randName  ·  3Comments
erinxocon picture erinxocon  ·  3Comments
Californian picture Californian  ·  3Comments
jacebrowning picture jacebrowning  ·  3Comments
leileigong picture leileigong  ·  3Comments