Pipenv: Pipfile.lock hashes change between operating systems

[frankh]

Currently if I lock a Pipfile in OSX and then try to install the lock file in linux it fails as many of the packages seem to have different versions based on the OS, with different hashes

An error occured while installing!
THESE PACKAGES DO NOT MATCH THE HASHES FROM Pipfile.lock!. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    gevent==1.2.1 from https://pypi.python.org/packages/a1/41/65eee925d5fe1e1a83ab75e2e83952515f6789aaf561a8183e4c515b49a6/gevent-1.2.1-cp27-cp27mu-manylinux1_x86_64.whl#md5=8b09f79ffb73bfa47806f2719d0b34f1 (from -r /tmp/tmpvJReOn-requirements.txt (line 6)):
        Expected sha256 f68fa3b23e518ce12f8b14264ba567eb2f8a22923f85257d5043fb81e78dbc43
             Got        5b9ba1abd75ed7e3d667dd52baba60e902ab0de583aa92d5f80c6fecdae06a28

    greenlet==0.4.12 from https://pypi.python.org/packages/fd/b8/dd04dd0ce13ba7412eac94f6e0456e26e45d0e1df5b8019fb15832d850b0/greenlet-0.4.12-cp27-cp27mu-manylinux1_x86_64.whl#md5=1719cc1f62dd52fd5af1ee4dc38c83d2 (from -r /tmp/tmpvJReOn-requirements.txt (line 21)):
        Expected sha256 e4c99c6010a5d153d481fdaf63b8a0782825c0721506d880403a3b9b82ae347e
             Got        21232907c8c26838b16915bd8fbbf82fc70c996073464cc70981dd4a96bc841c

    cryptography==1.7.2 from https://pypi.python.org/packages/99/df/71c7260003f5c469cec3db4c547115df39e9ce6c719a99e067ba0e78fd8a/cryptography-1.7.2.tar.gz#md5=fade66de437392ed1ba6980768626204 (from -r /tmp/tmpvJReOn-requirements.txt (line 24)):
        Expected sha256 64c875d0384c8c1dc6134fc7ddcfe39b09ea9d5bb2c7616792006e83d85db52c
             Got        878cb68b3da3d493ffd68f36db11c29deee623671d3287c3f8d685117ffda9a9

    billiard==3.3.0.23 from https://pypi.python.org/packages/64/a6/d7b6fb7bd0a4680a41f1d4b27061c7b768c673070ba8ac116f865de4e7ca/billiard-3.3.0.23.tar.gz#md5=6ee416e1e7c8d8164ce29d7377cca6a4 (from -r /tmp/tmpvJReOn-requirements.txt (line 26)):
        Expected sha256 c0cbe8d45ba8d8213ad68ef9a1881002a151569c9424d551634195a18c3a4160
             Got        692a2a5a55ee39a42bcb7557930e2541da85df9ea81c6e24827f63b80cd39d0b

    cffi==1.9.1 from https://pypi.python.org/packages/5f/bc/9fd2ce704560d2be1b13fe8939bca4941c8cf60e7fc97a0abd73edd44686/cffi-1.9.1-cp27-cp27mu-manylinux1_x86_64.whl#md5=dd6e35f133ad90e18cfdb44e0aa7d62b (from -r /tmp/tmpvJReOn-requirements.txt (line 48)):
        Expected sha256 d3e3063af1fa6b59e255da9a812891cdaf24b90fbaf653c02797871069b7c4c9
             Got        9163f7743cf9991edaddf9cf886708e288fab38e1b9fec9c41c15c85c8f7f147

[nateprewitt]

Hey @frankh, we're currently only able to support lock files on similar platforms because of this. This also applies to lock files between Python 2 and 3 due to version specific wheels.

For now the suggested practice is to run pipenv lock on new environments before trying to use pipenv.

[mixxorz]

Is this something you're planning on fixing? This essentially means anyone who doesn't develop on Linux won't be able to deploy their applications without creating a new lock file for the production environment, which kind of defeats the purpose of the lock file.

[nateprewitt]

Hey @mixxorz, this was originally discussed in #147. pip will automatically try to download pre-built wheels when possible which can be platform (and major version) specific. This means we'll usually be hashing files specific to that platform.

The "fix" is either to drop wheel support (which I think is very unlikely), or to download every file for a given version and keep multiple hashes. The issue with the latter is downloading every possible copy of a package from possibly multiple sources becomes unacceptably costly as project sizes increase. Trying to predict every platform a project may be moved to is something that's hard to get right with the amount of customization allowed.

As I stated in #147, I agree re-locking does somewhat defeat the purpose of the hash to have to run a lock before deploy to a specific type of architecture. Until a better solution presents itself though, I'm not sure we can do anything helpful for all users here. If you have suggestions/thoughts on specifics, I'd be happy to hear them.

[nateprewitt]

Also, as I failed to document after the discussion in #147, you can create a pip.conf file on all of your deployments with no-binary=:all:. This will force pip to only download source archives which should be universal. This is not something we're going to explicitly support though, so it's a voluntary workaround for the time being.

[jacebrowning]

@nateprewitt Is there any way to disable hashing? FWIW, Bundler doesn't have this problem.

[nateprewitt]

@jacebrowning, that's probably a question for Kenneth. I don't think we want to disable hashes though.

As for Bundler, I'm honestly not incredibly well versed in how gems are distributed but I do know Bundler uses more specificity including a Platforms section. I haven't spent much time in Ruby which is why I'm likely missing some obvious insights here. That said, pipenv isn't intended to be a python Bundler clone either. I'll try to do some reading this weekend and see if there's anything simple we're missing.

[jacebrowning]

Gemfile.lock does not include hashes of Gems installed from package repositories, but does include SHAs from dependencies installed from source. If I am unable to develop on one system and deploy on another, than Pipfile.lock serves no purpose for me.

[nateprewitt]

@jacebrowning yeah, I completely get where you're coming from with that. I've been doing a bit of reading since my last post. I'll try to get some time to talk to @kennethreitz about options here, unless he has further thoughts he'd like to post.

[wasabigeek]

@nateprewitt to clarify https://github.com/kennethreitz/pipenv/issues/210#issuecomment-283182256, if I'm running into hash conflicts, I should not commit Pipfile.lock (edit: or, it doesn't matter), and add run pipfile lock in my deployment script before pipenv install?

[nateprewitt]

@wasabigeek this issue is quite old. You should be able to move your lockfiles between machines now. If you're experiencing issues with pipenv 8.2.7+ please open a new issue. Thanks!

[wasabigeek]

@nateprewitt thanks - I got an error about the hashes, but on further inspection seems it's due to the pre-included packages in my host's images, not related to Pipenv. I'll update if I find out otherwise otherwise. Thanks!

[wasabigeek]

@nateprewitt I did pip --upgrade to 8.3.1 but still encountered the following error after running my deploy script:

THESE PACKAGES DO NOT MATCH THE HASHES FROM Pipfile.lock!. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    oauthlib==1.0.3 from file:///usr/share/pip-wheels/oauthlib-1.0.3-py3-none-any.whl (from -r /tmp/pipenv-az9xn_nu-requirement.txt (line 1)):
Expected sha256 ef4bfe4663ca3b97a995860c0173b967ebd98033d02f38c9e1b2cbb6c191d9ad
Got        5454640e006c0eecce902638cd7a3fda384d083abf3f7412173fb075610e2256

I confirmed the hashes using pip download and pip hash on local and on the server, they're as above. Are there other factors that could contribute to this?

(In the end I did pipenv install <package> on the server itself)

[kingb]

@nateprewitt

The "fix" is either to drop wheel support (which I think is very unlikely), or to download every file for a given version and keep multiple hashes. The issue with the latter is downloading every possible copy of a package from possibly multiple sources becomes unacceptably costly as project sizes increase. Trying to predict every platform a project may be moved to is something that's hard to get right with the amount of customization allowed.

As I stated in #147, I agree re-locking does _somewhat_ defeat the purpose of the hash to have to run a lock before deploy to a specific type of architecture. Until a better solution presents itself though, I'm not sure we can do anything helpful for all users here. If you have suggestions/thoughts on specifics, I'd be happy to hear them.

One thought. What about updating the Pipefile.lock format such that any platform-specific wheels get their own section/hashes. Meaning when another user locks on a new platform, the lock file gets updates with the new wheels and hashes them for the new platform, but also updates the platform-specific wheels already in the .lock file?

In other words, narrow down the wheels that need to be hashed to only the platforms users have locked from previously?

[pedz]

Similar to the previous comment ... would it work to have the ability to add the list of platforms that the user wants to lock file to work on. As stated, trying to guess would be hard but in my case, I need the lock file on two platforms.

[grazius]

Have the same need to generate an universal Pipfile.lock
I make development on win32 and deploy in a docker image based on a linux distribution.
Without this ability the Pipfile.lock is unusable no ?