Pip-tools: piptools depcache is shared between implementations causing incorrect dependencies

[asottile]

Environment Versions

$ uname -a
Linux asottile-MacBookPro 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.3 LTS
Release:    18.04
Codename:   bionic
$ /tmp/x/venv36/bin/pip freeze --all
Click==7.0
pip==20.0.2
pip-tools==4.4.0
setuptools==45.1.0
six==1.14.0
wheel==0.34.1
$ /tmp/x/venv36/bin/python --version --version
Python 3.6.9 (default, Nov  7 2019, 10:44:02) 
[GCC 8.3.0]
$ /tmp/x/venvpp36/bin/python --version --version
Python 3.6.9 (1608da62bfc7, Dec 23 2019, 10:50:04)
[PyPy 7.3.0 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)]

Steps to replicate

set -euxo pipefail

cache="$PWD/cache"
export XDG_CACHE_HOME="$cache"
export CUSTOM_COMPILE_COMMAND='<<redacted>>'

rm -rf "$cache" venv36 venvpp36 reqs*.txt

virtualenv venv36 -ppython3.6 >& /dev/null
venv36/bin/pip install -qq pip-tools
virtualenv venvpp36 -ppypy3 >& /dev/null
venvpp36/bin/pip install -qq pip-tools

echo 'cffi' > requirements.in

venv36/bin/pip-compile --output-file reqs36_1.txt
venvpp36/bin/pip-compile --output-file reqspp36_1.txt

rm -rf "$cache"
venvpp36/bin/pip-compile --output-file reqspp36_2.txt
venv36/bin/pip-compile --output-file reqs36_2.txt

diff -u reqs36_1.txt reqs36_2.txt
diff -u reqspp36_1.txt reqspp36_2.txt

Expected result

I expect no diff to be produced and the script to exit 0

Actual result

$ bash t.sh 
+ cache=/tmp/x/cache
+ export XDG_CACHE_HOME=/tmp/x/cache
+ XDG_CACHE_HOME=/tmp/x/cache
+ export 'CUSTOM_COMPILE_COMMAND=<<redacted>>'
+ CUSTOM_COMPILE_COMMAND='<<redacted>>'
+ rm -rf /tmp/x/cache venv36 venvpp36 reqs36_1.txt reqs36_2.txt reqspp36_1.txt reqspp36_2.txt
+ virtualenv venv36 -ppython3.6
+ venv36/bin/pip install -qq pip-tools
+ virtualenv venvpp36 -ppypy3
+ venvpp36/bin/pip install -qq pip-tools
+ echo cffi
+ venv36/bin/pip-compile --output-file reqs36_1.txt
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    <<redacted>>
#
cffi==1.13.2
pycparser==2.19           # via cffi
+ venvpp36/bin/pip-compile --output-file reqspp36_1.txt
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    <<redacted>>
#
cffi==1.13.2
pycparser==2.19           # via cffi
+ rm -rf /tmp/x/cache
+ venvpp36/bin/pip-compile --output-file reqspp36_2.txt
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    <<redacted>>
#
cffi==1.13.2
+ venv36/bin/pip-compile --output-file reqs36_2.txt
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    <<redacted>>
#
cffi==1.13.2
+ diff -u reqs36_1.txt reqs36_2.txt
--- reqs36_1.txt    2020-01-30 08:46:36.913122945 -0800
+++ reqs36_2.txt    2020-01-30 08:46:41.696946374 -0800
@@ -5,4 +5,3 @@
 #    <<redacted>>
 #
 cffi==1.13.2
-pycparser==2.19           # via cffi

as you can see here, the pypy 3.6 cache is poisoning the result of the cpython 3.6 pip-compile

[atugushev]

Hello @asottile! 🎉🎉🎉

Thanks for the issue! Pip-tools has the same dependency cache file for cpython and pypy, see: https://github.com/jazzband/pip-tools/blob/90a052bc210fdf9ce5c09f2c450d4715745f7fdd/piptools/cache.py#L54-L55

The file name could be depcache-pp{}.json if hasattr(sys, "pypy_version_info") is True.

[asottile]

1050 has a patch for this, it passes my interactive test above