pip install --target --upgrade removes existing packages scripts

[changsijay]

Environment

• pip version:20.0.2
• Python version:3.8.2
• OS:MacOS

Description
pip with -t removed other file in target bin folder

Expected behavior

when install package ipython should not remove package pylint bin files

How to Reproduce

mkdir test
pip install -t test pylint
ls test/bin/
epylint  isort  pylint  pyreverse  symilar

pip install -U -t test ipython
ls test/bin/
easy_install  easy_install-3.8  iptest  iptest3  ipython  ipython3  pygmentize

Output

After install ipython, pylint is deleted.

[uranusjr]

Title edited to improve readability. I hope it is alright.

[deveshks]

As per the docs at https://pip.pypa.io/en/stable/reference/pip_install/#cmdoption-t , providing -U flag while installing will replace existing files/folders in the target directory.

This can also be observed by not passing the -U flag in the second pip install, due to which the contents of test/bin don't change, and warnings related to not being able to install because of files already being present are raised.

(.env) DeveshSinghMac:Desktop devesh$ pip --version
pip 20.0.2 from /Users/devesh/Desktop/.env/lib/python3.8/site-packages/pip (python 3.8)
(.env) DeveshSinghMac:Desktop devesh$ python --version
Python 3.8.2

(.env) DeveshSinghMac:Desktop devesh$ pip install -t test ipython
....
WARNING: Target directory /Users/devesh/Desktop/test/decorator-4.4.2.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/appnope already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/ipython-7.13.0.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/wcwidth already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/backcall already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/jedi-0.17.0.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/easy_install.py already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/pygments already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/pickleshare-0.7.5.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/IPython already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/Pygments-2.6.1.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/decorator.py already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/jedi already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/ipython_genutils already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/pickleshare.py already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/setuptools-46.1.3.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/__pycache__ already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/prompt_toolkit-3.0.5.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/six-1.14.0.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/prompt_toolkit already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/ipython_genutils-0.2.0.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/pexpect already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/ptyprocess-0.6.0.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/backcall-0.1.0-py3.8.egg-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/ptyprocess already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/six.py already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/pexpect-4.8.0.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/traitlets-4.3.3.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/appnope-0.1.0.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/setuptools already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/pkg_resources already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/parso-0.7.0.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/parso already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/traitlets already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/wcwidth-0.1.9.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/bin already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/devesh/Desktop/test/share already exists. Specify --upgrade to force replacement.

(.env) DeveshSinghMac:Desktop devesh$ ls test/bin/
epylint     isort       pylint      pyreverse   symilar

So I would say that providing -U, which ends up cleaning test/bin is the expected behaviour IMO

[gutsytechster]

@deveshks, the documentation you referred, says

Use --upgrade to replace existing packages in \

What I can infer is that it would replace existing packages with new versions. Should it remove the existing packages if not specified?

[deveshks]

Should it remove the existing packages if not specified?

It seems so from the output. All the files present in test/bin/ have been replaced after using --target --upgrade

[leos]

With pip 20.1:

~/ ❯❯❯ pip install -t test pylint --no-deps                                                                                                                                                 
Collecting pylint
  Using cached pylint-2.5.0-py3-none-any.whl (324 kB)
Installing collected packages: pylint
Successfully installed pylint-2.5.0
~/❯❯❯ pip install -t test ipython --no-deps                                                                                                                                                     
Collecting ipython
  Using cached ipython-7.14.0-py3-none-any.whl (782 kB)
Installing collected packages: ipython
Successfully installed ipython-7.14.0
WARNING: Target directory /home/vagrant/beeswaxio/test/bin already exists. Specify --upgrade to force replacement.
~/❯❯❯ ls test/bin                                                                                                                                                                               
epylint  pylint  pyreverse  symilar
~/❯❯❯ rm test
~/❯❯❯ pip install -t test pylint ipython --no-deps
Collecting pylint
  Using cached pylint-2.5.0-py3-none-any.whl (324 kB)
Collecting ipython
  Using cached ipython-7.14.0-py3-none-any.whl (782 kB)
Installing collected packages: pylint, ipython
Successfully installed pylint-2.5.0 ipython-7.14.0
~/beeswaxio ❯❯❯ ls test/bin                                                                                                                                                                               
epylint  iptest  iptest3  ipython  ipython3  pylint  pyreverse  symilar

I think the bug here is that in --target mode across separate runs pip doesn't treat the bin/ directory as a shared directory.

Without target mode it has no problem adding script symlinks to/from a virtualenv's bin directory without either removing the whole thing or failing to add symlinks.

I'd like to be able use --target to install multiple packages into one directory across multiple pip operations but it looks like that doesn't work right now.

[dulfox]

I have the same issue with python 3.6 and pip 20.1.1.
It's a real drag to make optimized AWS lambda layers.
Is there a workaround for this issue ?

[leos]

@dulfox my workaround for this has been to pip install --no-deps each package into its own separate directory and then manage the symlinks myself. It's not as bad as it sounds, ~20 lines of python.

[dulfox]

I'm looking for an answer for the broadest possible case : I want to bring as many dependencies on board as I can.
I use a shell script via Jenkins to generate my AWS layers with _virtualenv_ and _pip install -t /layer_dest_.
The output directory will be then zipped and loaded with terraform.
But @leos your python code may be useful for others, maybe you could publish it on a github gist...

[MumblingFumbler]

On a windows system, I define
target in a pip.ini file
Then I run
'pip install ipython'
which installs binaries in ...\site-packagesbin
then I run
'pip install cython'
I get complaints from pip about 'target directories already exist', and a recommendation to run
'pip install cython --upgrade'
when I do that, pip deletes the ....\site-packagesbin directory (including ipython binaries),
creates a new ....\site-packagesbin directory and puts cython binaries there.
This may be the expected behavior for pip, but it is not reasonable behavior.
It is not reasonable that the installation of binaries from one package would clobber
installation of binaries from another. Neither package should assume ownership of
....\site-packagesbin directory, as binaries from many packages should be able to coexist in this
directory. Why wouldn't the installation of one just add files to ....\site-packagesbin directory? Or
at least pip should have an option to do that...

[sbraz]

Hi, is there any chance to see a fix for this issue in the near future? This behaviour is rather counter-intuitive and I don't really see any easy workaround apart from installing to different targets and merging directories afterwards.