Pip: New feature suggested in "pip list": option to differentiate packages installed by pip and conda.
I am using Anaconda, some packages installed by conda, but some other avalable only from "pip install". I have problem with packages updates, since Anaconda packages should be updated with conda and pip packages should be updated respectively with pip (normally).
Updating of conda packages can be done properly using "conda update --all", but "pip list outdated" reports all outdated packages, including just updated conda ones. Problem is that some packages can be installed in both ways and ones from pip are oftenly newer, since Anaconda doing prpagation with some delay. Normally it is better to use slightly older conda-installed packages with conda, but pip-only packages have to be updated and I want to see who did install package originally.
There is optional column "Installer" in "pip list format=columns -verbose" with pip/conda values inside, but experiments shows this information is useless for me. It may just indicate that package
uses metadata in "*dist-info" folder format and it is true for many conda-installed packages.
Solution: I need additional column with additional information.
My new suggested output format can be enabled using new option "format=columnsx".
Pay attention to new column "Owner" with "Anaconda"/"ThePip" values.
You can also see that column "Installer" is useless, while new "Owner" is more useful...
Act: d:\InstSoft\Python\Conda3x64\Scripts\activate.bat
Run: d:\InstSoft\Python\Conda3x64\Python.exe
Arg: -m pip list --format=columnsx
Package Version Installer Owner
------------------------------------ ------------------- --------- --------
adal 1.2.4 pip ThePip
alabaster 0.7.12 pip Anaconda
anaconda-client 1.7.2 Anaconda
anaconda-navigator 1.9.12 Anaconda
anaconda-project 0.8.3 conda Anaconda
ansiwrap 0.8.4 pip ThePip
appdirs 1.4.3 Anaconda
applicationinsights 0.11.9 pip ThePip
argh 0.26.2 Anaconda
asn1crypto 1.3.0 pip Anaconda
astroid 2.4.1 pip Anaconda
astropy 4.0.1.post1 Anaconda
async-generator 1.10 pip ThePip
atomicwrites 1.4.0 conda Anaconda
attrs 19.3.0 conda Anaconda
augment-nd 0.1.1 pip ThePip
autopep8 1.4.4 pip Anaconda
azure-common 1.1.25 pip ThePip
azure-core 1.6.0 pip ThePip
azure-graphrbac 0.61.1 pip ThePip
azure-identity 1.3.1 pip ThePip
azure-mgmt-authorization 0.60.0 pip ThePip
azure-mgmt-containerregistry 2.8.0 pip ThePip
azure-mgmt-keyvault 2.2.0 pip ThePip
azure-mgmt-network 10.2.0 pip ThePip
azure-mgmt-resource 10.0.0 pip ThePip
azure-mgmt-storage 10.0.0 pip ThePip
azureml 0.2.7 pip ThePip
azureml-automl-core 1.6.0 pip ThePip
azureml-automl-runtime 1.6.0.post1 pip ThePip
azureml-contrib-notebook 1.6.0 pip ThePip
azureml-core 1.6.0.post1 pip ThePip
azureml-dataprep 1.7.0 pip ThePip
azureml-dataprep-native 14.2.0 pip ThePip
azureml-defaults 1.6.0 pip ThePip
azureml-explain-model 1.6.0 pip ThePip
azureml-interpret 1.6.0 pip ThePip
azureml-model-management-sdk 1.0.1b6.post1 pip ThePip
azureml-pipeline 1.6.0 pip ThePip
azureml-pipeline-core 1.6.0 pip ThePip
azureml-pipeline-steps 1.6.0 pip ThePip
azureml-sdk 1.6.0 pip ThePip
azureml-telemetry 1.6.0 pip ThePip
azureml-train 1.6.0 pip ThePip
azureml-train-automl 1.6.0 pip ThePip
azureml-train-automl-client 1.6.0.post1 pip ThePip
azureml-train-automl-runtime 1.6.0 pip ThePip
azureml-train-core 1.6.0.post1 pip ThePip
azureml-train-restclients-hyperdrive 1.6.0 pip ThePip
azureml-widgets 1.6.0 pip ThePip
Babel 2.8.0 conda Anaconda
backcall 0.1.0 pip Anaconda
backports.functools-lru-cache 1.6.1 conda Anaconda
backports.shutil-get-terminal-size 1.0.0 pip Anaconda
backports.tempfile 1.0 pip Anaconda
backports.weakref 1.0.post1 Anaconda
bcrypt 3.1.7 conda Anaconda
beautifulsoup4 4.9.1 pip Anaconda
bitarray 1.2.1 pip Anaconda
bkcharts 0.2 Anaconda
black 19.10b0 conda Anaconda
bleach 3.1.4 conda Anaconda
blinker 1.4 Anaconda
bokeh 2.0.2 pip Anaconda
boto 2.49.0 Anaconda
boto3 1.9.66 Anaconda
botocore 1.12.189 conda Anaconda
Bottleneck 1.3.2 pip Anaconda
cachetools 3.1.1 conda Anaconda
certifi 2020.4.5.1 Anaconda
cffi 1.14.0 pip Anaconda
chardet 3.0.4 conda Anaconda
click 7.1.2 conda Anaconda
cloudpickle 1.4.1 conda Anaconda
clyent 1.2.2 Anaconda
colorama 0.4.3 conda Anaconda
comtypes 1.1.7 Anaconda
conda 4.8.3 Anaconda
conda-build 3.18.11 Anaconda
conda-package-handling 1.7.0 conda Anaconda
conda-verify 3.4.2 conda Anaconda
configparser 5.0.0 conda Anaconda
contextlib2 0.6.0.post1 conda Anaconda
coverage 5.0 Anaconda
cryptography 2.9.2 pip Anaconda
cycler 0.10.0 Anaconda
Cython 0.29.17 pip Anaconda
cytoolz 0.10.1 Anaconda
dask 2.17.2 conda Anaconda
decorator 4.4.2 conda Anaconda
defusedxml 0.6.0 Anaconda
diff-match-patch 20181111 conda Anaconda
dill 0.3.1.1 conda Anaconda
distributed 2.17.0 pip Anaconda
distro 1.5.0 pip ThePip
docker 4.2.1 pip ThePip
docutils 0.16 pip Anaconda
dotnetcore2 2.1.14 pip ThePip
entrypoints 0.3 Anaconda
ephem 3.7.7.1 pip ThePip
et-xmlfile 1.0.1 Anaconda
fastcache 1.1.0 pip Anaconda
ffmpeg 1.4 pip ThePip
filelock 3.0.12 pip Anaconda
fire 0.3.1 pip ThePip
flake8 3.7.9 conda Anaconda
Flask 1.1.2 pip Anaconda
fsspec 0.7.4 conda Anaconda
fusepy 3.0.1 pip ThePip
future 0.18.2 conda Anaconda
gensim 3.8.0 pip Anaconda
This new option is most useful with "pip list --outdated". In the example below
it shows: all outdated items are from conda and pip update is not needed.
Act: d:\InstSoft\Python\Conda3x64\condabin\conda.bat activate
Run: d:\InstSoft\Python\Conda3x64\Python.exe
Arg: -m pip list --outdated --format=columnsx
Package Version Latest Type Installer Owner
------------------------ ----------------- ---------- ----- --------- --------
appdirs 1.4.3 1.4.4 wheel Anaconda
astroid 2.4.1 2.4.2 wheel pip Anaconda
autopep8 1.4.4 1.5.3 sdist pip Anaconda
bitarray 1.2.1 1.2.2 sdist pip Anaconda
bleach 3.1.4 3.1.5 wheel conda Anaconda
boto3 1.9.66 1.13.24 wheel Anaconda
botocore 1.12.189 1.16.24 wheel conda Anaconda
cachetools 3.1.1 4.1.0 wheel conda Anaconda
certifi 2020.4.5.1 2020.4.5.2 wheel Anaconda
coverage 5.0 5.1 wheel Anaconda
Cython 0.29.17 0.29.19 wheel pip Anaconda
dask 2.17.2 2.18.0 wheel conda Anaconda
distributed 2.17.0 2.18.0 wheel pip Anaconda
flake8 3.7.9 3.8.2 wheel conda Anaconda
gensim 3.8.0 3.8.3 wheel pip Anaconda
gevent 1.4.0 20.6.0 wheel Anaconda
google-api-core 1.17.0 1.19.1 wheel pip Anaconda
google-auth 1.14.1 1.16.1 wheel conda Anaconda
google-cloud-storage 1.28.0 1.28.1 wheel conda Anaconda
google-resumable-media 0.5.0 0.5.1 wheel conda Anaconda
googleapis-common-protos 1.51.0 1.52.0 wheel pip Anaconda
greenlet 0.4.15 0.4.16 wheel pip Anaconda
hypothesis 5.11.0 5.16.0 wheel conda Anaconda
importlib-metadata 1.6.0 1.6.1 wheel pip Anaconda
ipykernel 5.1.4 5.3.0 wheel pip Anaconda
ipython 7.13.0 7.15.0 wheel pip Anaconda
jedi 0.15.2 0.17.0 wheel pip Anaconda
jmespath 0.9.4 0.10.0 wheel pip Anaconda
json5 0.9.4 0.9.5 wheel conda Anaconda
jupyterlab 1.2.6 2.1.4 wheel Anaconda
jupyterlab-server 1.1.4 1.1.5 wheel conda Anaconda
keyring 21.1.1 21.2.1 wheel pip Anaconda
lazy-object-proxy 1.4.3 1.5.0 wheel conda Anaconda
lightgbm 2.3.0 2.3.1 wheel pip Anaconda
lxml 4.5.0 4.5.1 wheel pip Anaconda
matplotlib 3.1.3 3.2.1 wheel pip Anaconda
nltk 3.4.5 3.5 sdist pip Anaconda
numpy 1.18.1 1.18.5 wheel Anaconda
numpydoc 0.9.2 1.0.0 wheel conda Anaconda
packaging 20.3 20.4 wheel conda Anaconda
pandas 1.0.3 1.0.4 wheel pip Anaconda
parso 0.5.2 0.7.0 wheel conda Anaconda
path 13.1.0 14.0.1 wheel pip Anaconda
pathspec 0.7.0 0.8.0 wheel conda Anaconda
pip 20.0.2 20.1.1 wheel Anaconda
portalocker 1.5.2 1.7.0 wheel pip Anaconda
prometheus-client 0.7.1 0.8.0 wheel conda Anaconda
protobuf 3.11.4 3.12.2 wheel Anaconda
pyarrow 0.11.1 0.17.1 wheel Anaconda
pyasn1-modules 0.2.7 0.2.8 wheel Anaconda
pycodestyle 2.5.0 2.6.0 wheel Anaconda
pydocstyle 4.0.1 5.0.2 wheel conda Anaconda
pyflakes 2.1.1 2.2.0 wheel pip Anaconda
pylint 2.5.2 2.5.3 wheel pip Anaconda
PyNaCl 1.3.0 1.4.0 wheel Anaconda
pyodbc 4.0.0-unsupported 4.0.30 wheel pip Anaconda
pytest 5.4.2 5.4.3 wheel Anaconda
pytest-cov 2.8.1 2.9.0 wheel conda Anaconda
pytest-doctestplus 0.5.0 0.7.0 wheel conda Anaconda
python-language-server 0.31.10 0.33.1 wheel conda Anaconda
pyzmq 18.1.1 19.0.1 wheel Anaconda
QtAwesome 0.7.0 0.7.2 wheel conda Anaconda
regex 2020.5.14 2020.6.8 wheel pip Anaconda
ruamel-yaml 0.15.87 0.16.10 wheel Anaconda
s3transfer 0.1.13 0.3.3 wheel Anaconda
scikit-image 0.16.2 0.17.2 wheel pip Anaconda
scikit-learn 0.22.1 0.23.1 wheel pip Anaconda
sortedcollections 1.1.2 1.2.1 wheel pip Anaconda
sortedcontainers 2.1.0 2.2.2 wheel Anaconda
sphinxcontrib-websupport 1.2.1 1.2.2 wheel conda Anaconda
sympy 1.5.1 1.6 wheel pip Anaconda
tenacity 6.0.0 6.2.0 wheel pip Anaconda
tifffile 0.15.1 2020.6.3 wheel Anaconda
toml 0.10.0 0.10.1 wheel pip Anaconda
tqdm 4.46.0 4.46.1 wheel conda Anaconda
typing-extensions 3.7.4.1 3.7.4.2 wheel pip Anaconda
ujson 1.35 3.0.0 wheel Anaconda
urllib3 1.25.8 1.25.9 wheel pip Anaconda
wcwidth 0.1.9 0.2.4 wheel conda Anaconda
wrapt 1.11.2 1.12.1 sdist pip Anaconda
XlsxWriter 1.2.8 1.2.9 wheel conda Anaconda
yapf 0.28.0 0.30.0 wheel conda Anaconda
zope.interface 4.7.1 5.1.0 wheel pip Anaconda
I have Win-X64, conda version : 4.8.3, conda-build version : 3.18.11, python version : 3.7.7.final.0.
I did modify my local pip 20.0.2, everything works good for me, see above. We can discuss these changes more if it will be considered as new useful feature for "pip list". Old behavior completely unchanged, new feature available only with new option.
vx1920
All 5 comments
pip already shows the installer information in the verbose output (pip list -v). It should satisfy your need if Conda provides that information (and you鈥檒l need to ask Conda to provide the info if it doesn鈥檛 already do).
uranusjr
on 9 Jun 2020
I already wrote that "pip list -v" provide additional column "Installer" with irrelevant information. If you don't believe - open first Details marker in my original post and review first two lines. In my system I have package "adal" installed by pip, while next package "alabaster" installed by conda.
Your "pip list -v" marks both of them as "pip". With my new column I can see "alabaster" marked as "Anaconda", while "adal" marked "ThePip". I decide to combine both columns (old "Installer" and my new "Owner") to illustrate that "Installer" column contains something useless.
vx1920
on 9 Jun 2020
It鈥檚 a conda bug if packages it installs do not report the installer correctly. I don鈥檛 think pip should be responsible for fixing things other tools do wrong.
uranusjr
on 9 Jun 2020
I am not sure that it is conda bug, but I am sure that conda developers will tell you and me that there is no any bug in conda and conda itself can provide ability to update conda-only packages ("conda update --all"). Also "installer=pip" may legitimately indicate that conda uses "setuptools" (it is general tool used both by pip and conda) to install some particular package. Some other conda tools may set marker as "conda".
Conda has tracking items installed by conda and pip should do the same. If I use "pip list --outdated" - it should inform about outdated packages installed by pip and don't care about items installed by conda.
My option solves this problem somehow by adding extra marker column. Mybe it is better to have special option "pip list --outdatedpip", but with my option I could do some system cleanup by uninstalling pip packages and installing same name conda ones.
Maybe somebody else you can consider my option as useful. Any case: I have it now, I use now and I solve some of my problems.
vx1920
on 9 Jun 2020
I concur with @uranusjr - pip reports the value in the INSTALLER metadata file. If conda populates that fire correctly, pip will report as you wish. If conda don't do so, they are not respecting the existing interoperability standard, and it's not down to pip to work around that problem.
pfmoore
on 9 Jun 2020
Related issues
dmfigol
路
3Comments
reynoldsnlp
路
3Comments
jiapei100
路
3Comments
cjerdonek
路
3Comments
kkom
路
3Comments
Most helpful comment
It鈥檚 a conda bug if packages it installs do not report the installer correctly. I don鈥檛 think pip should be responsible for fixing things other tools do wrong.