pip freeze does case sensitive comparsion

[abs51295]

Environment

• pip version: 18.0
• Python version: 3.6
• OS: Mac OS X

Description

Running pip freeze -l -r requirements.txt does a case sensitive comparison with dependencies mentioned in requirements.txt.

Expected behavior
Running pip freeze -l -r requirements.txt should do case insensitive comparison with dependencies mentioned in requirements.txt or at least allow that via a command line option.

How to Reproduce

1. Create a virtualenv.
2. Create a requirements.txt file.
3. Run pip install -r requirements.txt.
4. Run pip freeze -l -r requirements.txt.

Output

#
# This file is autogenerated by pip-compile
# To update, run:
#    pip-compile --output-file requirements.txt requirements.in
asn1crypto==0.24.0
attrs==18.1.0
certifi==2018.4.16
cffi==1.11.5
chardet==3.0.4
click==6.7
colorama==0.3.9
coverage==4.5.1
cryptography==2.2.2
flake8-polyfill==1.0.2
flake8==3.5.0
Requirement file [requirements.txt] contains flask-cors==3.0.4, but that package is not installed
Requirement file [requirements.txt] contains flask==1.0.2, but that package is not installed
gevent==1.3.1
greenlet==0.4.13
gunicorn==19.8.1
idna==2.6
itsdangerous==0.24
Requirement file [requirements.txt] contains jinja2==2.10, but that package is not installed
mando==0.6.4
Requirement file [requirements.txt] contains markupsafe==1.0, but that package is not installed
mccabe==0.6.1
more-itertools==4.1.0
pluggy==0.6.0
psycopg2==2.7.4
py==1.5.3
pycodestyle==2.3.1
pycparser==2.18
pycrypto==2.6.1
pyflakes==1.6.0
Requirement file [requirements.txt] contains pyjwt==1.6.3, but that package is not installed
pytest-cov==2.5.1
pytest-mock==1.10.0
pytest==3.5.1
radon==2.2.0
requests-futures==0.9.7
requests==2.18.4
semantic-version==2.6.0
six==1.11.0
Requirement file [requirements.txt] contains sqlalchemy==1.2.7, but that package is not installed
urllib3==1.22
Requirement file [requirements.txt] contains werkzeug==0.14.1, but that package is not installed
## The following requirements were added by pip freeze:
Flask==1.0.2
Flask-Cors==3.0.4
Jinja2==2.10
MarkupSafe==1.0
PyJWT==1.6.3
SQLAlchemy==1.2.7
Werkzeug==0.14.1

[abs51295]

@cjerdonek I will like to work on this issue. Let me know, when we have classified this as a bug or a feature request.

[pradyunsg]

Hey @abs51295!

I'd consider this a bug since this is essentially a case where pip is not normalizing the given package names correctly.

[abs51295]

Thanks @pradyunsg I will send a fix soon! 😄

[abs51295]

@pradyunsg how should the fix look like? Should we change the dependency names to lower case explicitly?

[pradyunsg]

There's a normalize_name or a canonical_name (or similarly named) function in pip's codebase, which should be used to normalize names.

(sorry I missed you on IRC - I slept off with the client open)

[abs51295]

Thanks @pradyunsg will look into this.

( I don't see you on #pypa-dev anymore on IRC)

[abs51295]

@pradyunsg I found this: https://github.com/pypa/pip/blob/master/src/pip/_vendor/packaging/requirements.py#L86-L89. Looks like this requires a discussion.

[pradyunsg]

Hey @abs51295! I was away from a PC for most of the time over the weekend.

I've not looked carefully but you'd want to start looking here: https://github.com/pypa/pip/blob/b13cee6388adea5ddaf1ddb982beeec37d07ffda/src/pip/_internal/commands/freeze.py

Basically, whenever pip's getting information from the user/distlib/pkg_resources, you'd want to normalize the name using pip._vendor.packaging.utils.canonicalize_name.

I found this: /src/pip/_vendor/packaging/requirements.py@master#L86-L89.

That's a vendored library (packaging) and while I do think _that_ change needs discussion, the change in pip doesn't and they're fairly separate.

[abs51295]

Basically, whenever pip's getting information from the user/distlib/pkg_resources, you'd want to normalize the name using pip._vendor.packaging.utils.canonicalize_name

@pradyunsg I did that, but why I was referring to the https://github.com/pypa/pip/blob/master/src/pip/_vendor/packaging/requirements.py#L86-L89, is because of the following:

1. Canonicalizing the name when pip's getting the information, which can be achieved by canonicalizing the name here, solves only half the problem.

2. The other half of the problem lies when the requirement line is parsed from requirements.txt file and compared with existing installations as obtained in step 1.

3. During the above comparsion, if we don't canonicalize the name obtained from the requirements.txt file, then we will again run into the same problem of case sensitive comparison. To mitigate that, we need to canonicalize the name of this requirement which leads to this and hence the comment.

[abs51295]

@pradyunsg any updates on this?

[pradyunsg]

Sorry, I haven't taken a deeper look yet. Real life has kept me more than busy lately.

I'll try to take a look today or this weekend. Feel free to ping me again in that time.

[abs51295]

Thanks. Will wait for your reply 🙂

[abs51295]

@pradyunsg any updates?

[chetabahana]

any updates? hope it badly to regenerate requirements.txt
all the warned packages were actually been added by pip already

• pip version: 19.1 (python 3.7)
• existing packages: this requirements.txt
• new package added: django-storages[google]

(virtual-env)$ pip --version
pip 19.1 from /env/lib/python3.7/site-packages/pip (python 3.7)

(virtual-env)$ pip install -r requirements.txt
... done
Successfully installed all

(virtual-env)$ pip install django-storages[google]
... done
Successfully installed 
cachetools-3.1.0 google-api-core-1.9.0 google-auth-1.6.3 google-cloud-core-0.29.1 
google-cloud-storage-1.15.0 google-resumable-media-0.3.2 googleapis-common-protos-1.5.9 
protobuf-3.7.1 pyasn1-0.4.5 pyasn1-modules-0.2.5 rsa-4.0

(virtual-env)$ pip freeze -r requirements.txt
-i https://pypi.python.org/simple
amqp==2.4.2
aniso8601==3.0.2
autopep8==1.4.4
WARNING: Requirement file [requirements.txt] contains babel==2.6.0, but package 'babel' is not installed
billiard==3.6.0.0
bleach==2.1.4
boto3==1.9.138
botocore==1.12.138
braintree==3.49.0
cairocffi==1.0.2
WARNING: Requirement file [requirements.txt] contains cairosvg==2.3.1, but package 'cairosvg' is not installed
celery==4.3.0
certifi==2019.3.9
cffi==1.12.3
chardet==3.0.4
cssselect2==0.2.1
defusedxml==0.6.0
dj-database-url==0.5.0
dj-email-url==0.2.0
django-appconf==1.0.3
django-babel==0.6.2
django-bootstrap4==0.0.8
django-cache-url==3.0.0
django-celery-results==1.0.4
django-countries==5.3.3
django-debug-toolbar-request-history==0.0.10
django-debug-toolbar==1.11
django-elasticsearch-dsl==0.5.1
django-filter==2.1.0
django-graphql-jwt==0.2.1
django-impersonate==1.4
django-js-asset==1.2.2
django-measurement==3.0.0
django-mptt==0.10.0
django-phonenumber-field==2.3.1
django-prices-openexchangerates==1.0.1
django-prices-vatlayer==1.0.2
django-prices==1.0.4
django-recaptcha==1.5.0
django-redis==4.10.0
django-render-block==0.5
django-silk==2.0.0
django-storages==1.7.1
django-templated-email==2.3.0
django-versatileimagefield==1.10
django-webpack-loader==0.6.0
WARNING: Requirement file [requirements.txt] contains django==2.2, but package 'django' is not installed
docutils==0.14
elasticsearch-dsl==6.0.1
elasticsearch==6.3.1
WARNING: Requirement file [requirements.txt] contains faker==1.0.5, but package 'faker' is not installed
freezegun==0.3.11
google-i18n-address==2.3.4
google-measurement-protocol==1.0.0
gprof2dot==2016.10.13
graphene-django-optimizer==0.4.0
graphene-django==2.2.0
graphene==2.1.3
graphql-core==2.1
graphql-relay==0.4.5
gunicorn==19.9.0
html5lib==1.0.1
idna==2.8
ipaddress==1.0.22
WARNING: Requirement file [requirements.txt] contains jinja2==2.10.1, but package 'jinja2' is not installed
jmespath==0.9.4
jsonfield==2.0.2
kombu==4.5.0
WARNING: Requirement file [requirements.txt] contains markdown==3.1, but package 'markdown' is not installed
WARNING: Requirement file [requirements.txt] contains markupsafe==1.1.1, but package 'markupsafe' is not installed
maxminddb-geolite2==2018.703
maxminddb==1.4.1
measurement==2.0.1
mpmath==1.1.0
oauthlib==3.0.1
phonenumberslite==8.10.10
WARNING: Requirement file [requirements.txt] contains pillow==5.4.1, but package 'pillow' is not installed
prices==1.0.0
promise==2.2.1
psycopg2-binary==2.8.2
purl==1.5
pycodestyle==2.5.0
pycparser==2.19
WARNING: Requirement file [requirements.txt] contains pygments==2.3.1, but package 'pygments' is not installed
WARNING: Requirement file [requirements.txt] contains pyjwt==1.7.1, but package 'pyjwt' is not installed
WARNING: Requirement file [requirements.txt] contains pyphen==0.9.5, but package 'pyphen' is not installed
python-dateutil==2.8.0
python3-openid==3.1.0
pytz==2019.1
raven==6.9.0
razorpay==1.2.0
redis==3.2.1
requests-oauthlib==1.2.0
requests==2.21.0
WARNING: Requirement file [requirements.txt] contains rx==1.6.1, but package 'rx' is not installed
s3transfer==0.2.0
singledispatch==3.4.0.3
six==1.12.0
social-auth-app-django==3.1.0
social-auth-core==3.1.0
sqlparse==0.3.0
stripe==2.27.0
sympy==1.4
text-unidecode==1.2
tinycss2==1.0.2
typing==3.6.6
urllib3==1.24.2
WARNING: Requirement file [requirements.txt] contains uwsgi==2.0.18 ; platform_system != 'Windows', but package 'uwsgi' is not installed
vine==1.3.0
WARNING: Requirement file [requirements.txt] contains weasyprint==47, but package 'weasyprint' is not installed
webencodings==0.5.1

The following requirements were added by pip freeze:

astroid==2.2.5
Babel==2.6.0
cachetools==3.1.0
CairoSVG==2.3.1
configparser==3.7.4
Django==2.2
django-fsm==2.6.0
Faker==1.0.5
google-api-core==1.9.0
google-auth==1.6.3
google-cloud-core==0.29.1
google-cloud-storage==1.15.0
google-resumable-media==0.3.2
googleapis-common-protos==1.5.9
isort==4.3.17
Jinja2==2.10.1
lazy-object-proxy==1.3.1
Markdown==3.1
MarkupSafe==1.1.1
mccabe==0.6.1
Pillow==5.4.1
protobuf==3.7.1
pyasn1==0.4.5
pyasn1-modules==0.2.5
pyflakes==2.1.1
Pygments==2.3.1
PyJWT==1.7.1
pylint==2.3.1
Pyphen==0.9.5
rsa==4.0
Rx==1.6.1
typed-ast==1.3.5
uWSGI==2.0.18
WeasyPrint==47
wrapt==1.11.1

[abs51295]

@pradyunsg @cjerdonek I have created #6857 in respect to this. Please have a look. Thanks.