Pipenv: Double entry in the lock file

[kfirbreger]

When requiring django and djangorestframework the lock file generates double dependency for django.

[singleton11]

Not only django with drf, confirmed

[kennethreitz42]

is this with the latest versions?

[kennethreitz42]

Proper casing should be used for older versions. The latest version should use proper casing automatically.

[kennethreitz42]

re-open if this is still an issue with the latest version

[kfirbreger]

As far as I can see it is still a case, even with the latest versions. Pipfile has:
djangorestframework = "*" and django = "*".

It is indeed casing. Using Django for dependencies sloven this.
I cannot reopen the ticket.

[kfirbreger]

Also elasticsearch-dsl creates a double entry of itself.

[kennethreitz42]

Proper casing is currently required in Pipfiles. I'll work on a fix :)

Sent from my iPhone

On Jan 25, 2017, at 3:18 PM, Kfir Breger [e-mail] wrote:

Also elasticsearch-dsl creates a double entry of itself.

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub, or mute the thread.

[kennethreitz42]

this is now fixed!

[kennethreitz42]

Proper casing should always be used in Pipfiles. This tool now automatically corrects mistakes.

[singleton11]

The latest version
This bug occurs with django-extensions

[[source]]
verify_ssl = true
url = "https://pypi.python.org/simple"

[dev-packages]
ipython = "*"

[packages]
django-extensions = "*"
Django = "*"
djangorestframework = "*"
django-loginas = "*"
psycopg2 = "*"

Please, reopen issue

[nateprewitt]

Ok, I think I see what's happening. The lock file generated from your Pipfile contains both

"django-extensions": "*"
and

"django_extensions": {
    "hash": "sha256:b79abdb0c9a0f6ba61701f299c7605830c4b2a8495d31461ac401d027bab2e10",
    "version": "==1.7.6"
}

The hash and version are correct for the module "django-extensions" but the wheel filename returned from PyPI is "django_extensions-1.7.6-py2.py3-none-any.whl". I haven't peaked at the changes from yesterday yet, but I'm betting pipenv is substringing a name from the wheel. Since the wheel name doesn't match the provided package name, we end up with two entries. I'll take a closer look.

[jpadilla]

@nateprewitt that's exactly it, just reproduced it. It seems that wheel replaces - with _. Should we have proper_case() use a none .whl link? I can put something together to fix it.

Edit: Another option instead of filtering out .whl links might be to get a link that starts with package_name.

Edit: Oops nevermind, the issue seems to be in get_downloads_info().

[nateprewitt]

@jpadilla we're going to need to do that because PyPI doesn't seems to standardize naming on package links in the repo. The only place I'm seeing the name standardized is the <h1> tag on the package page. Since we're already parsing a name out, that seems like an acceptable alternative solution.

I'm going to punt the actual determination on this to @kennethreitz though, just so you're not wasting time on something.

Edit: apparently <h1> tags are processed as html in comments. Who knew?

[kennethreitz42]

So, what is the expected input and output here?

[kennethreitz42]

Also reported with django-cleanup.

[jacebrowning]

I also experienced this with pytest-django.

[kennethreitz42]

fixed!

[jpadilla]

@kennethreitz curious to know how it was fixed.

[nateprewitt]

It ended up being a few steps, @jpadilla. Apologies for not getting back to you about a PR, there ended up being some more nuance to it than expected.

Our original problem was parsing files downloaded from pip for the package name. In some cases the filename didn't exactly match the package name we wanted, which was leading to the bad Pipfile.lock entries.

The first step was fixing our proper_case function, which was returning bad data in certain cases. It relied on parsing a package's PyPI simple page for the last download link, which might not be identical to the actual package name (elasticsearch-dsl vs elasticseach_dsl). We modified the parser to instead retrieve the package name from the page header generated by PyPI itself. This now avoids issues with poorly named releases.

Once we had a way to get correct package names, we needed to map them to the files we downloaded for hashing. The problem here is we not only need to download the packages in the Pipfile, but all of their dependencies for hashing as well. Since we didn't have a list for those names, we had to parse pip's output to determine which packages we downloaded, and what their proper names were. Once we had that output, we simply kept track of the relation from a downloaded filename to its proper package name.

With the mapping, we were then able to solve the actual problem of generating the Pipfile.lock with duplicates. Since we previously hadn't known the names of the files pip was downloading (due to dependencies), we had to parse them all while hashing. This had been done the same way we'd handled links in proper_case, and was thus subject to the same flaw. We now just look up the filename before hashing, use the associated proper name, and everything seems to be grand.

Let us know if you're still encountering any issues with 3.2.8 and thanks for the interest :)

[jpadilla]

@nateprewitt bravo, thanks for the great recap!