Spack: URL's from PyPI

[JavierCVilla]

I'm trying to add my first python packages to Spack and I've realized that for some versions their tarfiles are located twice in PyPI with different urls.

Example with 4Suite-xml (both exist):

https://pypi.python.org/packages/source/4/4Suite-XML/4Suite-XML-1.0.2.tar.gz

and

https://pypi.python.org/packages/0e/ae/3c5b4fffb12be7c3a80c99475853349e1cf8477f99051921ea06fbf5e3b9/4Suite-XML-1.0.2.tar.gz

However, for others the more recent versions are only in the second type of url (IMHO, ugly to put in the package.py because of the length and numbers, I know, it's just a detail, but I would like to know why this difference between packages).

Example with lxml (only one url works):

https://pypi.python.org/packages/66/45/f11fc376f784c6f2e77ffc7a9d02374ff3ceb07ede8c56f918939409577c/lxml-3.7.2.tar.gz#md5=8dcf8d6c692b7aed9370f7462ff09935

but not in:

https://pypi.python.org/packages/source/l/lxml/lxml-3.7.2.tar.gz

whereas its older versions are:

https://pypi.python.org/packages/source/l/lxml/lxml-2.3.tar.gz 

Both 4Suite-XML-1.0.2 and lxml-2.3 are very old, so was there any change in the PyPI allocation policy or does it follow some rules that I'm missing?

[adamjstewart]

Yes! PyPI changed their download directory layout. Sometimes you can still get away with the old layout, but not always. See https://github.com/LLNL/spack/issues/1939#issuecomment-252160649 for a pretty good summary of the ways we can get around this. pypi.io is probably the most reliable solution for now. In the long term, we plan on querying PyPI directly through their API to see where to download a package from. I'm working on that in #2718, although I'm waiting on @tgamblin for a refactor of that logic as it's pretty hacky.

[citibeth]

After our discussion in #2718 I've concluded it's best for now to just provide per-version URLs. It's a little ugly and is a bit more work, but there's otherwise nothing wrong with it.

[JavierCVilla]

Thanks you both for the answer and references!