Twine: Namespace packages with dots in them fail to upload to PyPI

Created on 15 Mar 2021  路  17Comments  路  Source: pypa/twine

Your Environment

1) Operating system:

python:slim-buster

2) Python version:

3.7.8

3) twine installation method:

pip from official PyPI

4) Installed twine version:

twine-3.4.0-py3-none-any.whl

$ twine --version
twine version 3.4.0 (importlib_metadata: 3.7.3, packaging: 20.9, pkginfo:
1.7.0, requests: 2.25.1, requests-toolbelt: 0.9.1, tqdm: 4.59.0)

5) Package repository target

  • The package's PKG-INFO file
Metadata-Version: 2.1
Name: mosaik.SimConfig
Version: 0.1.0rc20210315231626
Summary: The missing implementation of mosaik's SimConfig dictionary.
Home-page: https://gitlab.com/offis.energy/mosaik/mosaik.simconfig
Author: Bengt L眉ers
Author-email: [email protected]
License: UNKNOWN
Description: # mosaik SimConfig

        The missing implementation of mosaik's SimConfig dictionary.

        [...]

Platform: UNKNOWN
Classifier: License :: OSI Approved :: GNU Lesser General Public License v2 (LGPLv2)
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: Implementation :: PyPy
Description-Content-Type: text/markdown

.pypirc file

cat: /root/.pypirc: No such file or directory

The Issue

When I run twine, I get an error:

$ twine upload --verbose dist/*
Using configuration from /root/.pypirc
Uploading distributions to https://upload.pypi.org/legacy/
  dist/mosaik.SimConfig-0.1.0rc20210315223057-py2.py3-none-any.whl (14.0 KB)
username set by command options
password set by command options
username: Bengt
password: <hidden>
Uploading mosaik.SimConfig-0.1.0rc20210315223057-py2.py3-none-any.whl
100%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅| 20.7k/20.7k [00:00<00:00, 31.8kB/s]
Content received from server:
<html>
 <head>
  <title>400 Start filename for 'mosaik-simconfig' with 'mosaik-simconfig'.</title>
 </head>
 <body>
  <h1>400 Start filename for 'mosaik-simconfig' with 'mosaik-simconfig'.</h1>
  The server could not comply with the request since it is either malformed or otherwise incorrect.<br/><br/>
Start filename for &#x27;mosaik-simconfig&#x27; with &#x27;mosaik-simconfig&#x27;.
 </body>
</html>
HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/
Start filename for 'mosaik-simconfig' with 'mosaik-simconfig'.

Source: https://gitlab.com/offis.energy/mosaik/mosaik.simconfig/-/jobs/1100239627

Additional occurrences:

https://gitlab.com/mosaik.hdf5-storage/mosaik.hdf5-storage/-/jobs/1100176050
https://gitlab.com/offis.energy/mosaik/mosaik.scenario-tools/-/jobs/1100175841
https://gitlab.com/offis.energy/mosaik/mosaik.eid/-/jobs/1100176922
https://gitlab.com/offis.energy/mosaik/mosaik.householdsim_semver/-/jobs/1100175569

Steps to Reproduce

        -   python setup.py bdist_wheel
        -   pip install --quiet --upgrade twine
        -   twine --version  # required for reporting bugs
        -   twine upload --verbose dist/*

Source: https://gitlab.com/offis.energy/mosaik/mosaik.simconfig/-/blob/master/.gitlab-ci.yml#L88

picture Bengt
👍2

Most helpful comment

The fix has been released: https://pypi.org/project/twine/3.4.1/

Thanks @pablogsal for the quick fix, and @Bengt and @godlygeek for the initial report.

picture bhrutledge on 17 Mar 2021
❤3

All 17 comments

The breaking change appears to be https://github.com/pypa/twine/commit/0bd26af11b8f97ca97ccb9859693d7a14acf15c3#diff-30ee60b48548622c54dfa266e095021f403ad23ab4848c74b2deed5df329ae00R62 

>>> import pkg_resources
>>> pkg_resources.safe_name("mosaik.SimConfig")
'mosaik.SimConfig'
>>> import packaging.utils
>>> packaging.utils.canonicalize_name("mosaik.SimConfig")
'mosaik-simconfig'
godlygeek picture godlygeek on 16 Mar 2021
👀1

Same problem here, just broke my build. Downgrading to 3.3.0 as a workaround solves the issue.

kgreav picture kgreav on 16 Mar 2021
🎉1

@jaraco Are you able to look into this?

bhrutledge picture bhrutledge on 16 Mar 2021

In the meantime, if someone could open a PR with a failing test that reproduces this (maybe in test_package.py), it would be greatly appreciated.

bhrutledge picture bhrutledge on 16 Mar 2021

Ah, I forgot that I cand easily DDoS an issue by mentioning the issue number in the commit message :)

In the meantime, if someone could open a PR with a failing test that reproduces this (maybe in test_package.py), it would be greatly appreciated.

Opened PR here: https://github.com/pypa/twine/pull/745

pablogsal picture pablogsal on 16 Mar 2021

Looks like PyPI wasn't ready for this change, since it also uses pkg_resources.safe_name: https://github.com/pypa/warehouse/blob/ea21a674b6165d61340ae976da67a08729902f6a/warehouse/forklift/legacy.py#L1193

I guess a decent plan would be to merge #745, and then start allowing both canonical / safe_name on warehouse's end point and then revert #745.

/cc @ewdurbin @di

pradyunsg picture pradyunsg on 16 Mar 2021

It's clear that packaging.utils.canonicalize_name and pkg_resources.safe_name do different things, but it's not clear to me that canonicalize_name is the "correct" way. If it is, and if we follow @pradyunsg's suggestion, would that result in changing/breaking existing project's names?

bhrutledge picture bhrutledge on 16 Mar 2021

I'm pretty sure we want to get to the point where we're using canonicalize_name everywhere -- it's backed by a PEP (503, I think?) -- but safe_name is purely implementation defined. AFAIK this won't cause any changes to the served index pages, because the names served by the index are normalised using canonicalize_name (or something equivalent).

pradyunsg picture pradyunsg on 16 Mar 2021

I agree that https://github.com/pypa/twine/commit/0bd26af11b8f97ca97ccb9859693d7a14acf15c3#diff-30ee60b48548622c54dfa266e095021f403ad23ab4848c74b2deed5df329ae00L61-R62 is an invalid change.

As a utility it seems that pkg_resources.safe_name is generating a valid distribution name per PEP 508 from an arbitrary string.

packaging.utils.canonicalize_name certainly has a different purpose per PEP 503 which is to normalize names for the Simple Index and PyPI URLS only.

The possible result of packaging.utils.canonicalize_name is a subset of valid project names, which seems to be what is causing the issue here. An otherwise valid distribution name passed through this function may be modified.

ewdurbin picture ewdurbin on 16 Mar 2021

I'm not positive warehouse's behavior is correct here either. Seems we might need a validator utility in packaging that ensures a given distribution name is valid? Currently warehouse defines the regex from PEP 508 inline at https://github.com/pypa/warehouse/blob/ea21a674b6165d61340ae976da67a08729902f6a/warehouse/forklift/legacy.py#L199-L201

ewdurbin picture ewdurbin on 16 Mar 2021

It may make sense for warehouse to use canonicalize name in the comparison noted by @pradyunsg at https://github.com/pypa/warehouse/blob/ea21a674b6165d61340ae976da67a08729902f6a/warehouse/forklift/legacy.py#L1192-L1193, however I'm concerned that this might paper over the impedance mismatch noted in https://github.com/pypa/twine/issues/743#issuecomment-800179782

ewdurbin picture ewdurbin on 16 Mar 2021

Hmm... I wonder how disruptive it would be to require the incoming names (going "into" PyPI) to be canonical as PEP 503 defines.

The benefit is that PyPI (and publishing tooling) would only use PEP 503 names (other than for validation / legacy projects) and pip (and other consumption tooling) would need to deal with PEP 508 stuff.

pradyunsg picture pradyunsg on 16 Mar 2021

That would certainly break valid distribution names like namespace.foobar. I don't think it's a great idea. Remember, 503 is about the Simple Index specification, not metadata.

ewdurbin picture ewdurbin on 16 Mar 2021
👍3

I guess one final thought here specifically is that what we're calling namespaces here should probably be spoken as "pseudonamespaces" since there is really no implication to them in the project name currently.

Actual namespace support is something that has been discussed and is likely to see fruition in the near future.

ewdurbin picture ewdurbin on 16 Mar 2021
👍1

The fix has been released: https://pypi.org/project/twine/3.4.1/

Thanks @pablogsal for the quick fix, and @Bengt and @godlygeek for the initial report.

bhrutledge picture bhrutledge on 17 Mar 2021
❤3

I can confirm that 3.4.1 fixes all issues I was having. Thanks for all the discussion and the quick fix, everybody.

Bengt picture Bengt on 17 Mar 2021

@Bengt thanks for the excellent report that made it easy to track down and fix this

sigmavirus24 picture sigmavirus24 on 17 Mar 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

bhrutledge picture bhrutledge  路  4Comments
cs01 picture cs01  路  5Comments
jaraco picture jaraco  路  7Comments
techtonik picture techtonik  路  9Comments
anentropic picture anentropic  路  8Comments