Twine: HTTP 400 "file already exists" after retry due to HTTP 500

[edmorley]

I get why this happened (other than not knowing the root cause of the HTTP 500), but it seems like in the retry case, twine could perhaps more intelligently detect the file already exists and either skip it or compare it to the current file to see if it uploaded successfully?

$ twine upload dist/*
Uploading distributions to https://pypi.python.org/pypi
Enter your password:
Uploading treeherder_client-3.0.0-py2-none-any.whl
[================================] 13899/13899 - 00:00:34
Received "500: Internal Server Error" Package upload appears to have failed.  Retry 1 of 5
Uploading treeherder_client-3.0.0-py2-none-any.whl
[================================] 13899/13899 - 00:00:00
HTTPError: 400 Client Error: A file named "treeherder_client-3.0.0-py2-none-any.whl" already exists for  treeherder-client-3.0.0. To fix problems with that file you should create a new release. for url: https://pypi.python.org/pypi

Using:
Python 2.7.11
twine version 1.7.4 (pkginfo: 1.3.2, requests: 2.10.0, setuptools: 24.0.2,
requests-toolbelt: 0.6.2, clint: 0.5.1)

[edmorley]

Although that said, seems like it's perma-HTTP500-ing:
(second upload of just the sdist this time, since the wheel was already uploaded above)

$ twine upload dist/treeherder-client-3.0.0.tar.gz
Uploading distributions to https://pypi.python.org/pypi
Enter your password:
Uploading treeherder-client-3.0.0.tar.gz
[================================] 12448/12448 - 00:00:33
Received "500: Internal Server Error" Package upload appears to have failed.  Retry 1 of 5
Uploading treeherder-client-3.0.0.tar.gz
[================================] 12448/12448 - 00:00:07
HTTPError: 400 Client Error: A file named "treeherder-client-3.0.0.tar.gz" already exists for  treeherder-client-3.0.0. To fix problems with that file you should create a new release. for url: https://pypi.python.org/pypi

[sigmavirus24]

So there's two things you can do:

1. Upload to pypi.io (which is the default pypi URL in 1.7.x)
2. Use --skip-existing which will ignore the 400 when it retries.

PyPI has been 500'ing for quite a while intermittently. It will not be fixed because Warehouse (pypi.io) is almost complete and will soon replace it.

[edmorley]

I'm using 1.7.4 already, however it turns out my .pypirc contained:

[pypi]
repository = https://pypi.python.org/pypi
username = <SNIP>

I'm pretty sure I copied that from somewhere when setting credentials previously (before password could be entered on the fly perhaps?). I had no idea it was set.

[edmorley]

eg:
https://wiki.python.org/moin/TestPyPI

Anyway, thank you for suggesting --skip-existing and helping me spot the stale .pypirc entry. I wonder how many people will hit the same - is it worth adding a warning to twine if it sees repository set to https://pypi.python.org/pypi?

[sigmavirus24]

Eh. It's plausible that it could be helpful, but it's technically not deprecated... yet. @dstufft would it be better to print something when a 5xx error occurs that they should try pypi.io in your opinion? I think we should only start warding people off as they see errors that affect their usage.

[ryneeverett]

On a slightly tangential note, are the valid repository urls listed anywhere? All I could find was "https://upload.pypi.io/legacy/" on https://packaging.python.org/distributing/ but there are obviously more.

[sigmavirus24]

On a slightly tangential note, are the valid repository urls listed anywhere? All I could find was "https://upload.pypi.io/legacy/" on https://packaging.python.org/distributing/ but there are obviously more.

@ryneeverett I'm not sure what you mean by "there are obviously more". What do you consider to be a "valid repository url"? A valid repository URL is a URL that points to a repository that behaves like Cheeseshop. If you want "valid" Cheeseshop URLs, then there are only two, https://upload.pypi.io/legacy/ and https://pypi.python.org/pypi/.

[ryneeverett]

If you want "valid" Cheeseshop URLs, then there are only two, https://upload.pypi.io/legacy/ and https://pypi.python.org/pypi/.

Thanks, that is what I meant.

[sigmavirus24]

So what made you think that there were obviously more than those two?

[ryneeverett]

There were obviously more than the _one_ that is documented.

[dstufft]

Would it be better to print something when a 5xx error occurs that they should try pypi.io in your opinion? I think we should only start warding people off as they see errors that affect their usage.

This seems like a reasonable thing to do.