Warehouse: Special characters in metadata fields cause Gunicorn to 500

[codingjoe]

Describe the bug
I want to publish a new package (new package, not only new version).
Twine comes back with a 500 server error.

Expected behavior
Package should be registered and uploaded.

To Reproduce

$ twine upload dist/*
Uploading distributions to https://upload.pypi.org/legacy/
Uploading django_dynamic_filenames-1.0.0-py3-none-any.whl
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10.3k/10.3k [00:01<00:00, 7.11kB/s]
Received "500: Internal Server Error" Package upload appears to have failed.  Retry 1 of 5
Uploading django_dynamic_filenames-1.0.0-py3-none-any.whl
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10.3k/10.3k [00:00<00:00, 26.4kB/s]
Received "500: Internal Server Error" Package upload appears to have failed.  Retry 2 of 5
Uploading django_dynamic_filenames-1.0.0-py3-none-any.whl
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10.3k/10.3k [00:00<00:00, 20.8kB/s]
Received "500: Internal Server Error" Package upload appears to have failed.  Retry 3 of 5
Uploading django_dynamic_filenames-1.0.0-py3-none-any.whl
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10.3k/10.3k [00:00<00:00, 22.4kB/s]
Received "500: Internal Server Error" Package upload appears to have failed.  Retry 4 of 5
Uploading django_dynamic_filenames-1.0.0-py3-none-any.whl
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10.3k/10.3k [00:01<00:00, 9.32kB/s]
Received "500: Internal Server Error" Package upload appears to have failed.  Retry 5 of 5
HTTPError: 500 Server Error: Internal Server Error for url: https://upload.pypi.org/legacy/
(env) MONDO-0262:django-dynamic-filenames johannes$ twine --version
twine version 1.11.0 (pkginfo: 1.4.2, requests: 2.19.1, setuptools: 40.2.0,
requests-toolbelt: 0.8.0, tqdm: 4.25.0)

See also: https://travis-ci.com/codingjoe/django-dynamic-filenames/jobs/144208217

My Platform
Both locally and on travis-ci

$ twine --version
twine version 1.11.0 (pkginfo: 1.4.2, requests: 2.19.1, setuptools: 40.2.0,
requests-toolbelt: 0.8.0, tqdm: 4.25.0)

https://travis-ci.com/codingjoe/django-dynamic-filenames/jobs/144208217

Additional context

That's the package that I am trying to release:
https://github.com/codingjoe/django-dynamic-filenames/tree/1.0.0

[codingjoe]

Update I fixed it now, but still the server error should not happen. it should return a 400 with a proper message. I will leave the bug open. The fix is here
https://github.com/codingjoe/django-dynamic-filenames/commit/f3284fdfa4a257b3a749c3fd36b74bce2deaba1d

[di]

I can reproduce this, the problem is that your original license line had an extra character at the end that our HTTP server couldn't handle:

>>> open('setup.cfg').readlines()[12]
'    License :: OSI Approved :: MIT License±\n'

I'll update the title accordingly.

[di]

The tracktrace from Warehouse:

[2018-09-07 15:25:37 +0000] [508] [ERROR] Error handling request /legacy/
Traceback (most recent call last):
  File "/opt/warehouse/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 135, in handle
    self.handle_request(listener, req, client, addr)
  File "/opt/warehouse/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 182, in handle_request
    resp.write(item)
  File "/opt/warehouse/lib/python3.6/site-packages/gunicorn/http/wsgi.py", line 333, in write
    self.send_headers()
  File "/opt/warehouse/lib/python3.6/site-packages/gunicorn/http/wsgi.py", line 329, in send_headers
    util.write(self.sock, util.to_bytestring(header_str, "ascii"))
  File "/opt/warehouse/lib/python3.6/site-packages/gunicorn/util.py", line 507, in to_bytestring
    return value.encode(encoding)
UnicodeEncodeError: 'ascii' codec can't encode character '\xb1' in position 90: ordinal not in range(128)

[tirkarthi]

Related upstream issue : https://github.com/benoitc/gunicorn/issues/1214 ?

Thanks

[di]

I just tested Warehouse against https://github.com/benoitc/gunicorn/commit/dc7b5d5c4876b49f86ea2460698a335d0f5ef7c9 (the latest commit for Gunicorn 20.x) and this is fixed (specifically by this commit: https://github.com/benoitc/gunicorn/commit/879651bb6f873631725cc31cbda783349fdf9dc3).

This change is not currently backported to the 19.x branch, however I've requested that it gets backported as it's quite a small change, and it looks like it might be a while before 20.x gets released.

[hiromipaw]

I might be experiencing this issue. I am currently receiving the following from the server:

Content received from server:
<html>
  <head>
    <title>Internal Server Error</title>
  </head>
  <body>
    <h1><p>Internal Server Error</p></h1>

  </body>
</html>

[di]

@hiromipaw This should be resolved with #7112, if you're still experiencing it after that PR is merged, please comment with more details.