Aiohttp: Accented characters in headers are not properly handled

[haibin]

Long story short

Custom headers with accented characters are not handled properly.

Expected behaviour

é should remain é

Actual behaviour

é becomes é

Steps to reproduce

client.py

import aiohttp
import asyncio

async def fetch(session):
    async with session.get('http://localhost:5000/', headers={'header1': 'é'}) as response:
        return await response.text()

async def main():
    async with aiohttp.ClientSession() as session:
        html = await fetch(session)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

server.py

from flask import Flask, request
app = Flask(__name__)

@app.route("/")
def hello():
    print(request.headers['header1'])
    return "Hello World!"

Start the server

$ FLASK_APP=server.py flask run

Run the client

$ python client.py

The output:

 * Serving Flask app "server.py"
 * Environment: production
   WARNING: Do not use the development server in a production environment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
é
127.0.0.1 - - [20/Jun/2019 12:09:13] "GET / HTTP/1.1" 200 -
^C⏎                                                                    

Your environment

Linux
aiohttp 3.5.4
asyncio 3.4.3
Python 3.6.7

[asvetlov]

What encoding do you expect for é?
aiohttp uses utf-8

[haibin]

Hmm. I'm not too sure. I thought utf-8 would support é. Two things that I've tested

1. For aiohttp, if é is in the request body, it is fine.
2. The python requests library supports é in the header.

[asvetlov]

aiohttp uses utf-8 for headers encoding.
requests and flask operates with latin-1 (you cannot send 😀 char for example).
Both are not perfect, both don't follow the latest https://tools.ietf.org/html/rfc8187 specification.
The best you can do is avoiding non-ascii characters. Use percent-encoding or base64 instead if needed.

[haibin]

Why does é in request body works for aiohttp?

[webknjaz]

Record the request with Wireshark and look at the actual payload

[asvetlov]

Because the body encoding information is provided by Content-Type HTTP header. It is handed differently than headers content itself.