Aiohttp: Switch to yarl breaks non-str params

[duedal]

After the switch to yarl we can no longer use non-str params, such as numbers.

import aiohttp
import asyncio

async def main(loop):
    url = 'http://example.com/'
    p = {'a': 'foo', 'b': 42}
    async with aiohttp.ClientSession() as session:
        async with session.get(url, params=p) as resp:
            await resp.release()

loop = asyncio.get_event_loop()
loop.run_until_complete(main(loop))

Of course one can workaround it by mapping str() to all the params, but these things should not break with a minor version upgrade IMHO.

p = {k: str(v) for k, v in p.items()}

Tested on aiohttp 1.1.1, with yarl 0.5.3

Backtrace

  File "/Users/hans/.virtualenvs/play/lib/python3.5/site-packages/aiohttp/client_reqrep.py", line 79, in __init__
    url2 = url.with_query(params)
  File "/Users/hans/.virtualenvs/play/lib/python3.5/site-packages/yarl/__init__.py", line 607, in with_query
    for k, v in query.items())
  File "/Users/hans/.virtualenvs/play/lib/python3.5/site-packages/yarl/__init__.py", line 607, in <genexpr>
    for k, v in query.items())
  File "yarl/_quoting.pyx", line 46, in yarl._quoting._quote (yarl/_quoting.c:1384)
TypeError: Argument should be str

[asvetlov]

I might say that supporting non-str types was a design error.
Hopefully it's fixed now.

[duedal]

I guess that's one way of looking at it, and if the major version was bumped to indicate an API change I might agree with you, sadly it wasn't. aiohttp just silently broke stuff (again I might add)

Yarl also broke sending non utf-8 params, ie.

p = {'a': 'foo', 'b': '42', 'c': '%F8'}

Prior to aiohttp 1.1.0 this resulted in ?b=42&a=foo&c=%25F8 now it throws

  File "/Users/hans/.virtualenvs/play/lib/python3.5/site-packages/aiohttp/client_reqrep.py", line 80, in __init__
    q.extend(url2.query)
  File "/Users/hans/.virtualenvs/play/lib/python3.5/site-packages/yarl/__init__.py", line 55, in __get__
    val = self.wrapped(inst)
  File "/Users/hans/.virtualenvs/play/lib/python3.5/site-packages/yarl/__init__.py", line 342, in query
    ret = MultiDict(parse_qsl(self.query_string, keep_blank_values=True))
  File "/Users/hans/.virtualenvs/play/lib/python3.5/site-packages/yarl/__init__.py", line 55, in __get__
    val = self.wrapped(inst)
  File "/Users/hans/.virtualenvs/play/lib/python3.5/site-packages/yarl/__init__.py", line 361, in query_string
    return unquote(self.raw_query_string, unsafe='?&=+')
  File "yarl/_quoting.pyx", line 159, in yarl._quoting._unquote (yarl/_quoting.c:2836)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf8 in position 0: invalid start byte

I don't even think there is a workaround for this?

p = {'a': 'foo', 'b': '42', 'c': '\xF8'}

gives ?b=42&a=foo&c=%C3%B8

and

p = {'a': 'foo', 'b': '42', 'c': b'\xF8'}

Don't work because everything must be str, bytes won't do.

[asvetlov]

Yes, this is the problem.
I'll fix p = {'a': 'foo', 'b': '42', 'c': '%F8'} soon.
Thank you for raising the question.
Bytes shouldn't be supported as well as int, bool, dist and list.

[duedal]

Thank you @asvetlov

btw the venerable requests library handles it just fine

import requests
url = 'http://example.com/'
p = {'a': 'foo', 'b': 42, 'c': '%F8', 'd': b'\xF8'}
requests.Session().get(url, params=p)

?a=foo&c=%25F8&b=42&d=%F8

[asvetlov]

Hmm, maybe it makes sense.
The question is: what should be a result of '%F8'.
Obviously it is not UTF-8 character.
Looks like requests rules are:

• if the value is str convert percent into %25.
• if the value is bytes convert it into percent-encoding as is.

Let me reread https://tools.ietf.org/html/rfc3986.html and https://tools.ietf.org/html/rfc3987.html first.

[khazhyk]

int seems to have a well known conversion to string, what's the reason for not supporting int?

[asvetlov]

What reason for supporting int only -- there are many numberic classes in Python?

[khazhyk]

Not saying support int only, but allowing standard string conversions seems more beneficial and standard than the contrary, and disallowing results in (evidently) unexpected behavior.

[asvetlov]

@duedal after re-thinking I pretty sure that "%F8" -> "%25F8" conversion on quoting is wrong. The right behavior should be just "%F8" -> "%F8"

[asvetlov]

After this I believe YARL should not support bytes as mapping values for .with_query(mapping).
If you want to pass non-utf8 string use

from urllib.parse import quote
url.with_query({'a': quote('текст'.encode('koi8-r'))})

[asvetlov]

@duedal please upgrade yarl to 0.6.0

[asvetlov]

@khazhyk My first reaction was: no, int should be not silently processed by yarl.URL.
Just because https://tools.ietf.org/html/rfc3986.html operates with strings, not bytes or integers.
But now I inclining to accept int as value: convenience beats purity.

P.S.
yarl 0.7.0 was released, it does accept int but doesn't it for float, bool, bytes etc.

[asvetlov]

Feel free if you want to propose bytes but please create it in https://github.com/aio-libs/yarl project.

[lock[bot]]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a [new issue] for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that [new issue].