Aiohttp: Client session.post cuts payload

[apanasis]

Hello.
When i try send post request via clientsession.post response is '400 Bad request'. Its because payload
{"analyticsUserProperties": {}, "appId": "1:760348033671:android:f6afd7b67eae3860", "appInstanceId": "dsEm3daQDwg", "appInstanceIdToken": "dsEm3daQDwg:APA91bH_SaG71gROGmxH7Hyp5iiHrYGGQhA60zKBPiuE6UwJtCd4SyQCVrrYCV6ZNWtAexxZXOjb9l97QMUhuxijtovprK6VM1hQ4ZBV5Bbq2LL9vOurWWLNTF18likWOs4d7arAiAhi", "appVersion": "5.8.0", "countryCode": "", "languageCode": "ru", "packageName": "org.telegram.messenger", "platformVersion": "26", "sdkVersion": "17.0.0", "timeZone": "Europe/Moscow"}
send not fully and in http request looks like
{"analyticsUserProperties": {}, "appId": "1:760348033671:android:f6afd7b67eae3860", "appInstanceId": "dsEm3daQDwg", "appInstanceIdToken": "dsEm3daQDwg:APA91bH_SaG71gROGmxH7Hyp5iiHrYGGQhA60zKBPiuE6UwJtCd4SyQCVrrYCV6ZNWtAexxZXOjb9l97QMUhuxijtovprK6VM1hQ4ZBV5Bbq2LL9vOurWWLNTF18likWOs4d7arAiAhi", "appVersion": "5.8.0", "countryCode": "", "languageCode".

If i send same request via requests library everythink is good, send whole payload and response 200 ok.

Arch Linux, Python 3.7.4, aiohttp 3.5.4, asyncio 3.4.3

[asvetlov]

Have no idea what code did you use for making POST request.

[apanasis]

async with aiohttp.ClientSession() as session: response = await asyncio.wait_for(session.post(url=url, data=payload, headers=headers, params=querystring), 3)

[asvetlov]

Use post(..., json=payload) for sending json.
http://docs.aiohttp.org/en/stable/client_quickstart.html#json-request

[apanasis]

I tried, but same result...

async with aiohttp.ClientSession(connector=pconn, request_class=ProxyClientRequest) as session: response = await asyncio.wait_for(session.post(url=url, json=jpayload, headers=headers, params=querystring, proxy=proxy, ssl=False), 3)

[apanasis]

jpayload = {"analyticsUserProperties": {}, "appId": "1:760348033671:android:f6afd7b67eae3860", "appInstanceId": "dsEm3daQDwg", "appInstanceIdToken": "dsEm3daQDwg:APA91bH_SaG71gROGmxH7Hyp5iiHrYGGQhA60zKBPiuE6UwJtCd4SyQCVrrYCV6ZNWtAexxZXOjb9l97QMUhuxijtovprK6VM1hQ4ZBV5Bbq2LL9vOurWWLNTF18likWOs4d7arAiAhi", "appVersion": "5.8.0", "countryCode": "", "languageCode": "ru", "packageName": "org.telegram.messenger", "platformVersion": "26", "sdkVersion": "17.0.0", "timeZone": "Europe/Moscow"}

[apanasis]

request: {"analyticsUserProperties": {}, "appId": "1:760348033671:android:f6afd7b67eae3860", "appInstanceId": "dsEm3daQDwg", "appInstanceIdToken": "dsEm3daQDwg:APA91bH_SaG71gROGmxH7Hyp5iiHrYGGQhA60zKBPiuE6UwJtCd4SyQCVrrYCV6ZNWtAexxZXOjb9l97QMUhuxijtovprK6VM1hQ4ZBV5Bbq2LL9vOurWWLNTF18likWOs4d7arAiAhi", "appVersion": "5.8.0", "countryCode": "", "languageCode"

[socketpair]

1. Well, does your proxy correctly understand chunked encoding ? gzipped ?
2. We definitely can not debug this without triggering the bug. The only way is possible - is to have a way to trigger.

[apanasis]

1. Yes, my proxy (mitmproxy) correctly understand chunked encoding and gzipped too.
   And this bug manifest itself if i make request without proxy too.
2. How i can help?

[socketpair]

@apanasis does the bug trigger without a proxy ? Without https ?

[apanasis]

Without proxy - yes. Without https - yes.

[socketpair]

Great. Does the bug trigger if web-server is based on aiohttp ? If yes, please provide exact pieces of the client code and the server code. I definitely want to figure out what happens

[apanasis]

About web-server: no, i dont think so, its firebase server, i cant detect what kind of it techology...
https://firebaseremoteconfig.googleapis.com/v1/projects/760348033671/namespaces/firebase:fetch?key=AIzaSyAijSIM_t8bfdRrn2-mCumXRu37pPslqkU. My client code i wrote above.
Вut i think this bug not depends from web-servers technology, if i make same request to another server, situation repeat...And if i make request with requests python library to firebase server-everythink is good and 200 ok response.

[apanasis]

I started to suspect - this bug is not in aiohttp code, but deeper...may be in asyncio

[apanasis]

I traced until procedure write in class _SelectorSocketTransport(_SelectorTransport) in file selector_events.py of asyncio library and in row 860 n = self._sock.send(data) variable data contains all requests bytes. But i cant move on...

[asvetlov]

https://www.wireshark.org can capture network traffic.
Could you use it to make sure if data is sent correctly or not?

[apanasis]

I used tcpdump. Ok, can in wireshark. Same result. Data length is 476 bytes, but sent only 350... mystic...

[asvetlov]

So, you would say that transport.write() was called with full data but the data was not transferred over the network?

Another question. You provided a snippet:

async with aiohttp.ClientSession() as session: 
    response = await asyncio.wait_for(session.post(url=url, data=payload, headers=headers, 
                                                           params=querystring), 3)

Did you even try to read response data, e.g. await response.read()?
Some server can start sending response headers before reading the whole request body.
In this case request transfer can stop on exit from async with before the whole body is sent.

[socketpair]

Also, @apanasis switch off gzip please during tests.

[apanasis]

I apologize for the time spent. This is my dumb mistake. Everything is working. Sorry.

[socketpair]

@apanasis Please tell us what happened :)

[apanasis]

Content-length in headers was 350. And no recalculate according to real length of payload.

[socketpair]

Note, according to RFC, content-length should be ignored if chunked encoding is used.

[apanasis]

But in this case not ignored...encoding chunked