Opentrons: bug: RPC WebSocket disconnects if protocol is > ~1 MB

[nickcrider]

Overview

A seemingly valid PD protocol will cause the app to throw a ACK Timeout error when uploaded to the robot. No errors can be observed in the robot logs.

Update:
This seems to be caused by the robot server ending the real-time connection if any given message is greater than 1MB. In this case, the protocol itself is larger than 1MB, so the upload command causes the connection to end unexpectedly

Current behavior / Steps to reproduce

1. Unzip and upload ACK_Timeout_PD.zip to the robot on 3.19.0. My robot is on wifi, have not tested on USB.
2. You'll get this error after a few seconds
   
3. No errors are thrown when looking at journalctl -f
4. The JSON protocol will simulate successfully with opentrons_simluate.

[SyntaxColoring]

I could also reproduce this running the robot server locally with make dev.

[mcous]

Upon investigation, two problems:

• The app, for some reason, will only show a "lost connection" alert if the health status of the robot in question is not ok
  
  
  
  • So, if the robot is responding on its health endpoint and the websocket goes down, the websocket failure will not be surfaced to the user
  
  
  • This is a definite app side bug
  
  
• The protocol, reproducably, causes the websocket to disconnect

So the summary of events is:

• App sends RPC call to create a protocol with the contents of this JSON file
• WebSocket goes down pretty much immediately
• App misses the fact that the WS is down, hits its 10 second command acknowledge timeout, and surfaces that error
• Robot is disconnected but user was never alerted and app still shows robot as connected

Eventually, these logs show up in the robot server:

2020-07-28 14:16:02,374 robot_server.service.legacy.rpc.rpc ERROR [Line 161] While reading from socket:
Traceback (most recent call last):
  File "./robot_server/service/legacy/rpc/rpc.py", line 156, in handle_new_connection
    msg = await socket.receive_json()
  File "/Users/mc/.local/share/virtualenvs/robot-server-tntD7pR_/lib/python3.7/site-packages/starlette/websockets.py", line 98, in receive_json
    self._raise_on_disconnect(message)
  File "/Users/mc/.local/share/virtualenvs/robot-server-tntD7pR_/lib/python3.7/site-packages/starlette/websockets.py", line 80, in _raise_on_disconnect
    raise WebSocketDisconnect(message["code"])
starlette.websockets.WebSocketDisconnect: 1006

A 1006 is a generic error code that means the connection was closed abnormally

The app's dev tools report the upload command message to be 1171956 bytes long (~1.2 MB)

[mcous]

We seem to have a problem with our WebSocket server. If any given message is > 1 MB, the socket is closed

• https://github.com/encode/uvicorn/issues/432
• https://github.com/encode/uvicorn/issues/306

This ticket will track this bug. I will file a new ticket for the app missing WebSocket disconnects when the robot is otherwise healthy