Web3.py: Erorr in websockets.py: '<=' not supported between instances of 'int' and 'NoneType'

[davidahmed]

• web3 (4.3.0)
• websockets (4.0.1)
• Python: 3.6
• OS: osx HighSierra

What was wrong?

web3 = Web3(Web3.WebsocketProvider("ws://10.224.12.6:8546"))
web3.eth.syncing //returns data

The websocket is clearly open but when I run a filter which is supposed to have many entries, I get the following error trace:

Upon running: data = web3.eth.getFilterLogs(new_block_filter.filter_id), I get:

~/Desktop/contracts-py/contracts/lib/python3.6/site-packages/web3/providers/websocket.py in make_request(self, method, params)
     81             WebsocketProvider._loop
     82         )
---> 83         return future.result()

/anaconda3/lib/python3.6/concurrent/futures/_base.py in result(self, timeout)
    430                 raise CancelledError()
    431             elif self._state == FINISHED:
--> 432                 return self.__get_result()
    433             else:
    434                 raise TimeoutError()

/anaconda3/lib/python3.6/concurrent/futures/_base.py in __get_result(self)
    382     def __get_result(self):
    383         if self._exception:
--> 384             raise self._exception
    385         else:
    386             return self._result

~/Desktop/contracts-py/contracts/lib/python3.6/site-packages/web3/providers/websocket.py in coro_make_request(self, request_data)
     71         async with self.conn as conn:
     72             await conn.send(request_data)
---> 73             return json.loads(await conn.recv())
     74 
     75     def make_request(self, method, params):

~/Desktop/contracts-py/contracts/lib/python3.6/site-packages/websockets/protocol.py in recv(self)
    321             next_message.cancel()
    322             if not self.legacy_recv:
--> 323                 raise ConnectionClosed(self.close_code, self.close_reason)
    324 
    325     @asyncio.coroutine

~/Desktop/contracts-py/contracts/lib/python3.6/site-packages/websockets/exceptions.py in __init__(self, code, reason)
    145         self.reason = reason
    146         message = "WebSocket connection is closed: "
--> 147         if 3000 <= code < 4000:
    148             explanation = "registered"
    149         elif 4000 <= code < 5000:

TypeError: '<=' not supported between instances of 'int' and 'NoneType'

The same filter runs fine (albeit a bit slow) using Web3.HTTPProvider()

[davidahmed]

I have confirmed that if I alter the filter to include very few blocks (around 10), and the result is a transaction list with around 10 entries and things are fine.

A filter which reproduces the error:
new_block_filter = web3.eth.filter({"fromBlock":hex(5235798),"toBlock":"latest", "address":"0xef68e7c694f40c8202821edf525de3782458639f"})

A filter which works smoothly:
new_block_filter = web3.eth.filter({"fromBlock":hex(5335798),"toBlock":"latest", "address":"0xef68e7c694f40c8202821edf525de3782458639f"})

And again, everything is fine with HTTPProvider(); it's only the WebSocketProvider()

[carver]

The two lines that create a filter are the same. Can you update your comment @davehcker ?

[carver]

cc @voith

[davidahmed]

@carver my bad. I've updated the comment.

[voith]

@davehcker I can replicate the issue with infura.

from web3 import Web3

w3 = Web3(Web3.WebsocketProvider('wss://mainnet.infura.io/_ws'))
new_block_filter = w3.eth.filter({"fromBlock":hex(5235798),"toBlock":"latest", "address":"0xef68e7c694f40c8202821edf525de3782458639f"})
w3.eth.getFilterLogs(new_block_filter.filter_id)

Firstly, failing on NoneType is a bug in websockets :

~/Desktop/contracts-py/contracts/lib/python3.6/site-packages/websockets/exceptions.py in __init__(self, code, reason)
    145         self.reason = reason
    146         message = "WebSocket connection is closed: "
--> 147         if 3000 <= code < 4000:
    148             explanation = "registered"
    149         elif 4000 <= code < 5000:

TypeError: '<=' not supported between instances of 'int' and 'NoneType'

However, I need to investigate what is causing the connection to close!

[voith]

@davehcker The problem was that the filter returned a lot of data(more than the default size that a websocket connection allows). On my re attempt to fetch data from the filter I got the following error:

~/Projects/web3.py/.tox/py36-integration-parity-ws/lib/python3.6/site-packages/websockets/framing.py in read(cls, reader, mask, max_size, extensions)
    122                 "Payload length exceeds size limit ({} > {} bytes)"
--> 123                 .format(length, max_size))
    124         if mask:

PayloadTooBig: Payload length exceeds size limit (33572833 > 1048576 bytes)

The max_size has a default value of 2 ** 20:
https://github.com/aaugustin/websockets/blob/b7a2bfe4ed5a14208ee3658eaad9be9f689cb6fb/websockets/client.py#L329
I tried increasing the value of max_size and it worked. However, the current implementation of WebsocketProvider does not allow passing extra config, for which I have opened #914.

@davehcker Can you help me check if increasing the value of max_size fixes the issue for you?
Install web3 from my branch:
pip install git+https://github.com/voith/web3.py.git@allow-passing-websocket-kwargs
and then:

from web3 import Web3

w3 = Web3(Web3.WebsocketProvider('wss://mainnet.infura.io/_ws', max_size=999999999))
new_block_filter = w3.eth.filter({"fromBlock":hex(5235798),"toBlock":"latest", "address":"0xef68e7c694f40c8202821edf525de3782458639f"})
w3.eth.getFilterLogs(new_block_filter.filter_id)

However, I have no idea why it was giving that random NoneType error earlier.

Also, A suggestion for you: Instead of fetching such big chunks of data, break fromBlock and toBlock in to smaller batches!

[boneyard93501]

@voith just guessing but ws client default timeout is ten seconds, i believe, and explains the None for large requests. the first time around, no infura cache in place, i presume, the client times out. second time around, infura cache is primed, you get the size issue. passing a (much) larger timeout value with the kwargs should solve the issue client side.

[voith]

@boneyard93501 I know what happened. I have two different virtualenv's on my machine. One with with websockets v4 and one with v5. Unconsciously, I switched between the two while testing. 😆
I get the NoneType error in v4. However, a security fix in websockets released in v5 errors earlier with a helpful error message. Interestingly, we bumped the websockets version to v5 just yesterday.
Also setting max_size to a large value fixes the issue in v4 as well.

just guessing but ws client default timeout is ten seconds, , i believe, and explains the None for large requests.

That was not the issue here. Here, data was being received continuously. It was just beyond the max_size limit.

@carver I think It's time to cut a minor release if @davehcker can confirm that #914 fixes the issue for them.

[carver]

Sounds good. Once we get #914 in, I'll cut a release.

[davidahmed]

@voith
Yes, it was the data max_size issue. And setting it appropriately fixes the issue.