Web3.py: ValueError when using IPCProvider

[medvedev1088]

• Version: 4.2.1
• Python: 2.7/3.4/3.5
• OS: osx/linux/win

What was wrong?

When using IPCProvider sometimes ValueError is encountered if the response is incomplete. Looks like in this pull request https://github.com/ethereum/web3.py/pull/785/files the author forgot to add except ValueError to IPCProvider https://github.com/ethereum/web3.py/blob/master/web3/providers/ipc.py#L176

How can it be fixed?

Add except ValueError to IPCProvider.make_request https://github.com/ethereum/web3.py/blob/master/web3/providers/ipc.py#L176

[carver]

Ah, thanks for tracking that down! Another solution would be to re-raise a JSONDecodeError here:
https://github.com/ethereum/web3.py/pull/785/files#diff-0260bdf984abdd4b8ab8987ea75d032cR258

[boneyard93501]

@carver i'll look at it tonight. i think the reason for not re-raising was to avoid the nested (double) error. let me have a look.

[boneyard93501]

i'm having a bit of trouble reproducing the issue.
for, say,

curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"net_listening","params":[],"id":67}'

a complete result would be {"jsonrpc":"2.0","result":true,"id":67} and an incomplete result would be{"result":true,"id":67} assuming an incomplete response is one where one or more of the json-rpc required key-value pairs is missing.

>>> web3.__version__
'4.2.1'
>>> incomplete_res = b'{result":true,"id":67}'
>>> # passes the has_valid_json_rpc_ending check and
>>> IPCProvider().decode_rpc_response(incomplete_response)
{'result': True, 'id': 67}

which checks out. if instead we're looking at a structurally invalid json response other than invalid rpc endings:

>>> web3.__version__
'4.2.1'
>>> invalid_response = b'{"result":true,"id"}'
>>> IPCProvider().decode_rpc_response(invalid_response)
Traceback (most recent call last):
  File "/Users/93501/localDev/source-code/forks/web3.py/web3/utils/encoding.py", line 254, in json_decode
    decoded = json.loads(json_str)
  File "/Users/93501/.pyenv/versions/3.6.4/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/Users/93501/.pyenv/versions/3.6.4/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Users/93501/.pyenv/versions/3.6.4/lib/python3.6/json/decoder.py", line 355, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ':' delimiter: line 1 column 20 (char 19)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/93501/localDev/source-code/forks/web3.py/web3/providers/base.py", line 63, in decode_rpc_response
    return FriendlyJsonSerde().json_decode(text_response)
  File "/Users/93501/localDev/source-code/forks/web3.py/web3/utils/encoding.py", line 258, in json_decode
    raise ValueError(err_msg)
ValueError: Could not decode '{"result":true,"id"}' because of Expecting ':' delimiter: line 1 column 20 (char 19).
>>>

is raising the stacked Exception the issue?

[medvedev1088]

@boneyard93501 I meant the structurally invalid result. When the json rpc response is not fully received from the socket, the IPC provider will try to parse it anyway and if the parsing failed, it will continue to receive more chunks from the socket, until the json is valid.

An example of incomplete response would be:

'{"jsonrpc":"2.0","result":["test"]'

[boneyard93501]

k. thanks. appreciate the update.

at the surface, that looks more like a has_valid_json_rpc_ending error of omission since it's a little permissive. if we tighten that up, we don't even get to the json processing. at the mere minimum we could check for matching startswith/endswith.
however, having PersistentSocket parse the complete response ought to be the preferred course of action. maybe increase Timeout a tad and do a socket.reset before breaking from the while loop to at least prevent orphaned chunks to further pollute the results. that, in conjunction with a tighter valid rpc check, might give us much improved coverage. deferring to @carver

[carver]

has_valid_json_rpc_ending will always be optimistic, and fail sometimes. For example '{"jsonrpc":"2.0","result":[{"key":"val"}'. The provider's parser already handles that case by capturing JSONDecodeError, so all we need to do is raise a JSONDecodeError instead of a ValueError in our friendly decoder. It should be pretty quick, I'll take care of it.