Eos: 1.8.9 regression: RPC internal service error on block 100134250

[quyse]

After upgrade to 1.8.9 we got error 500 on block 100134250:

# curl -d '{"block_num_or_id":100134250}' http://127.0.0.1:8888/v1/chain/get_block
{"code":500,"message":"Internal Service Error","error":{"code":2,"name":"timeout_exception","what":"deadline 2020-01-14T09:50:11.321 exceeded by 51us ","details":[{"message":"deadline 2020-01-14T09:50:11.321 exceeded by 51us ","file":"json.cpp","line_number":499,"method":"escape_string"}]}}

It is reproducible on all 1.8.9 nodes we have.
It seems previous or following blocks are not affected. Downgrading to 1.8.8 solves the issue for this block as well.

In case it matters, we use our own docker images for EOS (nothing fancy)
Dockerfile: https://gitlab.com/coinmetrics/fullnodes/eos/blob/c12e2a212b35069473c0b2098dc08ebf40869778/Dockerfile
Images: https://quay.io/repository/coinmetrics/fullnode-eos?tab=tags

[matthewdarwin]

Probably you need to increase the value of http-max-response-time-ms from the newly introduced default of 30. See the release note for details https://github.com/EOSIO/eos/releases/tag/v1.8.9

[heifner]

As Matthew said, 2.0 and 1.8.9 includes the new option http-max-response-time-ms which is to help mitigate against abuse of api nodes. It allows you to specify a time limit on processing of a request. This currently means a limit on the time it takes to convert the response to json which is the most time consuming part of most api requests. We made the default 30ms to match the default of max-transaction-time. For large blocks, 30ms is likely not going to be enough time for your node to make the conversion to json. Depending of course on hardware etc. Feel free to put it at any level you are comfortable any one request taking to process.