Raspiblitz: bitcoind: WARNING: request rejected because http work queue depth exceeded, it can be increased with the -rpcworkqueue= setting

[openoms]

Error in the bitcoind debug.log:

sudo tail -n 20 /mnt/hdd/bitcoin/debug.log

WARNING: request rejected because http work queue depth exceeded, it can be increased with the -rpcworkqueue= setting

Error in JoinMarket:

[ERROR]  Failure of RPC connection to Bitcoin Core. Application cannot continue, shutting down.
[ERROR]  Critical error updating blockheight.    

Error in Specter:

Server error!
Exception: Server responded with error code 500: Work queue depth exceeded

Details:

Traceback (most recent call last):
  File "/home/bitcoin/.specter/.env/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/bitcoin/.specter/.env/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/bitcoin/.specter/.env/lib/python3.7/site-packages/flask_login/utils.py", line 272, in decorated_view
    return func(*args, **kwargs)
  File "/home/bitcoin/.specter/.env/lib/python3.7/site-packages/cryptoadvance/specter/controller.py", line 1164, in wallet_receive
    wallet.check_unused()
  File "/home/bitcoin/.specter/.env/lib/python3.7/site-packages/cryptoadvance/specter/wallet.py", line 105, in check_unused
    while self.rpc.getreceivedbyaddress(addr, 0) != 0:
  File "/home/bitcoin/.specter/.env/lib/python3.7/site-packages/cryptoadvance/specter/rpc.py", line 308, in fn
    r = self.multi([(method, *args)], **kwargs)[0]
  File "/home/bitcoin/.specter/.env/lib/python3.7/site-packages/cryptoadvance/specter/rpc.py", line 301, in multi
    "Server responded with error code %d: %s" % (r.status_code, r.text), r
cryptoadvance.specter.rpc.RpcError: Server responded with error code 500: Work queue depth exceeded

Mentions:
https://github.com/bitpay/bitcore-node/issues/463#issuecomment-269063216
https://github.com/zkSNACKs/WalletWasabi/pull/3056

Solution:
setting:

rpcworkqueue=64

in the bitcoin.conf (default is 16) and will monitor.

EDIT: same issue with rpcworkqueue=64
Increased to: rpcworkqueue=100

Note the extreme usecase with many services connecting to the JSON-RPC:
24 wallets active in Specter
2 JM YieldGenerators running
LND

[openoms]

It seems to be caused by Specter querying all the monitored addresses at the same time.
Problematic since LND might miss some onchain action while the JSONRPC is flooded.
Good news is that @stepansnigirev is already working in the solution for a future release of Specter: https://github.com/cryptoadvance/specter-desktop/pull/588

Until it is merged the safest option is to disable Specter on this node. (Can create a backup and import it on another machine.)

[rootzoll]

Putting this on the v1.7 milestone as a reminder to revist this then.

[stepansnigirev]

One more PR to Specter-Desktop that reduced number of RPC calls:
https://github.com/cryptoadvance/specter-desktop/pull/631
We plan a release tomorrow.

[rootzoll]

Created issue to update Specter for v1.6.2 release #1793 ... but will keep this issue listed for v1.7 to check if this is still an issue by then.