Gridcoin-Research 🚀 - RPC stops responding

Can you provide any other details?

What OS
What client version
How often are you calling the RPC for getmininginfo
debug4 flag can give how long it takes the RPC to respond in milliseconds
is getmininginfo the only rpc command your calling regularly?

Side Note: debug window does not use the actually RPC Connection interface.

This seems it could be a dead lock of some sorts

iFoggz on 23 Oct 2019

OS running wallet is Windows Server 2012
Client v4.0.5.0-gddaaf8d5a
In test phase I'm currently polling getmininginfo (and only getmininginfo) about once every minute or two (inexact repeat) - intention would be to reduce to three or four times a day I suppose.
debug4 I will try; generally speaking the response when it works is well under a second IIRC - certainly doesn't hang for any undue or irritating length of time until it just stops responding altogether
getmininginfo is the only rpc command being called on the timer. The wider application would call also getBalance and getInfo but only upon UI request, not on a timer. Every time the RPC has stopped responding I haven't used the UI in a long time; the only request will have been the background timer calling getmininginfo.

9swampy on 23 Oct 2019

...on the debug window side note; as I suspected but I thought it was worth pointing out.

9swampy on 23 Oct 2019

everything is worth pointing out, so no worries there. if this happens again check the debug.log and see how far apart the getmininginfo call was and any other calls made by ui. i'd like to know if they were closely called, etc. keep us up to date.

iFoggz on 23 Oct 2019

Also wanted to let you know a leisure of 4.0.6.0 is out.

There were some other RPCs I added in the past as getmininginfo is pretty intensive as it locks multiple areas for data. We have the features that could give you the data as well with less locking if your interested as well.

./src/gridcoinresearchd -testnet getwalletinfo
{
  "walletversion": 60000,
  "balance": 75457.93407000,
  "newmint": 0.00000000,
  "stake": 0.00000000,
  "keypoololdest": 1571849190,
  "keypoolsize": 101,
  "unlocked_until": 1581849325
}

./src/gridcoinresearchd -testnet getblockchaininfo
{
  "blocks": 1061475,
  "moneysupply": 369132415.76601731,
  "difficulty": {
    "proof-of-work": 0.006455256165368031,
    "proof-of-stake": 0.4397377744377047
  },
  "testnet": true,
  "errors": ""
}

./src/gridcoinresearchd -testnet getnetworkinfo
{
  "version": "v4.0.5.4-ga5af63e3a-dirty",
  "minor_version": 0,
  "protocolversion": 180323,
  "timeoffset": 0,
  "connections": 3,
  "paytxfee": 0.00010000,
  "mininput": 0.00000000,
  "proxy": "",
  "ip": "redacted",
  "errors": ""
}

iFoggz on 23 Oct 2019

Running with debug it's died again in the last couple of hours. Reviewing logs seems like from around the same time it died there started appearing batches of 6 or so:

10/23/19 16:30:38 Cleaning inbound connections
10/23/19 16:30:38 ERROR: AcceptToMemoryPool : Unable to Connect Inputs 92949363f6791aaae0e94f82e45bed46929b26f4de8ffd82da377cf807be58b5
10/23/19 16:30:38 ERROR: AcceptToMemoryPool : Unable to Connect Inputs a675e4bea12a21fcf388e98628bd20f7a2e40e6bc76373205c3ea79e4a61ba44
10/23/19 16:30:38 ERROR: AcceptToMemoryPool : Unable to Connect Inputs 964ad129720532f17925b2c926db72e64784f9cad87ce3e7e5a45357a0365bfb
10/23/19 16:30:38 ERROR: AcceptToMemoryPool : Unable to Connect Inputs 05f9328fb9cc23b6511c4a01e2e71ff289d821fb68b3e093e26c15dd3e4700d8

Randomly, certainly not every minute or so; anywhere from 5 minutes apart to 30 minutes silence then batches start again.

Final close of client appended:

10/23/19 16:39:01 Tally (v9)
10/23/19 16:39:01 {SBC} new best {31bd38e071cdf71093e72d4350a65c28324ffceec45670c031d67db2db81cff6 1745570} ; 
10/23/19 16:40:26  Received block 160504da993554c54d593dc1ad7602030a2b99f263ffafb38f74252902e7ebbb; 
10/23/19 16:40:27 {SBC} new best {160504da993554c54d593dc1ad7602030a2b99f263ffafb38f74252902e7ebbb 1745571} ; 
10/23/19 16:41:09  Received block e8e1caebf2fc2e641f97fb4c7c604c7716508bb28b5360b58d4eba00877e5bd4; 
10/23/19 16:41:10 {SBC} new best {e8e1caebf2fc2e641f97fb4c7c604c7716508bb28b5360b58d4eba00877e5bd4 1745572} ; 
10/23/19 16:41:43 Main calling Shutdown...
10/23/19 16:41:43 gridcoinresearch exiting...
10/23/19 16:41:43 ThreadSocketHandler exited
10/23/19 16:41:43 ipcThread exited
10/23/19 16:41:43 ThreadStakeMiner exited
10/23/19 16:41:43 ThreadMessageHandler exited
10/23/19 16:41:44 StopNode()
10/23/19 16:41:44 ThreadOpenConnections exited
10/23/19 16:41:44 Wait for 10 threads to join.
10/23/19 16:41:44 ThreadOpenAddedConnections exited (interrupt)
10/23/19 16:41:44 ThreadMapPort exited (interrupt)
10/23/19 16:41:44 ThreadNeuralNetwork exited (interrupt)
10/23/19 16:41:44 ThreadDumpAddress exited (interrupt)
10/23/19 16:41:44 Stop RPC IO service

Having shutdown the client was gone but the SysTray icon was still there; wouldn't respond to any clicks left or right, only Gridcoin Client popup upon hover. Only option was to terminate the process from TaskMan.

I've restarted with debug4. Will revert with further info in due course. Thanks for attention thus far.

9swampy on 23 Oct 2019

Re other calls it's specifically "staking" i'm needing to poll to compensate for the removal of auto_unlock. Before finding staking I had seen the unlocked_until which could have potentially sufficed were it populated when the wallet is unlocked via the UI. Unlocking via RPC sets the unlock_until but not the UI; unlock until remoains 0. Is that "fixed" in 4.0.6.0? The other two commands I don't see the link to staking, am I missing something?

9swampy on 23 Oct 2019

Maybe easiest answer would be to add a command purely to getStaking :)

9swampy on 23 Oct 2019

I understand "staking" part. that essentially is checking the miner last stake attempt time and miners weight sum and if that has data then staking is true. and That is not a bad idea to add a RPC to specifically staking data. And yes the ui does not affect the unlocked until since it is unlocked until locked or closed.

@jamescowens maybe we should consider adding whether or not the wallet is staking in the getwalletinfo as an additional field?. unlocked_until from nWalletUnlockTime is not very useful when unlocked from UI side.

@9swampy that clean inbound connections part has been changed in 4.0.6.0 I do recommend you upgrade to that version as we refactored the code and no longer does the Cleaning inbound connections.

iFoggz on 23 Oct 2019

I am just catching up with this. A couple of comments...

This is almost assuredly a deadlock issue. We need to reproduce this under gdb and do a backtrace and look for lll_wait to confirm.
The auto unlock feature provides about as much security as an in the clear password recorded in a script file to do the unlocking. Rather than unlocking from the UI, why don’t you unlock from a script instead with a defined time limit, so you know when it will expire rather than polling all of the time?