Nano-node: 'delegators' and 'delegators_count' RPC commands run very slowly or not at all

[kilkelly]

Issue:
When running the delegators or delegators_count RPC commands on a node the response is either very slow or can even lock up / crash the node.

Steps to reproduce the issue:
Run either the delegators or delegators_count via RPC on the node and wait for the results.

Describe the results you received:
My node never returned results and became unresponsive to further RPC commands i.e. locked up.

Describe the results you expected:
Expected to see results as documented in the Nano wiki.

Environment:

• OS information: Linux li293-21 4.18.16-x86_64-linode118 # 1 SMP PREEMPT Mon Oct 29 15:38:25 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Nano Node version: V17.0
• Docker: 18.06.1-ce
• System stats: 2 GB RAM, 1 CPU Core, Linode VPS

Possible Solution:
Optimize how the delegators and delegators_count RPC commands interact with the ledger.

[cryptocode]

There's no inverse lookup table for delegators, so this currently entails looping through the account tables (v0 and v1) and, for each account, reading the rep block to figure out if the block's representative matches the account being queried. Faster lookups are possible, but it seems like it will be at cost of denormalization (a larger ledger)

[kilkelly]

That was my hunch as well. It wouldn't be worth any ledger size increase just for this command. Looks like some stronger hardware is required on my side. Thanks @cryptocode 👍

[zhyatt]

Closing out due to optimization tradeoffs not worth it.

[meltingice]

For what it's worth, I'd take a larger ledger in order to speed up this call. It's a feature I like having in NanoCrawler, and the RPC call is painfully slow.

[zhyatt]

Reopening for further discussion of tradeoffs

[meltingice]

This is perhaps something that would greatly benefit from a customizable database adapter. Could leave the current lmdb implementation as default with the slow delegators call, but allow other database systems (or maybe even a reconfigured lmdb) if desired.

[kilkelly]

Alternatively I wonder if it is possible to have a config.json setting for enabling the denormalisation required to achieve the faster delegators call? A node operator probably wouldn't mind a larger ledger if they were working with that call and delegators_count a lot.

[zhyatt]

@kilkelly Indeed. Making this optional was an approach I was going to discuss with the team. Will get an update on this within the next week or so.

[zhyatt]

It is taking a bit longer than anticipated to dig into this but will be considering for future release.

[kilkelly]

Is the upcoming "sideband" functionality something that can be utilized for optional, configurable collection of delegator counts or is it just for block info?

BTW I did manage to collect delegator counts for representative addresses I am tracking without using the delegators_count RPC command but it required navigating through the entire ledger with the ledger RPC command.

[cryptocode]

@kilkelly what's the performance of that solution?

[kilkelly]

It varies between 30 minutes to almost 3 hours sometimes. Feels kind of random but that's probably because of regular transaction activity the node is processing also. On a more powerful node it shouldn't take too long.

I navigate through the ledger in chunks of 1000 frontiers at a time.

[cryptocode]

@kilkelly I'm not able to reproduce it taking that long (on master) - 10 seconds at most. Do you have an exact RPC call example?

[kilkelly]

Hi @cryptocode , it was a normal, vanilla delegators RPC call as outlined in the RPC wiki. I believe the time taken for the call to complete is directly related to the strength of the hardware. I'm currently upgrading my node hardware (8 core, 30 GB) and it takes around 10 - 15 seconds on there. My old, weak node (2 core, 2 GB) was the one giving the extremely slow call.