Nano-node: Get account history starting at a block and ending in head

[guilhermelawless]

Reported by Natrium.
Currently it is cumbersome for integrators to build an incremental account history internally. Scenario:

• Initially, request full account_history to cache it internally.
• When updating the history, to prevent getting too much data unecessarily (some accounts have hundreds of thousands of blocks), one would expect to use account_history starting at the internally cached head block, and up to the new head block.

Currently I see no easy way of doing this, and it seems like a pretty important thing to have. successors doesn't let you start at a block, only end. If that option existed, one could use successors followed by blocks_info as an alternative.

Ideally, account_history would have an optional reverse and could also start at a block. However, the semantics of the returned values stop making sense in that RPC, namely, the returned field "previous" would have to be "next" instead.

What would be the best way to approach this? I don't mind coding it once it's agreed upon.

---

Also, maybe we could add (block) "height" to the blocks in account_history?

[renesq]

You're right, such a feature could come in handy for the account_history command. Altering the behavior of the existing "head" option with an additional "reverse" option might be more align with other RPC commands, but in real life scenarios, a "end" option (="until block with hash"="recent blocks but no farther than...") would make more sense I guess, as a reversed block sequence is not needed in normal UI.

In your earlier edit, you misunderstood the successors command. It will actually list blocks that happened in the more recent time, unless the "reverse" option is set to true. But that command will only list block hashes, so you'd have to do some extra steps to get those block contents (either by using block_info one-by-one which is slow and tedious, or much better by querying account_history accordingly - including the "head" option to avoid race conditions, and the "count" parameter that matches the response of the successors command, and the "raw" parameter because account_history potentially leaves out blocks otherwise).

But, there is another simple, straightforward workaround to gather only new blocks, yet it involves more round trips than being able to set an "end", and has a slight chance of race conditions you'd need to check for:

• After your first sync, count the number of blocks you gathered.
• Before the second sync, query the block_count via account_info (optionally, make sure that the frontier hash returned is still in local storage, in order to detect forks)
• In the second sync, set count to "current block-count minus first sync block count"
  Keep in mind that this will only work with the "raw" option activated on account_history, because the default query will leave out change blocks and epoch blocks, so the numbers wouldn't add up in some cases. Unfortunately, the raw option will take up to 4x longer than the default query according to my research.
  Also keep in mind that this workaround will only work if no new block occurred between requesting the block_count and doing the account_history query - which should normally be the case because you're in full control of your own account and won't insert new blocks in the meantime. If you want to incorporate a solution for this potential flaw: either reduce the "count" limiter on "account_history" by 1 so you can cross-check if you already got the last block (if not, try to sync again), or re-query the block_count after querying account_history. But this race condition should be rather rare.

I actually prefer the simple way to "sync": Don't cache anything locally, simply request the 200 most recent blocks of the Tx history and cut off after that. This way, it's quick, saves round-trips, and you'll never have to deal with forks and stuff. If a user wants to scroll farther back, they can either use a fully fledged block explorer, or one could implement pagination via the "head" option of the "account_history" command.
IIRC, canoe cached the Tx history locally, and manually inserted newly generated blocks in there before they were confirmed by reps, but this apparently caused some issues and race conditions, so users often had to use the "repair" feature.

Appendix:
A "raw" account_history block for reference:
```{
"type": "state",
"representative": "xrb_3jwrszth46rk1mu7rmb4rhm54us8yg1gw3ipodftqtikf5yqdyr7471nsg1k",
"link": "FD439DDD7940E90A2CA4B7AFE6743932008C9638EA321F0E5669E08F5DF54E84",
"balance": "4924131438934728148868649242841566583",
"previous": "A9775AC157DD91D72A27F6925940CC7278DEF3727D7905BB9CF6D309BD93CBB9",
"subtype": "send",
"account": "xrb_3zc5mqgqki9b3apcbfxhwst5kei1jkd5jtjk5w97eth1jxgzcmn6r95bqmg9",
"amount": "349092750000000000000000000000000",
"local_timestamp": "1552403461",
"hash": "32E73F47005A00D2DF5672A67994EE0903BC9DCA2B262D4B7651B7BF0173D085",
"work": "616a9c1a997fdea6",
"signature": "E6436E90DDAEFFC468D8EA04AD26F52C9AD984E8BA33F7DCCD023AA081A412B7EE456F945474A6ABAF371C4F4165ED07EECEC79C7D080655A113C0CDFC2CFF0B"
}

vs. the standard:

``` {
            "type": "receive",
            "account": "xrb_3btyufq6jnr4mx3eemfh965rf5381y8wmjyrytxcoyk5ehnu9t5n3qinr1dd",
            "amount": "393919542000000000000000000000000",
            "local_timestamp": "1552405206",
            "hash": "203A9E0F9D02D3647E6C697780AE6A2727BEA07CFCB4F5705CA6A6B33260EE18"
        },
        {
            "type": "receive",
            "account": "xrb_3o1texi7gy48cq9895swdkoanecati674n4drmae1s5gpbgw7jis7id1spoa",
            "amount": "15429540139990000000000000000000000",
            "local_timestamp": "1552405205",
            "hash": "9AFC408D37DD6CA2BCA61A158C9CCCC0E0C21F7A1CBE717D7061578767EF5BD0"
        },
        {
            "type": "send",
            "account": "xrb_3zc5mqgqki9b3apcbfxhwst5kei1jkd5jtjk5w97eth1jxgzcmn6r95bqmg9",
            "amount": "349092750000000000000000000000000",
            "local_timestamp": "1552403461",
            "hash": "32E73F47005A00D2DF5672A67994EE0903BC9DCA2B262D4B7651B7BF0173D085"
        },

Also the successors query for the same three blocks, but from the bottom up:
```"action": "successors",

"block": "32E73F47005A00D2DF5672A67994EE0903BC9DCA2B262D4B7651B7BF0173D085",
"count": "3"
}' '[::1]:7076'
{
"blocks": [
"32E73F47005A00D2DF5672A67994EE0903BC9DCA2B262D4B7651B7BF0173D085",
"9AFC408D37DD6CA2BCA61A158C9CCCC0E0C21F7A1CBE717D7061578767EF5BD0",
"203A9E0F9D02D3647E6C697780AE6A2727BEA07CFCB4F5705CA6A6B33260EE18"
]
```

[guilhermelawless]

Thank you for digging into this. I was indeed wrong about successors so it does offer an alternative solution.

Some takeaways then:

• account_history documentation should reflect that only send and receive blocks are returned if raw is false. It is not clear at all.
• In my opinion, successors documentation should say "Returns a list of block hashes in the account chain endingstarting at block up to count"
• account_history needs some enhancements, namely "reverse" and perhaps "height" of each block

[SergiySW]

@guilhermelawless Expand description

Change blocks are skipped, open blocks will appear as receive (unless raw is set to true - see optional parameters below).

successors definition depends on other descriptions especially on chain & account_history. Default order for chain there is from newest blocks to older. "Returns a list of block hashes in the account chain starting at block up to count" is exactly "chain" description while it should show difference. More clear description may be required, right

Both height & reverse should be easy to impement. Why not

[guilhermelawless]

@SergiySW Sorry, missed that part of the description.

You can leave height & reverse to me.

[renesq]

Guilherme is right with his proposal for the successors description. chain works differently than successors, the latter essentially is 'going forward' from the given hash's standpoint and puts the given hash on top, so it should be rephrased from "ending" to "starting". Ultimately, real life examples will help understanding the mechanism.

When I wrote this

Change blocks are skipped, open blocks will appear as receive (unless raw is set to true - see optional parameters below).

state blocks and epoch blocks didn't exist yet. Their role may be worth mentioning.

You can also add the undocumented "offset" parameter for account_history to the wiki page

[SergiySW]

@guilhermelawless if you want to make it :)

[guilhermelawless]

I looked through the documentation and the only mention of "starting" and "ending" related to blocks in an account is in chain and successors, so you could just switch the meanings there:

• Successors: starts at account
• Chain: ends at account

I know in the code/protocol it works the other way but for whoever is reading the RPC documentation it makes more sense as I wrote, I think (and we all seem to agree).

[guilhermelawless]

I wonder if #1355 will be merged soon so I can use the core_test from there.

[SergiySW]

@guilhermelawless @renesq chain, successors & account_history definitions in wiki slightly updated for clarification

[SergiySW]

Closed automatically, reopening

[guilhermelawless]

What is missing? @SergiySW

[SergiySW]

Wiki check with fresh eyes

[guilhermelawless]

I think it might be worth repeating Will list all blocks back to the open block of this chain when count is set to "-1". The requested block hash is included in the answer. in successors.

For account_history, offset was available before, just not documented. Since v11 it's there: https://github.com/nanocurrency/nano-node/blob/release_v11/rai/node/rpc.cpp#L1747

[SergiySW]

@guilhermelawless thx

[zhyatt]

Reviewed wiki updates and they appear accurate.