Status-react: No transaction history when address has a lot (100+) of transactions

[Serhy]

Bug Report

_Expected behavior updated 1/11/19. Issue is ready for pick-up._ [RH]

Problem

For the recovered accounts having a lot of transactions it very long to fetch and display transaction history in History tab. Another note is that we apparently fetch them from the earliest to latest transaction which makes much less sense than if we reverse this order (so currently I see TXs I had 2 years ago and not those I sent/received today).

So with the account having ~650 transaction during the last 1.5 year on Mainnet it took ~10 minutes for last transactions (ETH and ERC20) I had couple of days ago to be displayed.
And for the account with 11000+ transactions (in Ropsten network):

• after ~4 minutes account recovered I've got first 6 ERC20 transactions made 1 year ago in History tab
• after ~20 minutes pass I've got bunch of 100 transactions made 1 year ago more
  It might take forever for me to see tx sent/receive today for this 11k-tx account (seed phrase of this acc provided in steps below)

#### Expected behavior
_[TBD as it needs input from dev, design but for now...]_
Disregard the number of transactions address has I see last 10 transactions in less than 1 second performed with this address.

Step 1 / Fetch latest transactions before earliest. Rely on Etherscan to get latest 20.
Step 2 / Fetch if on wifi or user scrolls.
Step 3 / Take over for Etherscan by Infura if fetching slows.

Actual behavior

It takes 20+ minutes for list of 600 transactions to appear on Wallet -> Account -> History tab

Notes

It seems that issue appeared after we moved Wallet to status-go.

Acceptance Criteria

• navigating to Wallet -> Status account -> History tab of an account with 11000+ transactions takes < 1 second for last 10 transactions to appear
• pulling down on History tab loads more transactions [10-txs in batch?]

Reproduction

• Install fresh Status build
• Recover the next account tree weekend ceiling awkward universe pyramid glimpse raven pair lounge grant grief
• Switch to Ropsten network
• Navigate to Wallet -> Status account -> History tab
• Verify there is list of last 10 txs same as in etherscan for it's address https://ropsten.etherscan.io/address/0xf184747445c3B85CEb147DfB136067CB93d95F1D

Additional Information

• Status version: develop nightly 0.14.0 (2910101102)
• Operating System: Android and iOS

[Serhy]

@rachelhamlin, included this in V1 scope since highly likely the issue to be seen by users

[rachelhamlin]

Yes, let's try to fix. 👍

[flexsurfer]

as a quick simple solution, show only new transactions in history

[rachelhamlin]

@yenda do you have any thoughts on how we could improve this one?

Step 1 - show newest transactions first/only (until when is 'new'?)
Step 2 - ?

[yenda]

@rachelhamlin I would say

• step 1 implement fetching transactions from etherscan (only the first 20 transactions)
• step 2 if on wifi or user starts scrolling, fetch more
• step 3 make sure fetching from infura can take over if etherscan is down

[rasom]

@Serhy @yenda @flexsurfer

Yesterday I checked how it works on status-go side. It seems that there is some kind of binary search which happens on the interval from the first block until the current block. The algorithm checks if there's any change in balance value during that interval, if there was a change divides it into two intervals and checks balance change in these intervals, and so on, until a specific block with transaction will be found. Although it might be O(log n) in case if there is only a single transaction in history, the worst case here will be m log n, where m is the number of transactions. That's why there is a correlation between number of transactions in history and fetching time.

Although we definitely can fallback to etherscan, as far as understand we switched to a new implementation because we planned to not rely on centralised services and so we would need to remove ethers dependency later anyway.

Before switching to etherscan I would try to have ~4 separate wallet_getTransfers calls for day, week, month and the rest of history (of course approximately, because we can only pass block numbers there). It sounds pretty straightforward to me and we could check whether it will be faster this way.

Also it would be great to have some feedback from status-go when search for transfers has ended without any results, in this case we also could show user that fetching is in progress, instead of showing No transactions label.

Let me know if I missed some discussion on this regard and Etherscan has already been chosen as a best solution for this case.

[flexsurfer]

@hesterbruikman @errorists @rachelhamlin i also think we shouldn't retrieve any history by default and have an option for for day, week, month and the rest of history because we don't want to rely on etherscan, and transaction history from blockchain is too expensive operation, so for v1 i suggest to do not retrieve any tx history by default, and have options to retrieve for day, week and month, wdyt?

[errorists]

@flexsurfer I don't disagree but do you imagine that would be a manual fetch, so at first it's all empty right. you press a button to load last 10 items in history, as you start scrolling down you fetch backwards in batches of 10.

what I'm saying is, I don't fancy placing buttons there like we did for chat to fetch missing gaps in history, those suck

[rasom]

Just tried to play with that wallet_getTransfers method and it seems that search through the whole history happens regardless of requests parameters. Request itself only returns data which already exists in database, and routine which scans history dispatches signals with found transactions. So we don't have any control on this on status-react side, and solution which I proposed will not work.

The way to fix it is to rewrite scanning on status-go side, so that instead of scanning the whole chain at once we do this in smaller portions (7-dayish number of blocks or something) and start with latest history.

[yenda]

@rasom correct observations, I think though that @rachelhamlin @andremedeiros have already been discussing with etherscan about using their API.
If we want to fix rewrite the on-chain parsing though, I think that it is better to rely on the number of transactions than a time portion as it is more meaningful for the user. You can request how many transactions have been made on a particular address over a certain period of time afaik

[rasom]

@yenda there is a method which returns how many transactions was made, but I doubt that it's possible to make such request for a period of time.

[yenda]

@rasom yes that is the one I'm talking about, it takes a block argument so you can use it with a dichotomic search to find a timespan than contains for instance 10 transactions. I'm not 100% sure about token transactions though, outgoing would be included because that's a contract call and still a transaction, but I think you can only be aware of incoming token transactions to your address through events.So that could be a partial solution to improve the on-chain lookups, but at this point I think we are going for etherescan it's much simpler, and the current way can stay as a fallback, and also nothing stops us from verifying the correctness of the transaction data provided by etherescan. Later we can explore a third method with something like the graph.

[flexsurfer]

added v1 label back because 7k requests are nuts, the device is dying and other ethereum requests might be blocked, @rasom don't you mind to take this one and implement history only for N last blocks you've discussed with Eric, thanks

[rasom]

@yenda it seems that getTransactionCount counts only outgoing transactions, not sure how you wanted to use it

[rasom]

https://github.com/MyCryptoHQ/eth-scan

[rasom]

@cammellos provided the log of requests for a new account
https://gist.github.com/rasom/42a4e1756b62f887b514a3447140897f

[yenda]

@rasom if you combine getTransactionCount and getBalance you avoid the risk if missing out transactions that cancel out. Outgoing transactions will always at least cost gas, while incoming transaction will at least increase the balance. So if balance and transaction count didn't change then there is no new transaction in the range you are checking.

Btw eth_scan only checks balances and it's typescript so not sure it will be any help. We could check support for batched queries though if we aren't doing it already

[rasom]

@yenda https://github.com/status-im/status-react/blob/24f290d7869a555ffeba991ee125cef10929934b/src/status_im/utils/scan.cljs

[rasom]

my suggestions are:

1. we add caching for eth_getBalance/eth_transactionCount requests here, it will reduce number of calls 3-4 times
2. we add eth_transactionCount check in case if in checked range balances were equal to zero (we probably need to check it even if non-zero balances are equal, but by not doing this we will make ~2x less requests)
3. we stop fetching outgoing token transactions twice, as these transactions are covered by dichotomic search
4. we request only the last month of history and the rest of it only by additional request from a user

[rasom]

tested batch requests, it seems it doesn't make the whole process faster
https://github.com/status-im/status-react/blob/24f290d7869a555ffeba991ee125cef10929934b/src/status_im/utils/scan.cljs#L202

  ;; Get blocks res (batch-calls-limit = 1):
  ;; 0x2f88d65f3cB52605a54A833ae118fb1363aCccd2, mainnet, 131 transactions
  ;; {:from               0
  ;;  :to                 9025828
  ;;  :total-rpc-requests 5446
  ;;  :cached-entries     1369
  ;;  :blocks-count       131
  ;;  :single-calls-count 11
  ;;  :batch-calls-count  1358}
  ;;
  ;; batch size => duration of the whole process in ms 
  ;; 50 => 157853 
  ;; 25 => 95751 
  ;; 10 => 49344 
  ;; 5  => 30286 
  ;; 3  => 23877 
  ;; 2  => 20429 
  ;; 1  => 18062

[flexsurfer]

we'll use cloudflare probably ,so we could test batch requests with cloudflare

[rasom]

@flexsurfer sure we can, why do you think results will be different?

[rasom]

@yenda:

also for erc20 transactions there is no need to search from block 0, deployment block of the first erc20 token contract is enough

https://ethereum.stackexchange.com/questions/38378/what-is-the-first-ever-erc20-token-creation-transaction

[rasom]

It seems that we scan blockchain twice for each acc at the moment, I'm trying to figure why. I found that even when I add caching of requests we have 2 times more requests on go side than in that cljs test code.
` t=2019-12-17T14:36:23+0000 lvl=info msg="request balance" account=0x2f88d65f3cB52605a54A833ae118fb1363aCccd2 block=7584036 t=2019-12-17T14:36:24+0000 lvl=info msg="request balance" account=0x2f88d65f3cB52605a54A833ae118fb1363aCccd2 block=7584036 t=2019-12-17T14:36:29+0000 lvl=info msg="hit cache" account=0x2f88d65f3cB52605a54A833ae118fb1363aCccd2 block=7584036 balance=1432433598716618778 t=2019-12-17T14:36:29+0000 lvl=info msg="hit cache" account=0x2f88d65f3cB52605a54A833ae118fb1363aCccd2 block=7584036 balance=1432433598716618778 t=2019-12-17T14:36:35+0000 lvl=info msg="hit cache" account=0x2f88d65f3cB52605a54A833ae118fb1363aCccd2 block=7584036 balance=1432433598716618778 t=2019-12-17T14:36:35+0000 lvl=info msg="hit cache" account=0x2f88d65f3cB52605a54A833ae118fb1363aCccd2 block=7584036 balance=1432433598716618778

[yenda]

@rasom is it per account or for all? Maybe status-go is also checking the chat account

[rasom]

@yenda per account, but maybe the problem is how I wrote that caching, checking it now

[rasom]

no it is almost likely not related to my code

t=2019-12-17T15:49:40+0000 lvl=info msg="eth historical downloader finished successfully" total transfers=101 time=35.01709509s
t=2019-12-17T15:50:23+0000 lvl=info msg="eth historical downloader finished successfully" total transfers=101 time=28.721055941s

[rasom]

so when I restore acc 0x2f88d65f3cB52605a54A833ae118fb1363aCccd2 app scans chain for these addresses

t=2019-12-17T16:51:17+0000 lvl=info msg="fastIndex account" address=0x2f88d65f3cB52605a54A833ae118fb1363aCccd2
t=2019-12-17T16:51:17+0000 lvl=info msg="fastIndex account" address=0x2f88d65f3cB52605a54A833ae118fb1363aCccd2
t=2019-12-17T16:51:17+0000 lvl=info msg="fastIndex account" address=0xE911Def9de2Fd159e73921d218Cf2041e47fE69E

First question why 0x2f88d65f3cB52605a54A833ae118fb1363aCccd2 twice, second why 0xE911Def9de2Fd159e73921d218Cf2041e47fE69E... more and more questions

[rasom]

scanning of this acc https://etherscan.io/address/0x2f88d65f3cB52605a54A833ae118fb1363aCccd2
Data usage includes erc tokens history which hasn't been touched yet and the rest of app's communication

1. Duplication, no cache (that's what we have now), Data usage 53.9mb, 19.514074986s and 16.658236412s, app crashed

t=2019-12-18T09:50:48+0000 lvl=info msg="eth historical downloader finished successfully" address=0x2f88d65f3cB52605a54A833ae118fb1363aCccd2 total transfers=101 time=19.514074986s
t=2019-12-18T09:51:02+0000 lvl=info msg="eth historical downloader finished successfully" address=0x2f88d65f3cB52605a54A833ae118fb1363aCccd2 total transfers=101 time=16.658236412s

1. No duplication, no cache, Data usage 27.58mb, 12.98916826s

t=2019-12-18T09:40:09+0000 lvl=info msg="eth historical downloader finished successfully" address=0x2f88d65f3cB52605a54A833ae118fb1363aCccd2 total transfers=101 time=12.98916826s

1. No duplication, cache, Data usage 24.83mb, 8.916600494s

t=2019-12-18T09:33:40+0000 lvl=info msg="eth historical downloader finished successfully" address=0x2f88d65f3cB52605a54A833ae118fb1363aCccd2 total transfers=101 time=8.916600494s

[rasom]

If we do not load transactions right away when scanning db blockchain, the search for blocks which contain transactions ends fast even for our favourite account with more than 10k transactions https://ropsten.etherscan.io/address/0xf184747445c3B85CEb147DfB136067CB93d95F1D

t=2019-12-19T12:14:37+0000 lvl=info msg="eth historical downloader finished successfully" address=0xf184747445c3B85CEb147DfB136067CB93d95F1D total transfers=0 time=25.803520713s

so the next step will be to load transactions only from the last ~50 blocks or something, and then load rest of it only on scrolling transactions list. In this case we can keep scanning the whole history for each acc and actually have more less meaningful pagination by number of transactions instead of number of blocks and guessing if there were transactions at all.

[yenda]

50 blocks is only around 10 minutes though

[rasom]

@yenda what I mean is that we can determine all blocks which contain transactions in ~25s and then fetch transactions from the last N of them, not from the last N blocks in the blockchain. Currently if we found 10k blocks with transactions we try to fetch all transactions at once, these operations are much slower than eth_getBalance and we don't really need to fetch all 10k transactions as user, almost likely, will never scroll them all.

[rasom]

We can also improve that 25s time for the initial run, but I will leave that for later

[rachelhamlin]

Sounds great Roman. So user should see transactions from last N (50?) blocks, hopefully within a matter of seconds. As user scrolls, more transactions appear.

[rasom]

So user should see transactions from last N (50?) blocks

yep, that actually sounds very confusing. Let's just say that user should see N last transaction in a matter of seconds :)

[rachelhamlin]

@rasom how is this going? I think we'll start dogfooding a release candidate with the team & ambassadors before this fix is complete, but we will want it in the final build for users.

[rachelhamlin]

@churik @Serhy on my own account, Roman's PR seems to have fixed this. Are we happy to close it?

[churik]

yes