Go: Horizon missing historical failed transactions

[tyvdh]

What version are you using?

{
"horizon_version": "1.0.1-2c2d8fb0c3ae0516dcbbb39f189ca00731c9c8c3",
"core_version": "stellar-core 12.4.0 (c0136139802ac1a1ac8899424e7888fa9366414e)",
"ingest_latest_ledger": 28747380,
"history_latest_ledger": 28747380,
"history_elder_ledger": 1,
"core_latest_ledger": 28747380,
"network_passphrase": "Public Global Stellar Network ; September 2015",
"current_protocol_version": 12,
"core_supported_protocol_version": 12
}

What did you do?

Trying to add up the visible transaction, event and operation data for GAMFIL3N6PN5JSYF2DN3M5BJIXRA6AWZFFI3ZMYMNSIWC3CKGN4WE2ZL account and there's a 300 stroop difference between the account balance what what you get adding up all the transaction data.

What did you expect to see?

You should be able to "replay" the transactions to equal the balance so you know how you arrived at the balance you have. What's the point of a blockchain if blocks are missing or incomplete?

What did you see instead?

My assumption is there are missing failed transactions for GAMFIL3N6PN5JSYF2DN3M5BJIXRA6AWZFFI3ZMYMNSIWC3CKGN4WE2ZL as there are 300 stoops which are unaccounted for in Horizon for this account. There are some gaps in the sequence number for this account as well which would indicate the same.

[tyvdh]

For context the CEX.io exchange reported this issue and provided a TypeScript file for reviewing the issue. http://tyler.link/iwWWv0

ts-node is a great cli tool for running this file: https://www.npmjs.com/package/ts-node

[abuiles]

Quick note here since we just release Horizon 1.0.1 -- we get the same result in 1.0.0.

[ire-and-curses]

Slack thread - Bartek thinks we need to migrate the DB. I'm not sure exactly what this entails, and why we didn't do it already.

[abuiles]

I reingest from ledger 16002687 to 16093631 (which is the ranges where we have some missing txs) and found the failed transactions. After running @tyvdh's script, the balance adds up now

Balance parsed 1059795230
Balance from daemon 1059795230
Balance difference is 0

This file https://gist.github.com/abuiles/026d8282fe83883f396b012e366ff9fa contains all the txs

I checked against Satoshipay public horizon and they also have the same issue.

@ire-and-curses I think we should move this issue out of Horizon and start a conversation with ops about running a full ingestion in our public nodes.

[ire-and-curses]

Yeah sounds good.

[ire-and-curses]

Before asking ops to do this, we should verify the process and expected workload on one of our staging servers. So keeping this here until we do that (after testing for current Horizon release).

[abuiles]

@ire-and-curses do you mean doing a full re-ingestion on staging?

[ire-and-curses]

Yeah. I think we have to anyway otherwise we'll get false positives from our verify checks right?

[ire-and-curses]

This has been reingested, and we have the DB in staging.

We should run Go replay on staging and confirm performance is still ok.

We should confirm missing transactions are present in staging.

Then work with ops team to get this deployed. Confirm hydration is complete before checking performance and enabling in production.

[bartekn]

Heads up, we're going to switch the production cluster to reingested DB (with failed txs) on August, 12th.

[bartekn]

This is complete. Production DB contains all transactions now (including failed).