Go: Implement in-memory ingestion ledger backend

[ire-and-curses]

The ingestion prototype needs a full ledger reader to support use cases where a pipeline needs information more frequently than between checkpoints, and/or needs transaction data and metadata.

• Original ingestion design plan
• Related HistoryArchiveAdapter component
• Pipeline prototype
• Rough XDR format for needed core data
• Stellar core data tables description

Ultimately, it is hoped that stellar core will produce XDR files that can be stored offline (e.g. in S3) and accessed on demand. However, for rapid completion of the prototype ingestion pipeline, this implementation plans to access stellar core's database directly, as is currently done by Horizon.

The minimum interface provided by ledgerBackend is

GetLatestLedgerSequence()
GetLedger(sequence)

We won't provide a GetEarliestLedger() method unless it becomes clear this is a requirement for callers.

We won't provide an Ack(sequence) method to allow a caller to indicate data is safe to delete. Instead, when the ledgerBackend passes data upstream to the pipeline, it updates the cursor on stellar core, indicating that older data is now safe to delete. This is slightly less robust in the case of upstream errors, but much simpler for a prototype. Additionally, in the expected future case where the backend is a long-lived file system, an Ack(sequence) method would be redundant.

So the rough steps to implement are:

• [x] Evaluate Horizon approach to extracting stellar core data
• [x] Create internal Go data structures mirroring data schema
• [ ] Populate Go data structures dynamically as data becomes available in stellar core DB
• [x] Implement ledgerBackend interface
• [ ] Implement cursor update
• [ ] Test ledgerBackend interface using mock input
• [ ] Test ledgerBackend interface using real input
• [ ] Create simple command line tool to exercise backend with real input
• [ ] Connect ledgerBackend interface to prototype pipeline and test

[ire-and-curses]

Questions:

1. Are stellar core DB updates for a particular ledger atomic (i.e. can we assume that if some ledger data is available, it all is)?
2. How does the ledger backend know that data for ledger N is complete and available for retrieval? Do we just poll for new entries?
3. Is it correct that the cursor updates on calls to GetLedger()? An alternative is to read continuously into memory and update cursor as we do. This would be fragile to ledgerBackend failing, but removes responsibility from the caller to request ledgers in order from earliest. I am assuming that would use too much memory though (otherwise presumably there would be no need for SC to garbage collect in the DB) - is this correct?

[bartekn]

Are stellar core DB updates for a particular ledger atomic (i.e. can we assume that if some ledger data is available, it all is)?

Yes.

How does the ledger backend know that data for ledger N is complete and available for retrieval? Do we just poll for new entries?

In the prototype we need to check if new ledger is available in ledgerheaders table. Maybe we'll have a notification service provided by stellar-core in a future but it's not there yet and not certain something like this will be created.

Is it correct that the cursor updates on calls to GetLedger()? An alternative is to read continuously into memory and update cursor as we do. This would be fragile to ledgerBackend failing, but removes responsibility from the caller to request ledgers in order from earliest. I am assuming that would use too much memory though (otherwise presumably there would be no need for SC to garbage collect in the DB) - is this correct?

For this prototype I think we should update cursor when requested ledger data is read. Reading into memory doesn't really save the problem because in case of crash data in memory will be lost as well. Ultimately we may add ack method you mentioned but maybe we'll find another way.

[ire-and-curses]

@tomquisel @MonsieurNicolas @nikhilsaraf PTAL at the above and let me know if something doesn't match what you're expecting. I've been looking at Horizon's implementation and will start the real code today.

[tomquisel]

It looks good to me! I think polling is the best approach available. I'd also say don't be shy to bump up the polling rate to something like 10x a second. Those queries should be extremely cheap and it's valuable to avoid latency.

Another thought is about those data structures mirroring the schema: do those already exist in Horizon? Or do they potentially get generated from the .x files?

[tomquisel]

Oh, I'd also say add a super-simple command-line tool that exercises the ledgerBackend. Nikhil is doing / has done something similar for the history archive reader, so you should be able to build off of that. Something extremely simple, like you give it a core db and it just prints out the number of succeeded / failed txns each ledger.

Credit to @MonsieurNicolas for that idea

[bartekn]

I believe this was closed in #1404.