Go: Friendbot fund is 200 but account GET is 404

[tyvdh]

What version are you using?

"stellar-sdk": "^4.0.0"

What did you do?

Funded and then immediately attempted to GET an account.

http://tyler.link/z79z3k

You can replicate this issue here:
https://hack-stellar.now.sh/

Just click "Generate Keypair" and then "Fund Account".

Note it won't happen consistently, you may have to repeat the above sequence several times before you get your first 404.

What did you expect to see?

A successful 200 response for the account GET request

What did you see instead?

A 404. If the funding was successful you shouldn't get a 404 no matter how soon you make subsequent requests.

[leighmcculloch]

After some period of time does Horizon return 200 OK for the account? How long does it take?

Horizon ingests data from the network and there's no guarantee that it ingests the last ledger so fast that there is no period of time between the account existing and Horizon knowing it exists. Ideally this period of time is small but it is eventually consistent, not strongly consistent.

[tyvdh]

After some period of time does Horizon return 200 OK for the account? How long does it take?

Yes. Quick. Maybe ~1 second though I haven’t tested that specifically. My complaint is less that core and horizon aren’t in sync and more that friendbot should wait to respond until we’re sure a GET on that account will be 200. However I’m thinking now this is perhaps unnecessary as we’ll need to be building for these time gaps in every horizon POST, PATCH type call. No need to make an exception here.

I am curious though if we track the time gap and have any guarantee or even just average, max times that those gaps can be/have been. ~1 second is very different than ~30 seconds. Curious what affects that time and how susceptible it is to variation. Hard to build for inconsistency especially when a 404 could mean either doesn’t exist or just doesn’t exist yet... but it might... in a second or 3. Big difference between an out-of-sync 404 and an in-sync 404.

[bartekn]

Thanks for the report. Yes, this seems important and I suspect the data source for /accounts/{id} is to blame. Previously this was read from the Core DB, now (1.0.0-beta) we use ingested data in horizon DB.

The issue you are experiencing is happening because in txsub we check both sources: Core and Horizon DBs to check if tx was added to a ledger. It's possible that the latest ledger is still being ingested by Horizon thus 404. Unfortunately, we can't just remove Core DB check because some Horizon instances don't ingest failed transactions. I can see two fixes, long term and short term:

1. Long term fix is to always ingest all transactions (including failed). This may increase storage needs but failed transactions are essential for some components like fee stats or txsub. Because we start testing 1.0.0 next week I don't think we'll be able to properly fix it before 1.0.0.
2. To fix this before 1.0.0 we can skip Core DB check here:
   https://github.com/stellar/go/blob/eb2ef0e9e1a6acba8218c7bd525c0f53a53dda8f/services/horizon/internal/txsub/results/db/main.go#L41-L57
   if a Horizon instance is ingesting failed txs. This way txsub will keep quering Horizon DB until the latest ledger is ingested.

[ire-and-curses]

I agree with @bartekn's analysis. Doing fix 2 now to help some instances makes sense to me.

Before implementing the long term fix, we would need to assess the storage/query performance impact.

[tomerweller]

:+1: on including failed transactions by default. they're part of consensus and shouldn't be ignored

[bartekn]

Quick fix ready: https://github.com/stellar/go/pull/2277

[bartekn]

@tyvdh the fix has just been deployed to https://horizon-testnet.stellar.org/. Please check it out.

[tyvdh]

Can't get it to break so seems fixed! Very nice, thanks!

[bartekn]

Great, thanks! Let's keep it open so we can continue working on the long term fix in the future.

[bartekn]

Horizon 1.5.0 ingests all transactions (successful and failed). I think it should solve the issue.