Go: Friendbot slows down under heavy load along with increased `tx_bad_seq` errors

[nikhilsaraf]

you can see the stretch where I issued 11 in quick succession (req. num. 4 - req. num. 14), the response time slows down from 10.79s to 40s resulting in 3 failures caused by tx_bad_seq. When I re-issue requests it goes back to the 7-13s range until I start bombarding it agaon.

friendbot currently spawns new goroutines for every submission and also assumes that transactions will be submitted in the order in which they were created. These threads are probably taking a toll on the machine and also resulting in tx_bad_seq errors because of message ordering.
We are already locking before creating a new transaction so we only create one request at a time and issue only 1 seq number at a time. we submit using multiple goroutines so we’re not bottlenecked on ledger close times.

tx_bad_seq has been friendbot’s biggest enemy (because of high throughput requirements). we’ve considered using channels in the past etc.

The locking logic explains the slowness when issuing multiple requests. Although I'd imagine it's not that expensive to just make a tx and submit asynchronously so the amount of slowness is unexplained.

possible solutions:

• [ ] we should consider making use of stellar channels so we can have different pools of seq numbers to consume. I think if we were to have 10 such stellar channels that would be plenty. they wouldn’t need more than 100 XLM at any given point in time. we can also have friendbot itself monitor it’s own channel accounts and fund it if any of them ever get low. If we don’t want to manage these channel accounts, we can always have friendbot create these throwaway accounts on startup so we don’t have to manage their keys.
• [ ] batch requests into 2s chunks so we submit multiple ops / tx (susceptible to DoS attacks by pre-creating an account submitting to friendbot after inclusion in the batch, so tx fails all ops)

startup command can add a -n or --num-channels flag which specifies the total number of channels to initialize friendbot with. We can linearly initialize each channel account in batches of 100 each synchronously to keep things simple. We could artificially limit max number of channels to 1000 using a constant value so startup time is within 1000/100 * 5 seconds = 50 seconds. The faster way to initialize would be to first initialize with 100 channels and then use those hundred channels to then create more channels in a geometric fan-out model. We don't need that kind of scale with friendbot right now.

We would also need a background thread that monitors these source accounts. If any of them ever fall below a certain threshold, say 20 XLM, then this background thread can "top-up" these source accounts with a transaction that does not use a source account. This thread can run once per hour.

[debnil]

Thanks to @tomquisel and @nikhilsaraf for some great discussion; thought I'd post a short sketch of an approach before tackling this issue. The original issue outlines two possible approaches: (1) using multiple Stellar channels to submit transactions, or (2) batching requests. As mentioned above, the second is susceptible to denial-of-service attacks, so let's explore the first.

Let N be the total number of Stellar channels to use on friendbot. The user can define this value through a new flag on the startup command. The maximum possible value can be determined through experiments/benchmarks. The issue suggests 1000, with channels initialized synchronously in batches of 100, resulting in a total startup time of 50 seconds - that seems like a reasonable starting value.

A long-running goroutine would receive an account ID to fund, choose a Stellar channel, and send it to that channel. The channel would be chosen in a manner that appropriately balances load across them. Accordingly, each Stellar channel would have a corresponding Go channel to receive account ID's, and it submits payments to those ID's from the Stellar channel's source account. Each source account can start out with a balance of 500 XLM. We'll keep these source accounts only in-memory, so there won't be cleanup. Plus, we don't need to track their secret keys, because they're testnet lumens, which reduces operational overhead. Finally, a background goroutine would periodically monitor the channel source accounts and tops up their balances when below a certain threshold, say 100 XLM; this can run once per hour.

[bartekn]

I'd like to comment on a couple of things that caught my eye:

These threads are probably taking a toll on the machine

Go routines are very "cheap". The peak we observed last week was 168 requests per minute. Such traffic really shouldn't affect performance:

resulting in tx_bad_seq errors because of message ordering

Horizon should handle this, it has tx ordering built in. If a new transaction appears and it's sequence number is higher than account sequence + 1 it will wait for a couple of seconds for previous transactions.

possible solutions:

We shouldn't do batching as it's vulnerable to DoS, as explained in the original comment. Channels seems fine.

[nikhilsaraf]

@debnil looks great. We could probably select the Stellar channel using a round-robin approach for now.

Testnet will be reset tonight so we will probably get a lot of load on friendbot and a lot of data from that too, just something to keep in mind.

[nikhilsaraf]

@bartekn thanks for that context. interesting to know about horizon message ordering. would that still work if we hit different app servers for horizon?

I'm wondering if in that case it makes sense to just remove the lock that we have on friendbot (to synch seq number from friendbot account) and whether that would fix everything for us?
https://github.com/stellar/go/blob/master/services/friendbot/internal/friendbot.go#L45
It would certainly help with requests waiting on the one before which is provably causing a slowdown right now, maybe that kind of coordination isn't really needed?

[bartekn]

would that still work if we hit different app servers for horizon?

Good point, it probably doesn't unless there's are _sticky sessions_ configured on the load balancer. Even if it is configured, it doesn't work outside browser (or if _cookie jar_ is not added to server side code - _sticky sessions_ use cookies).

[tomquisel]

@debnil that plan sounds good to me! Just to double check:

• each Stellar channel has its own source account for txns it submits, but the payment comes from the single central friendbot account.
• as a result the txns will be signed by both the source account and the central friendbot account
• secret keys for the source accounts are stored ephemerally in memory, and if friendbot restarts it just creates new source accounts

Given that there are 100B XLM and friendbot funds 10k at a time, there's a max of 10M transactions that friendbot could possibly execute. You can probably initially fund source accounts with enough to exceed that theoretical maximum (100 XLM / num_channels should do it with constants as they are now), and save the effort of the background account topper-upper.

[nikhilsaraf]

^yes on all 3 points^