Go: add operation fee statistics endpoint

[tomerweller]

In case the network is busy, clients may choose to increase their proposed operation fee to guarantee submission. To make an educated decision, clients should be aware of operation fee statistics in latest ledgers.

This issue follows previous discussion in #293 and @nullstyle's suggestion.

I propose an /operation_fee_stats endpoint that will aggregate operation fees from transactions in recent ledgers and provide a statistical summary.

Arguments:

ledger_count : how many recent ledgers to aggregate stats from. Can default to 5 and should probably be capped (we don't want clients aggregating over 1000s of ledgers)

Response:

first_ledger (first ledger included in these stats)
last_ledger(last ledger included in these stats)
min_fee
max_fee
10th_percentile_fee
20th_percentile_fee
...
90th_percentile_fee

[nullstyle]

I think you're on a good track, but I'd like to propose something a little different architecture. I think now is the time to start to introduce real world dates and time ranges into the horizon "request api". That is to say, I'd really like to enable these sorts of future endpoints:

GET /days/2018-06-22/fee_stats # Show me the fee stats for June 22nd, 2018 according to the UTC timezone
GET /days/2018-06-22/ledgers   # Show me all the ledgers that closed (or perhaps overlapped) during the specified day

GET /weeks/2018-05/fee_stats   # same as above, but showing the aggregated results for the 5th week of 2018.  
GET /months/2018-02/fee_stats  # aggregated results for february 2018
GET /years/2017/fee_stats      # I think you get the point

IMO adding a new heirarchy of resources rooted at the timespan in concern adds a whole new layer of flexibility and discoverability to horizon. Thoughts?

---

I'm also wondering if "operation fees" provide the right guidance for transaction crafters. Since fees are collected at the tx level, I'm wondering what gaps we are asking our downstream users to cover between this resource and a signed, submitted transaction. On the surface it seems like the ask is "Multiply your chosen base fee by the number of ops in your tx", but I'm wondering if how we would judge the efficacy of that strategy. Any thoughts on other strategies a downstream user might take? Should we provide aggregations at both the TX and the operation level?

[tomerweller]

I like the idea of introducing more time series style queries but I have a few concerns around that. Mostly:

1. Do we have the infrastructure to support this? If not, are we willing to put the effort into building it?
2. Does this sit well with the vision for horizon?

And to a lesser extent, what's the value of adding these time ranges to the uri path as opposed to query parameters?

Don't get me wrong, I think that my endpoint proposal is a bit of an ugly, doesn't fit anywhere, kind of creature. Your approach is much more holistic.

Let's discuss on Wednesday's meeting?

---

Regarding ops vs. txs approach, It's hard to say. No wallet (that I'm aware of) currently supports modifying fees, so there's no downstream user experience to evaluate. Intuitively, I'd say provide ops level stats and let the clients do the math. Is there any significant difference in surge pricing behavior for multi-op transactions? @MonsieurNicolas?

[tomerweller]

A different issue, raised by @MonsieurNicolas last week, is that failed transactions are currently not ingested by horizon, which makes it difficult for clients to understand that insufficient fees are the reason for their rejected transactions.

[MonsieurNicolas]

I don't think failed transactions are related to this change: at the core layer, if a transaction is not rejected right away (because it meets the "minimum fee" criteria), but is not considered in a transaction set because of surge pricing, it will just be dropped by the network after some time. So the "symptoms" will be the same than a timeout.

[MonsieurNicolas]

As for the endpoint, I think the MVP could be to just expose /latest/fee_stats, maybe not even parametrized (for example, we always aggregate the last 30 seconds worth of ledgers for example - which allows us to do it in memory) and leave the door open to more complex time series in the future if people care about it.

[robertDurst]

I have begun working on this and want to lay out my thoughts for an mvp, so far I have discussed with @tomerweller and @bartekn but am posting here in case anyone else has feedback/input (largely based on what tomer suggested above). I think time, as proposed by @nullstyle will be useful, but am not thinking of making this part of the mvp.

Endpoint

/operation_fee_stats

Arguments

An optional ledger_count specifying how many ledgers worth of data to aggregate, with a cap such as 200.

Output

I am thinking we should return the per op fee.

I can think of three approaches of what to return:

1. A comprehensive breakdown of statistics as proposed by Tomer.
2. The smallest useful subset of these stats such as: mean, avg, minimum accepted, etc.
3. Enough data so that users can accurately calculate this data themselves, however they see fit. Thus, this would essentially be a sort of last X ledgers of transactions endpoint that returned only the relevant information per ledger such as:
   
   
   
   • fee_paid
   
   
   • number of operations
   
   
   • ledger number
   
   

Ultimately it comes down to whether calculations should be done by the horizon or the client.

[bartekn]

Ultimately it comes down to whether calculations should be done by the horizon or the client.

After thinking about it for a while I think that horizon should do the calculations and return a single statistic or even a fee interval that will likely be fine with the current fee market (something like https://bitcoinfees.earn.com/). In other words something like: "set the fee between A and B and your transaction will settle in 15 seconds, set it between C and D and will settle in 60 seconds...". In the beginning the algorithm behind it doesn't need to be super complicated (ex. average) but we can improve it over time.

I think that if someone wants to do something more complicated by themselves they can easily pull the latest 200 ledgers' transactions and construct an _online_ algorithm that keeps the cache of 200 (or more) ledgers in memory and just update the data as ledgers are closed. Stats won't help much if you want to do something more complicated.

[stanford-scs]

Please also include the baseFee from the latest ledger header, as this is crucial information. (E.g., if validators vote to raise the fee, the previous average might be lower than the new minimum.)

[bartekn]

I think that MVP from @robertDurst https://github.com/stellar/go/pull/586 is looking good. @MonsieurNicolas are you OK with me merging this, releasing in Horizon 0.14.0 rc4 and deploying to horizon-testnet?

[MonsieurNicolas]

this seems like feature creep, I'd rather only include stability fixes to turn 0.14.0 into a stable release and merge this for the next release.

[bartekn]

MVP #586 will be released in Horizon 0.15.0.