Rippled: Proposal: All Offers Expire

[mDuo13]

One of the long-term challenges for the XRP Ledger is managing the unbounded growth of ledger data size. History is one thing—it's inevitable that full historical data will grow indefinitely—but the size of the current state also has a tendency to get larger under normal usage as it accumulates ledger objects that no one wants or needs anymore—in a word, cruft.

One way we can get rid of some cruft is to make it so that all Offers in the decentralized exchange eventually expire, after which point they can be removed. Right now Offers only expire if their creator explicitly sets an Expiration time on them. A simple change that would improve the situation is to change the OfferCreate transactor so that if you omit the Expiration time, it gets set to a sensible default value such as 90 days. This alone could dramatically reduce the "incidental" introduction of cruft in the ledger.

There are lots of issued currencies in the XRP Ledger that are no longer used and as long as the XRP Ledger continues to exist it's likely that more issued currencies may pop up and die off. It's pointless to keep offers for these currencies around when they're no longer in use. There are also lots of dormant accounts that may never be used again; and even though it may be too hard to know which accounts can be reasonably deleted, there should be no problem with deleting those accounts' offers even in active markets. To my understanding, it's pretty normal for an exchange to have a maximum lifetime for any single offer. Most of the time the market is expected to move often enough that you don't _want_ an offer to hang around that long.

We could also look at setting explicit upper bounds for Expiration so you can't, say, set it to expire at MAXINT to make an effectively immortal offer. I think 1 year seems like a reasonable "longest offer term" value. If you can't be bothered to update your offers annually I don't know what you're doing. I guess a possible use case for immortal offers would be if you issued a currency and then intentionally blackholed your account?

[intelliot]

I would be fine with setting the upper bound for Expiration to 90 days. For example, Robinhood has only two options for "Time in Force". Orders expire either "Today" (at after-hours close) or in 90 days ("Good till canceled" is the term used -- even though they actually expire in 90 days unless cancelled before that).

One concern is that "old" offers are difficult to find (so that they can be removed). One approach for this could be to have a new transaction type that can only delete expired offers from the books. Similar to finishing an escrow that has passed its FinishAfter time, anyone (a "good samaritan"?) could help reduce ledger cruft by clearing out expired offers in this way.

[MarkusTeufelberger]

This proposal spams the transaction database vs. the ledger state. I'm not sure that this is a good trade-off, but it might be worth it after all - RAM is more expensive than disk at the moment.

I share Elliot's concern about the actual cancelling mechanism, though I find the "good samaritan" approach of Escrows already rather kludgy - and with offers there's nearly no incentive for anyone other than server operators to actually remove outdated offers.

I also have concerns about "magic values". What makes 90 days, a year or a decade (or 10 minutes) special or worthy enough to write into the core of the protocol, force cancelling an offer after that time?

If you are concerned about ledger state growth, a "swap" mechanism that moves rarely used portions of the state trie on slower storage might be a more general solution that doesn't need such invasive measures.

[mDuo13]

That's a good point that this hits the transaction database harder, since it means there will probably be more metadata for Offers that have been canceled. You mentioned that disk is cheaper than RAM "at the moment"—but realistically, disk will always be cheaper than RAM; if it weren't, people wouldn't use disk. But more importantly, the transaction database doesn't suffer from the same scaling problem as the ledger state. Full history is not necessary to process any future transactions; full state is. So it's absolutely achievable (even today!) to archive transactions beyond a certain age, whereas the tech to find and archive portions of the state trie that aren't in use is quite a bit more complicated and may not work at all for some possible ledger states.

I'm sympathetic to the issue of magic numbers, but at some point when designing a system you just have to _make choices_ so you aren't paralyzed by indecision. I could see the maximum offer expiration being set on a votable parameter like the fee and reserve settings, but I honestly don't think it's important enough. If people later decide they want more flexibility in max offer expiration, a follow-up amendment can change it.

As for how the expired offers get deleted from the books: worst case scenario, no one does anything and they sit around untouched, which is exactly the current status quo. So while we can debate possible improvements, I don't think that should be a showstopper for implementing global offer expiration, because it's _still_ an improvement in terms of keeping the ledger size in check.

On that subject, though, here's roughly how I would make the cleanup transaction work.

• Transaction Type (tentative name): CleanUpBook (normal transaction¹)
• Fields: Common fields, plus TakerGets and TakerPays.
  
  
  
  • The value amounts in these fields would be ignored. Otherwise these fields match the equivalent fields in OfferCreate.
  
  
• Effect: Finds the order books that would be matched by a theoretical "OfferCreate" with the same TakerGets and TakerPays currencies, except that it treats TakerGets (the amount being offered) as infinite and TakerPays (the amount being asked in return) as infinitesimal. Thus it walks through the matching books starting at the best possible rate of return. Whenever the transaction finds an unfunded² or expired offer, it deletes that offer and moves on. It stops after it has accumulated a maximum amount of metadata (so probably something like 1000 or 2000 offers like tecOVERSIZE), or if it has finished walking through all the books for this currency pair.
  
  
  
  • To prevent excessive work, it adds a LastCleaned timestamp to each offer directory it crawls. If the offer directory already has a LastCleaned timestamp that is more recent than some predefined interval (24 hours? 7 days?) then the transaction skips over that book entirely.
  
  

I have a feeling this transaction should have a higher than normal cost or have some other restrictions. It's likely a very intensive transaction to process in both RAM and CPU. Worse, running the transaction multiple times on the same book in quick succession provides little to no benefit but costs as much or more work as doing it on a very "cluttered" book where it removes a lot of expired offers.

¹ I considered making a pseudotransaction that servers could automatically propose, but I couldn't figure out an elegant mechanism for how they'd know which markets to submit cleanup transactions for. The nice thing of making these kinds of cleanup functions "normal transactions" is that someone can step up and handle the chore of doing it without burdening "ordinary" servers in the network with keeping track of when and how to do it, so it could be done entirely by one entity who's very good at it. The downside, of course, is the Tragedy of the Commons. Theoretically the community could donate to someone who does this sort of work to compensate them for the costs of inspecting the ledger for things that need cleanup, or something like that. Basically at some point the ledger may need a formal waste management utility or something but its structure and management don't need to be written into the XRPL protocol.

² There are likely some weird edge cases around declaring offers unfunded when the same person has placed offers at different depths of the same book. For example, a trader might have a small "front" offer at a cheap rate and a larger "backstop" offer at a more profitable rate. I'd like them to have the flexibility to keep the backstop offer on the books in case they cancel or replace their cheaper front offer. To achieve that, I think I'd like the CleanUpBook transaction to consider an offer unfunded only if the owner has _none_ of the offered currency, which should be evaluated independently for each offer.

[RareData]

I would prefer an approach using familiar concepts:

1. Change the OfferCreate transactor so that if you omit the Expiration time, it gets set to a sensible default value (90 days), as proposed by @mDuo13
2. Change the OfferCancel transactor to include a mandatory Owner field. Offers and Escrows already share the OfferSequence field. Allow expired offers to be cancelled by anyone, just like escrows.

On another note, why are Offers worthy of a dedicated cleanup transaction, but not PaymentChannels, Checks and Escrows?

[mDuo13]

I like the idea of OfferCancel having an Owner field so you can cancel other people's (expired/unfunded) Offers¹. It could be optional, and if omitted the transaction sender is the owner (just how the transaction works today).

¹ I also like the idea of having a transaction I can use to cancel other people's (non-expired, funded) offers. But that's only for me, not the rest of y'all. 😜