Rippled: Proposal: Reliable Transaction Submission in rippled

[mDuo13]

Rationale: Reliable/robust transaction submission is the biggest feature of Ripple-REST that isn't covered by RippleAPI. Despite the thorough documentation, this remains one of the hardest things to do properly when integrating a business with the XRP Ledger. Adding reliable transaction submission to a client library is infeasible since it involves running as a persistent service and persisting data (probably to a database). Those are both things that rippled does already. As an added bonus, putting this functionality in rippled gives us access to tools to conveniently backfill ledger history in case of an outage. We can reduce barriers to entry in the XRP Ledger ecosystem by building reliable transaction submission into rippled servers so businesses who run rippled always and automatically have access to a quality implementation of reliable transaction submission.

Prerequisites

To use this described implementation, users must have the following

• A way to generate UUIDs
• A way to track those UUIDs for future lookups
• Admin access to a rippled server
  
  
  
  • These features are admin-only for a similar reason to why wallet_propose is admin-only. They don't pose risks to the server administrator, but users would be very unwise to use them on servers that they don't control.
  
  
• The rippled server must have non-ephemeral storage for persisting reliable transaction submissions. This storage can be separate from the ledger store and shard store (and may not need nearly as many IOPS).

These requirements are very similar to the requirements for using Ripple-REST for reliable transaction submission, except that the user does not need to run a Node.js server with Ripple-REST.

Architecture

Add several new API methods for submitting transactions reliably and managing them. Because these API methods store secrets on behalf of the user, I suggest making them admin-only. Add a new data store, tentatively called the reliable submission store, for tracking reliably submitted transactions.

One or more backgrounds job should run as part of rippled to manage the state of any reliable transaction submissions whose current status is not final. These jobs' responsibilities include:

• Checking new validated ledgers to see if they contain the pending reliably-submitted transactions, and updating those transactions' status in the reliable submission store. (Updates occur when attempts at submitting the transactions are included in validated ledgers or exceed their LastLedgerSequence values, and when submitting follow-up attempts to transactions that failed initially.)
• Resubmitting reliably-submitted transactions up to the maximum number of attempts. This frequently involves changing some automatically-filled values and re-signing the transaction. (Note: transactions that result in tec codes are not retried, since the results of retrying such transactions are likely to be the same.)
• Backfilling ledger history so as to ascertain the outcome of reliably-submitted transactions, if there is missing ledger history. (For example, after restarting following an outage.)

API

This proposal adds several new admin methods to the JSON-RPC and WebSocket APIs of rippled. The methods are (names are placeholders):

• submit_reliable_tx: initiate reliable submission of a transaction
• get_reliable_tx: check the status of a reliably-submitted tx
• delete_reliable_tx: delete a reliable tx submission. (This does not directly affect the XRP Ledger data itself)

TODO: Maybe also a update_reliable_tx method to update the settings of a reliably-submitted (but not final) tx?

submit_reliable_tx

Submit a transaction for reliable submission.

The fields provided to this method are a superset of the fields supported by the sign method, except that the offline field of the sign method is not allowed. The reliable transaction submission system persists all the values from this request, including the provided secret/seed/seed_hex/passphrase, so that it can re-submit the transaction if appropriate. (Optionally, to reduce attack surface, rippled can delete the saved secret value when the transaction's outcome is final.) Certain fields in the tx_json may be omitted so that rippled can automatically fill them with appropriate values. Unlike with regular signing and submitting, rippled can change the auto-filled values if it needs to re-sign and re-submit the transaction. Fields explicitly specified MUST NOT be changed even for resubmission. (Exception: adding the reliable_submission_id to the Memos array.) See "Modifying for Submission" below for details.

_In addition to the sign parameters_, the user provides the following fields:

| Request Field | Value | Description |
|---|---|---|
| reliable_submission_id | String - UUID | An identifier provided by the client to track this transaction. If the transaction fails and must be re-submitted with a different transaction hash, the user can still identify it by this ID. This _must_ be a validly formed UUID. |
| max_attempts | Integer | _(Optional)_ Maximum number of times to submit the transaction if it is not included in a validated ledger. Defaults to 3. Cannot be less than 1. The initial submission counts as attempt 1. |
| ledger_index_offset | Unsigned Integer | _(Optional)_ How high to set the LastLedgerSequence relative to the _current_ ledger. Defaults to 3. |

When it receives the request, the reliable submission system does the following things in order: (Note: To avoid failures, it's important that the following occur in the proper order.)

1. It modifies the transaction for submission (see "Modifying for Submission" below) and signs the resulting transaction instructions.
2. It persists the reliable submission object to the reliable submission store, including the hash of the signed transaction. (See below for details of what's persisted.)
3. If and only if persisting the reliable submission object succeeded, it submits the transaction for consideration in the next open ledger or queue according to normal transaction submission.

The data persisted to the reliable submission store includes:

• All fields specified in the submit_reliable_tx request.
• The default values of request fields that weren't explicitly specified. This includes max_attempts, ledger_index_offset, fee_mult_max, fee_div_max, and build_path.
• The additional fields defined in the following table:

| Saved Field | Value | Description |
|---|---|---|
| submission_status | String | A string defining where the transaction is in the reliable submission process. When initially persisted, this has the value submitted. See submission_status Values |
| submitted_hashes | Array of Strings | An array of transaction hashes of attempts at submitting this transaction, ordered from newest to oldest, so that element 0 of the array is always the one most likely to succeed or have succeeded. The length of this array is the number of attempts made to submit the transaction so far, so it will never be greater than max_attempts. When initially persisted, this array is length 1, containing the hash of the first attempt (even before it has been submitted). |
| min_ledger_index | Number | One higher than the latest _validated_ ledger_index at the time the reliable transaction submission was received. In other words, the earliest possible ledger version in which this transaction could appear. |
| recent_last_ledger_sequence | Number | The LastLedgerSequence of the most recent attempt at submitting the transaction. |

Note: The maximum number of ledger versions the system may have to search to confirm a transaction's final result may be greater than ledger_index_offset because the auto-filled LastLedgerSequence is based on the _current_ ledger while the min_ledger_index is based on the latest _validated_ ledger. (This is by design; if there's a delay in consensus, there may be several closed, unvalidated ledgers, and in that case, it is unlikely but possible for a newly-submitted transaction to be included in those ledger indexes. To maximize chances of the transaction succeeding, we set the LastLedgerSequence based on the _current_ ledger, but to avoid failing we search as far back as the earliest ledger index it could _possibly_ appear in.)

get_reliable_tx

Look up the status of a reliably-submitted transaction.

The request contains just one field:

| Request Field | Value | Description |
|---|---|---|
| reliable_submission_id | String - UUID | The identifier of the reliably-submitted transaction to look up. |

The response contains the entire saved reliable submission object, including:

| Response Field | Value | Description |
|---|---|---|
| reliable_submission_id | String - UUID | The identifier of this reliably-submitted transaction. |
| tx_json | Object | The transaction instructions as provided in the original request. |
| secret / seed / passphrase / seed_hex | String | (One of these fields) The secret field provided in the request, in the same format as required by the sign command. TODO: Optionally, this field can be deleted automatically if the transaction has a final result. |
| key_type | String | The type of secret key provided. Either secp256k1 or ed25519. |
| build_path | Boolean | (Payment transactions only) Whether to automatically fill the Paths field of the transaction. Note: Unlike the sign command's current behavior, reliable transaction submission should use the value, not the presence, of this field. |
| fee_mult_max | Integer | Limit on how high the automatically-provided Fee can be. |
| fee_div_max | Integer | Divider for the fee_mult_max (1 if the request didn't specify it). |
| submission_status | String | A string defining where the transaction is in the reliable submission process. When initially persisted, this has the value submitted. See submission_status Values |
| submitted_hashes | Array of Strings | An array of transaction hashes of attempts at submitting this transaction, ordered from newest to oldest, so that element 0 of the array is always the one most likely to succeed or have succeeded. The length of this array is the number of attempts made to submit the transaction so far, so it will never be greater than max_attempts. |
| min_ledger_index | Number | One higher than the latest _validated_ ledger_index at the time the reliable transaction submission was received. In other words, the earliest possible ledger version in which this transaction could appear. |
| recent_last_ledger_sequence | Number | The LastLedgerSequence of the most recent attempt at submitting the transaction. |
| max_attempts | Integer | Maximum number of times to this transaction may be submitted. The initial submission counts as attempt #1. |
| ledger_index_offset | Unsigned Integer | How many ledger versions to allow between submitting a single attempt and failing that attempt (used for setting LastLedgerSequence values) |
| result | String | _(Omitted unless the submission_status is succeeded, failed, or rejected)_ The transaction engine code of the final attempt at submitting the transaction. If submission_status is succeeded, this is always tesSUCCESS. If submission_status is failed, this is always a tec-class code. If submission_status is rejected, this is either a tem-class code, tefPAST_SEQ, or tefMAX_LEDGER. |
| ledger_index | Number | _(Omitted unless the submission_status is succeeded or failed.)_ The validated ledger version in which this transaction appears. |

delete_reliable_tx

Remove a reliable submission object from the reliable submission store.

The request contains just one field:

| Request Field | Value | Description |
|---|---|---|
| reliable_submission_id | String - UUID | The identifier of the reliably-submitted transaction to delete. |

The response contains the last known state of the given reliable submission, in the same format as get_reliable_tx.

This method deletes the reliable submission object, but it does not affect the processing of any attempts to process that transaction that have already been submitted. It _does_ prevent future retries and aborts any backfilling that was necessary to determine the final status of this transaction.

Other Details

submission_status Values

The submission_status field of a Reliable Transaction Submission object has the following possible values:

| Value | Definition |
|---|---|
| submitted | The transaction has been received (and probably submitted) but its outcome is not yet final. This status applies before the transaction has been included in a validated ledger. When in this state, element 0 of the submitted_hashes array is the identifying hash of the currently-pending version of the transaction. In extreme cases, a reliable submission may be persisted in this state even if the attempt at submitting the transaction failed due to an outage. |
| queued | The transaction has been submitted or resubmitted, and the preliminary result of submission was terQUEUED, and the transaction has not yet been locally applied to any ledger version. If the transaction is included in an open or closed ledger (including as a result of switching to a different closed ledger received from peers in consensus), the status changes to submitted or resubmitted as appropriate. TODO: maybe remove this status in favor of just using submitted? |
| resubmitted | The transaction did not succeed on a previous attempt, but it encountered an error that could be retried, so it has been modified as necessary, signed again, and submitted again. As with the submitted status, element 0 of the submitted_hashes array is the currently-pending transaction hash. Later elements of the array are the hashes of previous submission attempts. TODO: maybe remove this status in favor of just using submitted? |
| succeeded | The transaction has been included in a validated ledger with a tesSUCCESS result. In this case, element 0 of the submitted_hashes array is the identifying hash of the transaction that succeeded. |
| failed | The transaction has been included in a validated ledger with a failed transaction result (a tec-class code). It will not be retried. |
| rejected | The transaction has not been included in a validated ledger and it will not be retried. This includes the following cases: (1) any "Malformed" transaction (a tem-class code), (2) the transaction had an explicitly-specified Sequence value but resulted in tefPAST_SEQ, (3) the transaction has been attempted max_attempts times and the last attempt has a LastLedgerSequence higher than the latest validated ledger version, or (4) the transaction instructions contain an explicitly specified LastLedgerSequence parameter which is lower than the latest validated ledger version. |
| unknown | Due to a gap in ledger history, the final outcome of the transaction is not known. If it has any reliably-submitted transactions in this state, rippled tries to backfill ledger history to determine the final outcome. This may occur if rippled was stopped or lost power when a transaction's outcome was pending. |

Caution: There's a slight difference between case 3 of rejected and a tefMAX_LEDGER result. Any given attempt at submitting the transaction can fail with tefMAX_LEDGER but the reliable transaction submission may not have failed finally if the transaction can be modified and submitted again. In other words, if max_attempts is higher than the number of attempts made so far _and_ the transaction instructions do not include a hard-coded LastLedgerSequence parameter, a tefMAX_LEDGER result is not final. Another case in which tefMAX_LEDGER is not final is if it results from a closed-but-not-validated ledger version that's higher than LastLedgerSequence. If a different version of a closed ledger is validated by consensus, the transaction could still become included in a validated ledger.

The following state diagram shows the possible transitions of submission_status values (for the proposal as written and for the variant where resubmitted and queued are removed):

TODO: It's worth discussing how unknown status should interact with ledger history and online delete. The most user-friendly behavior, assuming these commands remain admin-only, is that the server should automatically backfill ledgers even in excess of the desired number of historical ledgers to save, and online delete should not delete ledgers in the range of min_ledger_index to recent_last_ledger_sequence (inclusive) while a reliable submission's status is unknown. After the transaction's outcome is confirmed, it would be OK to delete those ledgers, but the ledger containing the transaction should not be deleted.

Modifying for Submission

Before the transaction is signed and submitted, rippled automatically fills certain fields the same way it auto-fills them when doing online signing. In addition to that, reliable transaction submission automatically adds the following:

• A memo containing the reliable_submission_id. If the transaction already has a Memos field, this memo is appended to the Memos array. See "Memo Format" for details.
• A LastLedgerSequence field with a value equal to the current ledger index plus the ledger_index_offset from the request.

The reliable transaction submission system does not persist the automatically-provided fields in the reliable submission store. This is to distinguish between explicitly specified fields (which are not modified) and automatically-filled fields (which can be modified when resubmitting the transaction).

If a reliably-submitted transaction fails to be included in a ledger, rippled can attempt to resubmit the transaction. This may require modifying certain fields of the transaction. This should always be based on the persisted tx_json instructions, not the signed instructions from any given attempt. In all cases except the Memo field, the modified fields MUST NOT overwrite fields that were explicitly defined in the tx_json of the original request.

• The auto-filled LastLedgerSequence should be modified if a previous attempt failed to be include the transaction in a validated ledger before its LastLedgerSequence. The new LastLedgerSequence value should be equal to the _current_ ledger plus the persisted ledger_index_offset.
  
  
  
  • Note: In many cases, the new LastLedgerSequence will be equal to the old LastLedgerSequence plus the ledger_index_offset. However, this is not true in case of power outages or busy servers, where additional time may pass between when the initial submission fails and when it gets resubmitted.
  
  
• The auto-filled Sequence field can be modified if the transaction results in tefPAST_SEQ. This is unlikely to occur unless a user is submitting transactions on multiple systems in parallel.
• The auto-filled Fee field can be modified if the provisional result of the transaction was a failure due to an insufficient transaction cost. The new Fee value must not be higher than the amount defined by the fee_mult_max and fee_div_max values of the initial request. Result codes that should prompt an increased transaction cost include telCAN_NOT_QUEUE, telCAN_NOT_QUEUE_FEE, telCAN_NOT_QUEUE_FULL, telINSUF_FEE_P, terINSUF_FEE_B. TODO: Maybe this should just happen automatically based on the expected transaction cost at the time of resubmission and not based on the result of the previous submission, since transaction costs may increase coincidentally around the time of resubmission even if the previous attempt didn't fail for transaction cost reasons?
• The auto-filled Paths of a Payment (if build_path was provided) can be updated at the time of resubmission.
• No other fields should be modified when resubmitting. (The Memos field should still be modified to include the reliable_submission_id memo.)

Note: The user may explicitly provide some auto-fillable fields, such as Fee, Sequence, or LastLedgerSequence. Those fields must never be modified, even though it may eliminate some opportunities to resubmit a transaction. For example, if the user provides a Sequence field and the sending account's Sequence field in a validated ledger is or becomes higher than the reliably-submitted transaction's Sequence field before even the first attempt is included in a validated ledger, the reliable submission object goes to the rejected status rather than being resubmitted with a higher sequence number.

When resubmitting, the reliable submission system should do the following steps in order (similar to the initial attempt):

1. Update the auto-filled fields and sign the updated transaction instructions.
2. Persist updates to the reliable submission object in the reliable submission store. In particular:
   
   
   
   • Prepend the new transaction hash to the submitted_hashes array.
   
   
   • Update the min_ledger_index to the latest validated ledger index plus one.
   
   
   • Update the transaction's submission_status to be resubmitted.
   
   
   • Update the recent_last_ledger_sequence with the LastLedgerSequence of the updated transaction.
   
   
3. If and only if persisting the updated reliable submission object succeeded, submit the updated transaction for consideration in the next open ledger or transaction queue according to normal transaction submission.

Memo Format

When transactions are submitted using this mechanism, rippled adds a memo with the reliable_submission_id so that you can recognize when a transaction is the result of a particular reliable-transaction-submission. The memo has the following format:

| Field | Value |
|---|---|
| MemoData | The reliable_submission_id UUID, in a binary big-endian encoding. (_Not_ Microsoft's COM/OLE mixed-endian encoding.) |
| MemoFormat | The value 0x55554944 (the ASCII for "UUID"). Note: Surprisingly, there isn't a MIME type for UUIDs. |

[BobWay]

I'll give this a close read, but I highly support revisiting better ways to do reliable transaction submission.

[mDuo13]

(Made slight edits to fix typos, add a link to the fields of the sign method, and clarify that the reliable submission store should not be on ephemeral storage.)

[sublimator]

Yeah, ugh. And add FirstLedgerSeqence symmetric to LastLedgerSequence for a
simple mental model, and don't cry about 4 bytes while using 160 bits for
"USD" and endless 0s for inner nodes :)

I recommend a dumb peer (https://github.com/ripple/rippled/issues/2413)
implemented on the JVM, that just tracks state from validators, and does
not process transactions. With a plugin system for user commands
(bots/whatever) and data processing :)

And useable as a library, and with a Pony with a ripple tattoo

[sublimator]

Totally agree this should be in rippled, and yes, impossible to put RTS in
client libs without handling persistence of txns before putting on the
network for handling crash recovery.

Nice work pushing this Rome.

What is a txn id???

[mDuo13]

I'm not sure I understand everything you said, @sublimator. Are you suggesting that FirstLedgerSequence should be added as a common transaction field? I think I could get behind that. It's not necessary for this proposal, but would be convenient for it.

Also, I'm adding a ledger_index field which should appear in the result when the transaction is confirmed into a validated ledger.

[BobWay]

noticed one typo in submission_status "unknown". Looks like the beginning didn't get deleted.

The transaction's Due to a gap in ledger history,

[BobWay]

On thing I found counterintuitive is that
submit_reliable_tx - is a superset of submit
get_reliable_tx - is not a superset of tx

It looks like you need to first get_reliable_tx(), then tx(submitted_hashes[0])
Is that what you are thinking?

[BobWay]

This looks awesome Rome. I highly support it!
If building this into rippled itself turns out to be troublesome, then maybe build it as an optional executable. One that's packaged with rippled and designed to run on the same box.

[mDuo13]

On thing I found counterintuitive is that
submit_reliable_tx - is a superset of submit
get_reliable_tx - is not a superset of tx

It looks like you need to first get_reliable_tx(), then tx(submitted_hashes[0])
Is that what you are thinking?

Yes, that is what I was thinking. It makes some kind of sense to change get_reliable_tx to be a superset of tx so it returns the tx response somewhere rather than just pulling certain fields out of it, if the transaction is in a validated ledger. I guess you only ever have one reliably-submitted transaction actually end up in a ledger, so you wouldn't need to worry about including multiple "tx" responses. You _would_ want to be extra clear and only include the tx response if the transaction is in a validated ledger.

I'll think on this and possibly update the spec to do that.

[ximinez]

This may be bike-shedding at this point, but I think the status name failed is going to cause confusion with people who don't read documentation. feeClaimed seems more intuitive to me.