Rippled: Unify JSON serialization format of transactions

[mDuo13]

The many API methods that can return transactions do so in a very inconsistent manner. See the following table for a breakdown of where different fields are located as of rippled v1.0.1.

Note: In the following table, the "top" level is with respect to the individual object representing the transaction; for methods that return arrays of transactions, the "top" level is per element of the array. "Transaction Instructions" refer to the canonical fields of the transaction as defined in the transaction format, such as the Account and Flags fields.

| Method | Transaction Instructions | Transaction Metadata | hash | ledger_index | inLedger | validated | ledger_hash | Binary blob | date |
|----|----|---|---|---|---|---|---|---|---|
| ledger (with transactions expanded) | Top level | metaData field (JSON)
_or_ in meta field¹ (binary) | Top level | Above transaction level | None | Above transaction level | None | tx_blob field¹ | None |
| tx | Top level | meta field | Top level | Top level | Top level | Top level | None | tx field¹ | Top level |
| tx_history (deprecated) | Top level | None | Top level | Top level | Top level | None | None | Not supported | None |
| account_tx | tx field | meta field | tx field | tx field | tx field | Top level | None | tx_blob field¹ | None |
| transaction_entry (would like to deprecate) | tx_json field | metadata field | tx_json field | Top level | None | Top level | Top level | Not supported | None |
| sign, sign_for, submit, and submit_multisigned | tx_json field | (N/A) | tx_json field | (N/A) | (N/A) | (N/A) | (N/A) | tx_blob field | None |
| Streams from subscribe | transaction field | meta field² | Top level | Top level² | None | Top level² | Top level² | Not supported | transaction field |
| Data API v2 methods | tx field | meta field | Top level | Top level | None | None (Data API _only_ reports validated transaction data) | None | tx¹ | Top level (as ISO8601 timestamp, not Ripple time) |

¹ Only if the request asked for binary data
² The transactions_proposed stream omits these fields because the transactions' outcomes are not yet final.

Recommendations

I recommend changing all of rippled's methods to serialize transactions to JSON in a single consistent format. The format I suggest is essentially the same format the Data API uses, with a couple modifications to accommodate for some formats only the rippled APIs handle.

Specific recommendations:

1. Always use tx_blob for binary format. Always use tx for JSON format transaction instructions.
2. Only include the canonical transaction instructions ("uppercase fields") and signing fields in the tx field with transaction instructions. Move all "lowercase" fields like hash and ledger_index out to the top level of the transaction. (Note: the issuer/currency/value sub-fields of amounts are canonical fields even though they are lowercase.)
3. Always use meta for JSON metadata. Use meta_blob for binary metadata.
4. Remove the inLedger field entirely.
5. Add the date field to all transactions that are from a closed ledger. Most of the time this would be the close time of the parent ledger. Make this a UTC ISO8601 timestamp with whole-seconds resolution, for example 2018-07-22T16:37:55Z. Omit this field from transactions that are not in closed ledgers.
6. Add the ledger_hash field to any transactions when pulling them from a closed ledger. Omit this field from transactions that are not in closed ledgers.
7. Add the validated boolean field to all transactions, even when this information is redundant because of context (for example, listing transactions in a validated ledger).

[nbougalis]

I am very supportive of this.

[mDuo13]

This _would_ break backwards compatibility. Perhaps we should put the version method to use.

(More specifically, we should start allowing users to request a specific API version so we can add such breaking changes in a non-disruptive way.)

[mDuo13]

To be crystal clear: the proposed format I am suggesting for all transactions is as follows:

| Field | Value | Description |
|:---------------|:--------------------------|:--------------------------------|
| hash | String - Hash | An identifying hash value unique to this transaction, as a hex string. |
| date | String | _(Omitted if from a non-closed ledger)_ The close time of the ledger that includes this transaction, as a UTC ISO8601 timestamp with whole-seconds resolution, for example 2018-07-22T16:37:55Z. |
| ledger_index | Number - Ledger Index | _(Omitted for the proposed transactions stream)_ The sequence number of the ledger that includes this transaction. |
| ledger_hash | String - Hash | _(Omitted if from a non-closed ledger)_ The unique hash of the ledger that includes this transaction. |
| tx | Object | _(Omitted if binary requested)_ The fields of this transaction object aka the transaction instructions, as defined by the Transaction Format. |
| tx_blob | String - Hex | _(Omitted unless binary requested)_ The binary format of the transaction instructions. |
| meta | Object | _(Omitted if binary requested and omitted from the proposed transactions stream)_ Metadata about the results of this transaction. |
| meta_blob | String - Hex | _(Omitted unless binary requested; omitted for the proposed transactions stream)_ The binary format of the transaction metadata. |
| validated | Boolean | Whether this transaction is in a validated ledger. |

[carlhua]

Consider this for API v2