Rippled: Format timestamps using RFC3339

[nbougalis]

During the discussion on #2838, I commented that:

As part of this PR we should ensure that we explicitly print the timezone (or offset) when we format times in a human readable format so that it's completely unambiguous that the time is being displayed in UTC.

For example, consider the output from ledger:

"close_time_human": "2019-Feb-17 22:09:01.000000000"

Or the output from server_info:

"time": "2019-Feb-17 22:03:57.799117"

It's unclear (at least to me) that the times are in UTC, and to verify I need (at a minimum) a quick mental calculation involving timezones and offsets. Should we be specifying UTC explicitly (or whatever is appropriate and accepted? GMT? +00:00?) to aid understanding?

Per RFC 3339, the above could be written as:

2019-02-17T22:03:57.799117Z

Now, that's a pain to read for a human. The RFC allows substituting a space for the T to aid human readability:

2019-02-17 22:03:57.799117Z

The Z is still esoteric; I don't know how many people know about "Zulu time" and this kind of changes has another downside: it is a breaking change for anyone attempting to parse this field programatically.

@HowardHinnant responded with:

How about just appending " UTC" everywhere? Human readers will understand it, and parsers can "quit early" (not parse the UTC) and still get the right semantics.

@MarkusTeufelberger countered with:

Using https://en.wikipedia.org/wiki/ISO_8601#Coordinated_Universal_Time_(UTC) is not very "esoteric" in my opinion, inventing another non-standard way to display UTC however would be.

I believe we should adopt the the RFC3339 notation. Unfortunately, it's not as cut-and-dried. The old-style timestamps are part of our published API, and we can't just up and change. At a minimum, we will need to coordinate this change with:

1. @intelliot, to make sure that ripple-lib is updated to support both formats;
2. @mDuo13 to ensure that the documentation is updated; and
3. the world, to ensure that we give people adequate warning.

Let's figure out a viable path forward to get this done.

[intelliot]

With regard to ripple-lib:

1. ripple-lib does not parse human readable timestamps.
2. It does parse the timestamps that are in "seconds since the Ripple Epoch", like close_time, parent_close_time, and ledger_time. The code for that is small:

/**
 * @param {Number} rpepoch (seconds since 1/1/2000 GMT)
 * @return {Number} ms since unix epoch
 */
function rippleToUnixTimestamp(rpepoch: number): number {
  return (rpepoch + 0x386D4380) * 1000
}

function rippleTimeToISO8601(rippleTime: number): string {
  return new Date(rippleToUnixTimestamp(rippleTime)).toISOString()
}

1. There are some cases where the human readable timestamps are directly passed through, so it could still be a breaking change. We would need an announcement and documentation for that, but no code changes in ripple-lib.

Personally, I haven't heard of any problems with the current human readable timestamp format that rippled uses. The timestamps are always in UTC, correct? I think it would be enough to make sure that's clearly stated in the docs.

One thing worth noting is that ripple-lib currently formats timestamps (sourced from close_time, parent_close_time, ledger_time) using JavaScript's toISOString() which returns strings in simplified extended ISO format (ISO 8601). For example:

"closeTime": "2018-12-07T11:10:30.000Z",
"parentCloseTime": "2018-12-07T11:10:22.000Z"

[intelliot]

Here's an idea. We could add a field with a name that makes it clear that it's UTC:

"close_time_human": "2019-Feb-17 22:09:01.000000000",
"close_time_human_utc": "2019-02-17T22:09:01.000000Z"

"time": "2019-Feb-17 22:03:57.799117",
"time_utc": "2019-02-17T22:03:57.799117Z"

The downside is that it kind of implies that the old field is not UTC even though it is, but the good thing is that this makes it a non-breaking change.

[HowardHinnant]

If we don't have to add duplicate fields, I'd rather not. It could cause confusion going forward. However if our clients can't manage the changing API (say by accepting both formats for a while), Elliot's idea is certainly a path forward.

[mDuo13]

We really need a process for gracefully introducing breaking changes to the API.

[intelliot]

This particular change is small enough that I would be fine with simply making the breaking change. ripple-lib will be fine with it.

I would advocate for API versioning via an optional api_version parameter sent with each request. Example:

{
    "id": 14,
    "command": "ledger",
    "api_version": 2,
...

The response would have close_time_human formatted like "2019-02-17T22:03:57.799117Z". When api_version is omitted, the response would be unchanged from what it currently is.

[mDuo13]

Yeah, from what I recall, the team actually did some of the prep work for a similar scheme, just never finished it.

[nbougalis]

@HowardHinnant, I know you're working on other things but when you get a chance, we should move forward with this.

[mDuo13]

Didn't we already make changes to add " UTC" to a bunch of the timestamps in the time since this ticket was created? I noticed the difference in server_info and in the server logs but I can't remember what release changed it.

[HowardHinnant]

This would've added "UTC" in many places: https://github.com/ripple/rippled/commit/ce589225dd5d9fa661007c794fac12c7d2b35b77

I need to survey and see if there are any more places that need attention, but it is likely we can just close this now.