Rippled: Write logs as JSON instead of text (structured logging)

[MarkusTeufelberger]

Currently rippled writes its logs as more or less structured lines of text (in some cases multiple lines). This means these logs might need to be parsed again to know that the fictitious User Foo with IP 1.2.3.4 signed a statement with an ed25519 key log message contains a user name, an IP address etc.

This is more brittle than maybe necessary and requires a lot of work to parse out information that already existed at the time this line was composed. It would be great if rippled could optionally (or exclusively) write/print its log files in a serialized format (JSON would be an obvious choice, since you use it extensively anyways). This would make it much nicer to ingest its logs in e.g. an ELK stack.

[dmitryrn]

Where can I see logs? I can't find any info about logs (file) in docs. Thanks.

[MarkusTeufelberger]

https://github.com/ripple/rippled/blob/e83d367f49cceb4b5f28830fb1965a5978744450/cfg/rippled-example.cfg#L918-L923

...or also just on STDOUT.

[sublimator]

If @rec had had his way

>

[rec]

Quite so.

As I pointed out years ago, the logs are basically freeform, contain vast amounts of information that will never be useful to even one person, and are wildly redundant.

I suggested JSON logs, or since rippled uses protocol buffers, those (because they're super-compressed, great for logs).

Vinnie's response: "We're not doing that. If you aren't smart enough to figure out the log files..."

---

Later, Ripple fired me within _minutes_ of my disclosing that I was having panic attacks. And why was I panicking? Because they wanted me to commit to releasing a product in six months that I was sure would take at least a year to get out.

It's been two-and-a-half years now and that product never came out.

[movitto]

It would be great if rippled could optionally (or exclusively) write/print its log files in a serialized format (JSON would be an obvious choice, since you use it extensively anyways). This would make it much nicer to ingest its logs in e.g. an ELK stack.

Can see the merits of both approaches, serialized logs would be great for digesting in a log analyzer / other system, but would be more difficult for humans to quickly read the raw logs. Yes admins could use external programs to inspect the deserialized logs, but it's also nice to support quickly extracting info via a simple "tail -f" on a live log file.

Configurable log formats would be an option, but this introduces the drawback of multiple-formats being supported meaning an external user would not know the format in which a log was written until it's inspected (though perhaps this could be distinguished via filename extension, eg: debug.log, debug.log.json, etc). Also care would need to be taken if the user switched the log format after existing log data has already been written.

Lastly, not a big fan of binary logs, ala the systemd approach. Yes they are more efficient but IMO maximilizing efficiency at at the expense of "human-understandability" the isn't a good idea in this use case (eg these logs)