Cosmos-sdk: Output logs as JSON format

[liangping]

Summary

I am working on Cosmos logs monitor with ELK. It's almost work perfect except some challenges in founding a pattern to extract useful information from plain text logs, particularly multi-line logs which contain many "\n". such as "block log"

Problem Definition

It's very difficult to parse a log which is consists of multi plain text lines, particularly in following scenarios:
1, Using tcp/udp input, one log will split in to several logs.

1. Multi cosmos-nodes, multi-line logs will mixed up.

What benefits does the SDK stand to gain by including this feature?
1, Easy parsing
2, Better performance.
3, Reducing Disc spaces for log server

Proposal

We need a better log format (JSON) than plain-text for log parsing.

---

For Admin Use

• [ ] Not duplicate issue
• [ ] Appropriate labels applied
• [ ] Appropriate contributors tagged
• [ ] Contributor assigned/self-assigned

[strotter]

For what it's worth, the Uber zap logger does this automatically and is lightning fast/low allocs, and I've had great results with it:

https://github.com/uber-go/zap

[alexanderbez]

I love the zap library and have used it extensively in the past...I think this would be too much of an overhaul right now though. Ideally the log formatting (text vs JSON) should be configurable based on some parameter/config.

[liangping]

thanks strotter.

Ideally the log formatting (text vs JSON) should be configurable based on some parameter/config.

I agree with you. and don't put "\n" in togs when output as JSON format.

[ValarDragon]

I'm concerned about the performance hit this may have. We want logs, and we want them to be cheap. I think parsing non-json will likely be faster despite being less convenient. I worry about slowing down the state machine for json encoding. (IIRC its kinda slow) This concern could be alleviated if we had some concrete benchmarks for how many log messages we expect to output per unit time, and how long JSON encoding would take.

[alexanderbez]

@ValarDragon structured JSON logging is pretty standard for production systems -- tools like zap should alleviate your concerns.

[ValarDragon]

Thanks for pointing that out, it does seem sufficiently fast!

[liangping]

parsing non-json will likely be faster despite being less convenient.

Yes, it's true if logs are simple text,
Cosmos logs are complex, and it's difficult to parse and extract useful information.
I think that each log should be a single JSON. such as: {k1:v1, k2:v2}. without "\n".
here are some references:
https://www.baeldung.com/java-log-json-output
https://logging.apache.org/log4j/2.x/manual/layouts.html#JSONLayout

[alexanderbez]

I believe this is already resolved now.