Goaccess: how can i parse this JSON logs

[AMHZR]

My applicaiton logs each request as follows and i want to analyze such using Goaccess..

{"statusCode":"200","method":"POST","url":"/my_url","responseTime":"13 ms","ip":"::ffff:10.0.1.42","userAgent":"Other 0.0.0 / Other 0.0.0","data":"{"location":"[{"lat":25.2239529,"lng":55.3407724,"tm_stmp":1569196799732,"acc":3,"alt":-9,"spd":25,"mock":0,"gps":1,"net":1,"bat_lvl":24,"d_acc":1,"bearing":229.61138916015625}]","access_token":"*********"}","response":"{"message":"Successful","status":200,"data":{}}","level":"info","message":"","timestamp":"2019-09-23T00:00:00.301Z"}

[rahra]

You could pre-process it with a script (e.g. perl) to reformat this into some common log file format.

[allinurl]

@AMHZR This should do it:

goaccess access.log --log-format='%^:"%s",%^:"%m",%^:"%U",%^:"%L %^",%^:"%h",%^:"%u",%^]"%^,%^,%^,%^,%^,%^,%^,%^:"%dT%t.%^"' --date-format='%Y-%m-%d' --time-format=%T --http-protocol=no

[AMHZR]

@AMHZR This should do it:

goaccess access.log --log-format='%^:"%s",%^:"%m",%^:"%U",%^:"%L %^",%^:"%h",%^:"%u",%^]"%^,%^,%^,%^,%^,%^,%^,%^:"%dT%t.%^"' --date-format='%Y-%m-%d' --time-format=%T --http-protocol=no

this works like charm....

[jaywilliams]

What an amazing workaround, allinurl. However, I think the bigger issue is that the application is using JSON for logging.

[pauloxnet]

Why do you think using JSON for logging is an issue?
We're storing log as JSON lines and it's very helpful to read them directly in Python loading each line as a dictionary.

[allinurl]

Glad that worked. Closing this.

[jaywilliams]

@pauloxnet, it depends on what you're using the logs for, but a well written plain-text log file, paired with the the UNIX coreutils is an amazing combo. In 30 years, I doubt people will be using JSON, as there will be some new format on the block, but plain-text will always exist.

[pauloxnet]

But in this case JSON Lines format is only plai- text with coma separator and key-value format, it's human readable and it's easy to to import line by line and analyze.