Caddy: Generate json log

[runningman84]

It would be gread if caddy could generate a full json like this example from kibana:

{
 "type": "response",
 "@timestamp": "2016-09-15T20:26:54Z",
 "tags": [],
 "pid": 12,
 "method": "post",
 "statusCode": 200,
 "req": {
  "url": "\/elasticsearch\/_msearch?timeout=0&ignore_unavailable=true&preference=1473971177442",
  "method": "post",
  "headers": {
   "host": "kibana.example.loc",
   "user-agent": "Mozilla\/5.0 (X11; Linux x86_64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/52.0.2743.82 Safari\/537.36",
   "content-length": "789",
   "accept": "application\/json, text\/plain, *\/*",
   "accept-language": "de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4",
   "content-type": "application\/json;charset=UTF-8",
   "kbn-version": "4.6.1",
   "origin": "http:\/\/kibana.example.loc",
   "referer": "http:\/\/kibana.example.loc\/app\/kibana",
   "x-forwarded-for": "172.18.0.1",
   "x-forwarded-proto": "http",
   "x-real-ip": "172.18.0.1",
   "accept-encoding": "gzip"
  },
  "remoteAddress": "172.18.0.14",
  "userAgent": "172.18.0.14",
  "referer": "http:\/\/kibana.example.loc\/app\/kibana"
 },
 "res": {
  "statusCode": 200,
  "responseTime": 249,
  "contentLength": 9
 },
 "message": "POST \/elasticsearch\/_msearch?timeout=0&ignore_unavailable=true&preference=1473971177442 200 249ms - 9.0B"
}

That would really simplify our logstash rules...

[mholt]

You can specify a log format where each line is JSON, does that help?

(Also, this is basically a duplicate of #425.)

[runningman84]

@mholt can you give me an example? The idea is to have all fields which are present included in a json...

[mholt]

You can use the {request} placeholder, which compacts the headers to a single line.

If you need the exact JSON output shown above, I'm not sure there is a way right now.

[runningman84]

Ok the {request} placeholder cannot be read in logstash without quite complex and error prone parsing. Having everything in a json object would help here. I think this project might work well:
https://github.com/harshavardhana/caddylogger/blob/master/caddylogger.go
Maybe such feature should be part of the core...

[mholt]

Hmm... if there's a generic way to support this capability (i.e. not tying it to any specific JSON format or provider) then we could probably work it into the log directive somehow.

[sanmai-NL]

@mholt: see also this blog for format inspiration.

[sandstrom]

Log output where each request is a JSON payload would be great.

If you do add it, please use this format: http://jsonlines.org/

(basically just JSON, one payload / log-entry per line)

[mholt]

@runningman84 I just got this working in Caddy 2 (sort of):

{
    "level": "info",
    "ts": 1571685845.192638,
    "logger": "foobar",
    "msg": "got request",
    "common_log": "::1 - - [21/Oct/2019:13:24:05 -0600] \"GET / HTTP/1.1\" 200 85",
    "request": {
        "method": "GET",
        "uri": "/test/abc?foo=bar",
        "url": "/test/abc?foo=bar",
        "proto": "HTTP/1.1",
        "remote_addr": "[::1]:53539",
        "host": "localhost:2080",
        "headers": {
            "User-Agent": [
                "curl/7.54.0"
            ],
            "Accept": [
                "*/*"
            ]
        }
    },
    "latency": 0.000885658,
    "size": 85,
    "status": 200
}

Still a WIP. But it'll be available in Caddy 2 soon.

[mholt]

Okay, draft PR is up: https://github.com/caddyserver/caddy/pull/2831

It's happening!

[mholt]

This has been merged and is going out with Caddy 2 beta 7 today. Check out the docs in the wiki and give it a try!