Caddy: [FEATURE][v2]: Provide a new "allow_list" to the log filter

[danlsgiga]

I'm in a current dilemma regarding my logs in Caddy v2.1.1...

I'm planning to have structured logs using the json encoder. I tried it but the chances of having sensitive info leaking in the logs is pretty high because a lot of our apps use multiple headers like Authorization: X-Auth-Token:, etc and due to compliance concerns, I can't (and don't want to) be managing what headers I delete via the delete log filter.

That being said, it would be nice to have some kind of log filter in place to allow only a predefined list of headers to be logged.

Example: The only header I care of at the moment is the User-Agent. So, having something like the log config below would be super nice to filter anything else but the User-Agent in the request.headers json block

{"logging": {
  "logs": {
    "log0": {
      "encoder": {
        "format": "filter",
        "wrap": "json",
        "fields": {          
          "request>headers": {
            "filter": "delete"
          },
          "request>headers>User-Agent": {
            "filter": "<the_name_of_the_new_filter>"
          }
        }
      }
    }
  }
}

[francislavoie]

See the ip_mask filter, you can specify options on a filter. I think what we want is an "except" option on the delete filter. That way we don't need to worry about ordering of the filters.

[danlsgiga]

Would that work if I delete the request>headers and have except = request>headers>Referer for example?

If yes, definitely simpler and way better ;)

[danlsgiga]

Also, except should be an array

[francislavoie]

Yeah that's what I'm thinking. An "except" option that takes an array of fields to keep after deleting, and those fields must be subfields of the one being deleted, I guess.

[francislavoie]

Sorry @danlsgiga 😢 I've spent a few hours today trying to figure out an alternative solution for this, but I can't come up with anything useful.

I couldn't do it at the log filter level, because filtering works on zap.Field objects which is a struct that wraps arbitrary types.

Another theory would be to do it when the HTTP header list is being marshaled here https://github.com/caddyserver/caddy/blob/309c1fec622c08bdc68a43de64fc0777f590d309/modules/caddyhttp/marshalers.go#L50 but I can't see a way to make that configurable. That function is pretty low-level in the pipeline, so I don't see a way forwards there, and it would only work for headers, not generally.

[danlsgiga]

Thanks anyways for trying @francislavoie... I guess I'll stick with the unstructure logging for now then... On that note, is there a way I can get the combined log format instead of the common?

[francislavoie]

Yeah @danlsgiga you can use the https://github.com/caddyserver/format-encoder plugin to write your own templated format. It's much less efficient though, because it can cause allocations to occur, which is why it's only provided as a plugin.