Ingress-nginx: ModSecurity in 0.19.0 no longer logs in fully valid JSON

[DaveAurionix]

Is this a request for help? No.

What keywords did you search in NGINX Ingress controller issues before filing this one? ModSecurity, JSON, SecAuditLogFormat

Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT

NGINX Ingress controller version: 0.19.0 (still testing in 0.20.0)

Kubernetes version (use kubectl version): 1.11.2

Environment:

• Cloud provider or hardware configuration: Azure Kubernetes Service
• OS (e.g. from /etc/os-release): Ubuntu 16.04
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What happened: When ModSecurity intercepts attacks a 403 is correctly returned and the attack is recorded in ModSecurity's audit log. In 0.13.0 this was recorded in JSON format. In 0.19.0 additional double-quote characters appear in two scenarios that break the JSON format and prevent audit logs being parsed by tooling.

What you expected to happen: safe JSON encoding is used when recording log entries so that values inside JSON properties have quote characters correctly escaped.

How to reproduce it (as minimally and precisely as possible): On a Kubernetes cluster with ModSecurity enabled and logging to stderr (see below), upgrade to the 0.19.0 image and trip a ModSecurity rule. Parse the resulting output with a JSON parser to find the errors. ModSecurity configuration is added using the following annotation:

nginx.ingress.kubernetes.io/configuration-snippet: |
      modsecurity_rules '
        SecRuleEngine On
        SecDefaultAction "phase:2,log,auditlog,deny,status:403"
        SecDefaultAction "phase:1,log,auditlog,deny,status:403"
        SecAuditLog /dev/fd/2
        SecAuditLogFormat JSON
        SecAuditLogParts ABIJHZ
        SecAuditLogType Serial
        SecAuditEngine RelevantOnly

Anything else we need to know: Docker is parsing the stderr output which is being picked up by Microsoft's OMS agent and reported to Log Analytics. Using the 0.13.0 Ingress image this was fine, using 0.19.0 we are seeing invalid JSON output.

Two places where the JSON is invalid:
1) An additional quote is added after the OWASP version number (search for OWASP_CRS/3.0.2 and note that it appears in the context of "components":["OWASP_CRS/3.0.2""] - note the extra closing quote.

2) When logging the response body (i.e. when OWASP rules trip due to response body content such as suspect php source leakage), double-quotes in the response body are not escaped at all, invalidating the JSON schema significantly. Additionally, quotes in ModSecurity error messages don't match up, e.g. "match":"Matched "Operator $$Rx' with parameter (etc) - in the actual logs a backtick is rendered instead of $$, but GitHub acts on the backtick. Note here that there is an erroneous double-quote before the word Operator but also the backtick before Rx doesn't match the single-quote after Rx. The second point is not causing invalid JSON but is an indicator that ModSecurity JSON serialisation is badly confused over the various quotes. It is possible that a JSON serialiser in 0.19.0 is trying to be too clever about detecting quotes-within-quotes and is replacing them with backticks/double-quotes as appropriate but is then getting confused by long chunks of text containing double-quotes, single-quotes, apostrophes and backticks. JSON format compliance simply requires an escaping backslash character before the quote rather than it being changed to a different type of quote.

[aledbf]

@DaveAurionix please use the image quay.io/kubernetes-ingress-controller/nginx-ingress-controller:dev

It is possible that a JSON serialiser in 0.19.0 is trying to be too clever about detecting quotes-within-quotes and is replacing them with

If the issue persists after the update to the dev image the problem is in modsecurity itself, not the ingress controller.

[DaveAurionix]

@aledbf Thanks for responding. The issue is definitely still present using that dev image.

Would you be able to let me know the versions of ModSecurity in use in the 0.13.0 and 0.19.0 builds of ingress-nginx? I can then raise an issue in the appropriate GitHub repo (and … forgive my ignorance but is this the source of the ModSecurity module used in ingress-nginx: https://github.com/SpiderLabs/ModSecurity)?

[aledbf]

0.13.0 -> 8d0f51beda5c031e38741c27f29b67f0266352bb
dev -> 9ada0a28c8100f905014c128b0e6d11dd75ec7e5

diff https://github.com/SpiderLabs/ModSecurity/compare/8d0f51beda5c031e38741c27f29b67f0266352bb...9ada0a28c8100f905014c128b0e6d11dd75ec7e5

https://github.com/SpiderLabs/ModSecurity

Correct

[aledbf]

@DaveAurionix please open an issue in the modsecurity repo an link my previous comment

[DaveAurionix]

Will do, thanks - I'll close this issue?

[aledbf]

No, it's not fixed (yet) :wink:

[DaveAurionix]

Great :) I want to do some more tests to get a more succinct repro so might take me a day or two but will raise it as soon as I have that.

[DaveAurionix]

@aledbf There are actually two issues that I thought were related but aren't.

1. (minor issue): ModSecurity is mangling quote marks in complex strings to create unmatching pairs of ' and backtick and ". This isn't actually causing any problem other than a presentational one.

2. (major issue): The logs written to stderr (and stdout?) by a Docker container running NGINX-Ingress 0.19.0 are being corrupted somewhere along the line as they are fed into Microsoft Log Analytics - \" sequences are becoming simply " and this is the source of unparsable JSON logs. Combined with point 1 the resulting delimiters are a mess.

The confusing thing is that this seemed to occur as we upgraded to Ingress-NGINX 0.19.0 but I now feel as if that must be a co-incidence.

I'm going to downgrade to Ingress-NGINX 0.13.0 to see if that rules out Ingress-NGINX, in which case I proposing ignoring point #1 above, closing this issue and I'll raise a separate issue (perhaps with Microsoft) about point 2.

I'll be in touch once I've tested the downgrade to Ingress-NGINX 0.13.0 in the coming days.

[aledbf]

@DaveAurionix friendly ping

[DaveAurionix]

Having to focus on issue 2 in the list above due to time constraints. That point isn't related to NGINX or ModSecurity so I'd like to close this issue. Thanks @aledbf for your help.