Ingress-nginx: ModSecurity does not block request, only logs, while SecRuleEngine is set to On

[kwaazaar]

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG

NGINX Ingress controller version: 0.25.0 (installed through helm chart nginx-ingress-1.10.1)

Kubernetes version (use kubectl version):
v1.12.7

Environment:

• Cloud provider or hardware configuration: Azure Kubernetes Service (AKS)
• OS (e.g. from /etc/os-release): Ubuntu 16.04.6 LTS
• Kernel (e.g. uname -a): 4.15.0-1052-azure
• Install tools: -
• Others: -

What happened:
Request is not blocked by ModSecurity, only logged. Debug log says:
[4] Not running disruptive action: pass. SecRuleEngine is not On.

What you expected to happen:
Blocked request by returning HTTP 403, because SecRuleEngine is specified as on.

How to reproduce it (as minimally and precisely as possible):

• Install nginx-ingress with this content in values.yaml (partial):
  controller:
  config:
  enable-modsecurity: "true"
  enable-owasp-modsecurity-crs: "true"
• Apply ingress with this content (partial):
  nginx.ingress.kubernetes.io/modsecurity-snippet: |
  SecRule REQUEST_HEADERS_NAMES "^x-waf-test" "log,deny,id:48,status:403,t:lowercase,msg:WAFWAFWAF"
  SecRequestBodyAccess On
  SecDebugLog /tmp/modsec_debug.log
  SecDebugLogLevel 9
  SecRuleEngine On
• Send request to ingress-host which should trigger ModSecurity rule(s):
  HTTP GET to /?test=%271%20OR%201=1&x=%3Cscript%3Ealert%28%27hello%27%29;%3C/script%3E
• Check mod-security debug log and see:
  [156458238464.652956] [/?test=%271%20OR%201=1&x=%3Cscript%3Ealert%28%27hello%27%29;%3C/script%3E] [4] Not running disruptive action: pass. SecRuleEngine is not On.

Anything else we need to know:
I check logs with: kubectl -n kube-system exec -it internal-ingresscontroller-nginx-ingress-controller-7f8c6774w84 -- tail -f /tmp/modsec_debug.log

[kwaazaar]

Nginx accepts the config and it not rejecting it and remaining on previous version of nginx.conf: to be sure I check the logs for the ingress controller pods and added an extra response header through a configuration snippet.

Nginx.conf: https://www.dropbox.com/s/2rj9r53ftbowl0o/nginx.conf?dl=0
Ingress definition: https://github.com/kwaazaar/nginx-ingress-adv (ingress.yml)

[Aanuka]

Can also verify that this is a issue.
Running the same setup with nginx 0.25.0 (Openresty 1.15.8.1), enabled modsecurity and geoip2 through values.yaml and enabling modsecurity through configuration-snippet.

Logs are being created but no block action being done.
Geoip2 filters on the same configuration snippet works.

[mhauer71]

I also have the same issue with nginx 0.25.0.
Tried the example in this article:

nginx.ingress.kubernetes.io/modsecurity-snippet: |
      SecRuleEngine On
      SecRequestBodyAccess On
      SecAuditEngine RelevantOnly
      SecAuditLogParts ABIJDEFHZ
      SecAuditLog /var/log/modsec_audit.log
      SecRule REQUEST_HEADERS:User-Agent \"fern-scanner\" \"log,deny,id:107,status:403,msg:\'Fern Scanner Identified\'\"

With curl https://mysite/ -k -H "user-agent: fern-scanner" it only logged to /var/log/modsec_audit.log
Editing the ngnix deployment to version 0.24.1 then I received _403 Forbidden_ as expected.

[Aanuka]

The issue is still seen with 0.25.1
Using the following configuration:

nginx.ingress.kubernetes.io/configuration-snippet: |
  modsecurity_rules '
    SecRuleEngine On
    SecAuditLog /var/log/modsec_audit.log
    SecAuditLogParts ABIJDEFHZ
    SecAuditEngine RelevantOnly
    SecAction \
      "id:900000,\
      phase:1,\
      nolog,\
      pass,\
      t:none,\
      setvar:tx.paranoia_level=1"

Warnings are being logged in modsec_audit.log but no blocking actions are being made.

[kwaazaar]

Please clarify if ModSecurity integration will get attention. If not, discontinue the feature, because currently it feels like we're waisting our time with it.

[aledbf]

Please clarify if ModSecurity integration will get attention

What do you mean?

We updated the configuration in the last update of the nginx image here https://github.com/kubernetes/ingress-nginx/commit/efc66451f4a75ed8bb4390e9d10d551fea317e33#diff-4a5113d28634e29f2ab0bca34da7a3e5R333 (not released yet).

Please use quay.io/kubernetes-ingress-controller/nginx-ingress-controller:dev to test this issue

[Aanuka]

Please clarify if ModSecurity integration will get attention

What do you mean?

We updated the configuration in the last update of the nginx image here efc6645#diff-4a5113d28634e29f2ab0bca34da7a3e5R333 (not released yet).

Please use quay.io/kubernetes-ingress-controller/nginx-ingress-controller:dev to test this issue

Just tried with :dev , this time it didn't even log anything to the /var/log/modsec_audit.log file when trying an command and just passes an 404

Snips from the nginx.conf

modsecurity on;
modsecurity_rules_file /etc/nginx/modsecurity/modsecurity.conf;
modsecurity_rules_file /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf;

modsecurity_rules '
SecRuleEngine On
SecAuditLog /var/log/modsec_audit.log
SecAuditLogParts ABIJDEFHZ
SecAuditEngine RelevantOnly
SecAction \
"id:900000,\
phase:1,\
nolog,\
pass,\
t:none,\
setvar:tx.paranoia_level=1"
';

nginx version: openresty/1.15.8.2

[aledbf]

To those affected by this issue, please check the gist test-modsecurity.sh

Use it:

1. wget https://gist.githubusercontent.com/aledbf/f6ccf154d4db8a3046b1067db80bd889/raw/6939cbc61b4035e8ceddddf7e8351fbe5b03bd6a/test-modsecurity.sh
2. chmod +x test-modsecurity.sh
3. ./test-modsecurity.sh

This script:

• starts a new kubernetes cluster using minikube.
• installs the ingress controller.
• updates the image to use the dev tag.
• creates an ingress that contains the modsecurity-snippet annotation
• makes a request with curl that returns 403 (expected behavior)
• dumps the ingress controller logs

[Aanuka]

To those affected by this issue, please check the gist test-modsecurity.sh

Use it:

1. wget https://gist.githubusercontent.com/aledbf/f6ccf154d4db8a3046b1067db80bd889/raw/6939cbc61b4035e8ceddddf7e8351fbe5b03bd6a/test-modsecurity.sh
2. chmod +x test-modsecurity.sh
3. ./test-modsecurity.sh

This script:

• starts a new kubernetes cluster using minikube.
• installs the ingress controller.
• updates the image to use the dev tag.
• creates an ingress that contains the modsecurity-snippet annotation
• makes a request with curl that returns 403 (expected behavior)
• dumps the ingress controller logs

Checked the code and reapplied the ingress on our test env, with the same issues i.e no blocking is being made. In fact, with the :dev release no detection is seen in the modsec_audit.log

[celamb4]

I ran into this issue. Even with SecRuleEngine ON I only get logs and no blocks (403) when testing. I see the logs correctly catching the attacks but not blocking them.

kind: ConfigMap
metadata:
  name: nginx-ingress-controller
apiVersion: v1
data:  
  enable-modsecurity: "true"
  modsecurity-transaction-id: "$request_id"
  modsecurity-snippet: |
    SecRuleEngine On
    SecRequestBodyAccess On
    SecAuditEngine RelevantOnly
    SecAuditLogParts ABIJDEFHZ
    SecAuditLog /var/log/modsec_audit.log
    Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf
    Include /etc/nginx/modsecurity/modsecurity.conf

Upon further review, I noticed the /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf still had detection mode only. I created my own file for both this and also crs-setup.conf files. To ensure (a) Engine is ON and (b) secAction is set to block. I then mounted them using extra volumes.

SecRuleEngine On #nginx-modsecurity.conf file
SecDefaultAction "phase:1,log,auditlog,deny,status:403" #crs-setup.conf file
SecDefaultAction "phase:2,log,auditlog,deny,status:403" #crs-setup.conf file

Deploy nginx with the following additional configurations:

    "--set controller.extraVolumes[0].configMap.name=modsecurity-config "
    "--set controller.extraVolumes[0].name=modsecurity-config "
    "--set controller.extraVolumeMounts[0].name=modsecurity-config "
    "--set controller.extraVolumeMounts[0].mountPath=/etc/nginx/modsecurity/modsecurity.conf "
    "--set controller.extraVolumeMounts[0].subPath=modsecurity.conf "
    "--set controller.extraVolumes[1].configMap.name=crs-setup-config "
    "--set controller.extraVolumes[1].name=crs-setup-config "
    "--set controller.extraVolumeMounts[1].name=crs-setup-config "
    "--set controller.extraVolumeMounts[1].mountPath=/etc/nginx/owasp-modsecurity-crs/crs-setup.conf "
    "--set controller.extraVolumeMounts[1].subPath=crs-setup.conf "

Modsecurity is now successfully blocking the requests

$ curl "https://example.com/login?q='1 OR 1=1"
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>openresty</center>
</body>
</html>

[aledbf]

Closing. Please update to 0.26.0. The release contains fixes for modsecurity. Please reopen if the issue persists.

[Aanuka]

This issue still seems to persist in 0.26.0 with even more strange behaviour.

I've tried with:

• Annotations in the configmap for the controller
• Annotations in the ingress for a specific URL

Without any success.
With 0.26.0 modsecurity dosen't even log any errors into /var/log/modsec_audit.log and just passes through the request like normal (I.e 404 for something that dosen't exist).

What's even more strange is that controller now seems to log everything to /var/log/modsec_audit.log, we can for an example see:
127.0.0.1:10256 127.0.0.1 - [28/Sep/2019:16:27:16 +0000] "GET /nginx_status HTTP/1.1" 200 106 - "Go-http-client/1.1" 15696880369.409824 - /var/log/audit//20190928/20190928-1627/20190928-162716-15696880369.409824 0 1315.000000 md5:2127b3a365aeddf5ae366618a2356bf3

[reddare]

Same with @Aanuka on 0.26.1 release.
@kwaazaar @aledbf please reopen.

[Aanuka]

This still seems to be an issue in 0.26.1
@kwaazaar @aledbf Can anyone of you reopen so we can try and solve this?

[Keralin]

I was getting crazy because of this bug, was in 0.25.1 with this problem, then updated to 0.26.1 and still the same behavior, looked at @mhauer71 post, downgraded to 0.24.1 and finally after 1 day and a half messing with this, now works as expected! Please reopen this issue, it's annoying until you realize that could be a bug @kwaazaar @aledbf

[kwaazaar]

I cannot reopen the issue, since a repo collaborator closed it.

It's sad that ModSecurity support seems to be ignored by the team.

[zetaab]

I can confirm that it does not work for 0.28.0 either like @celamb4 said the files are not correct. Files are only saying "pass" if it will detect something.

I have following in ingress configuration:

      nginx.ingress.kubernetes.io/configuration-snippet: |
        more_set_headers "server: hide";
      nginx.ingress.kubernetes.io/enable-modsecurity: "true"
      nginx.ingress.kubernetes.io/modsecurity-snippet: |
        SecRuleEngine On
        SecRequestBodyAccess On
        SecAuditEngine RelevantOnly
        SecAuditLogParts ABIJDEFHZ
        SecAuditLog /var/log/modsec_audit.log
        Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf
      nginx.ingress.kubernetes.io/modsecurity-transaction-id: $request_id

and when I do request for instance curl -s -o /dev/null "https://foobar.com/?username='%20or%20'1'%20=%20'" it will end up to /var/log/modsec_audit.log. But the request itself is passed as 200 HTTP OK.

[zetaab]

When I check generated nginx.conf (version 0.28.0) it looks following:

                        modsecurity on;

                        modsecurity_rules_file /etc/nginx/modsecurity/modsecurity.conf;

                        modsecurity_rules '
                        SecRuleEngine On
                        SecRequestBodyAccess On
                        SecAuditEngine RelevantOnly
                        SecAuditLogParts ABIJDEFHZ
                        SecAuditLog /var/log/modsec_audit.log
                        Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf

                        ';

                        modsecurity_transaction_id "$request_id";

And when I check /etc/nginx/modsecurity/modsecurity.conf it says SecRuleEngine DetectionOnly

When I compare this to 0.24.1 which is working (returning 403).

            modsecurity on;

            modsecurity_rules '
            SecRuleEngine On
            SecRequestBodyAccess On
            SecAuditEngine RelevantOnly
            SecAuditLogParts ABIJDEFHZ
            SecAuditLog /var/log/modsec_audit.log
            Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf

            ';

So the difference in newer versions is that modsecurity_rules_file /etc/nginx/modsecurity/modsecurity.conf; is loaded.

[zetaab]

@MRoci as you have contributed this change in https://github.com/kubernetes/ingress-nginx/pull/4080 can you please confirm that it is not breaking this whole thing

[MRoci]

@MRoci as you have contributed this change in #4080 can you please confirm that it is not breaking this whole thing

@zetaab reverting this could reintroduce https://github.com/kubernetes/ingress-nginx/issues/3585
Because owasp-csr alone doesn't do all the required cofiguration work to have a working setup.

Maybe we could enable modsecurity.conf when using default config and with owasp-csr and ignore it when using a custom config snippet

[pcallewaert]

I've got it working like this (with 0.27.1):

ConfigMap:

enable-modsecurity: "true"

Ingress Annotation:

    nginx.ingress.kubernetes.io/modsecurity-snippet: |
      SecRuleEngine On
      Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf

Now it blocks requests (403 Forbidden) like expected... Hope this helps

[Aanuka]

Also got it working in 0.27.1

ConfigMap:

enable-modsecurity: "true"
modsecurity-snippet: |
SecRuleEngine On
SecRequestBodyAccess On
SecAuditEngine RelevantOnly
SecRuleRemoveById 920350
Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf

I had to add the SecRuleRemoveById 920350 to the ConfigMap to not have the /var/log/modsec_audit.log filled with entries for internal request.

[zetaab]

I can confirm that @pcallewaert works. However, my use-case is that I would like to have following:

1) configmap setting that ALL do have this:

  modsecurity-snippet: |
    SecRequestBodyAccess On
    SecAuditEngine RelevantOnly
    SecAuditLogParts ABHKZ
    SecAuditLog http://modsec-audit-deployment.kube-system.svc.cluster.local:9998/audits
    SecAuditLogType HTTPS
    SecAuditLogFormat JSON
    SecRuleRemoveById 920350
    Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf

2) only selected ingresses could have SecRuleEngine On I do not want turn that on cluster wide, because it will break things.

I have not found way to do this. It is only possible to turn on in global way currently? tl;dr all ingresses are reporting secaudit things, but selected should have forced policy

edit: Cool I got it working:

configmap:

  enable-modsecurity: "true"
  modsecurity-snippet: |
    SecRequestBodyAccess On
    SecAuditEngine RelevantOnly
    SecAuditLogParts ABHKZ
    SecAuditLog http://modsec-audit-deployment.kube-system.svc.cluster.local:9998/audits
    SecAuditLogType HTTPS
    SecAuditLogFormat JSON
    SecRuleRemoveById 920350
    Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf

and then in ingress only this:

    nginx.ingress.kubernetes.io/modsecurity-snippet: |
      SecRuleEngine On

[JorgeGuerrero-dev]

I've got it working like this (with 0.27.1):

ConfigMap:

enable-modsecurity: "true"

Ingress Annotation:

    nginx.ingress.kubernetes.io/modsecurity-snippet: |
      SecRuleEngine On
      Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf

Now it blocks requests (403 Forbidden) like expected... Hope this helps

This work for me!
Thanks!

[Hello-Linux]

I'am not working like this (with 0.30): https://github.com/kubernetes/ingress-nginx/issues/5343

[JorgeGuerrero-dev]

I'am not working like this (with 0.30): #5343

Yes, I have same problem when I updated to 0.30 not working in 0.27.1 working fine, this is a problem because I can't update to latest version.
Thanks!

[fennellgb23]

Can confirm as well, not working in v 0.30, is anyone looking into this?

Update: the file /etc/nginx/modsecurity/modsecurity.conf has SecRuleEngine set to DetectionOnly. I edited the file to On and it now works. Is there an easy way to just override this setting since it does not seem us setting this in the Ingress is making any difference.

Update 2: I can confirm that this "DOES" work. After more investigations I found my team changed the configmap for my ingress so I was updating the wrong file. For anyone that lands here, if you 'kubectl edit configmap {name}' the correct configmap and save the changes you will see it working properly.

[junaid18183]

I can confirm Its working on "0.32.0", here is my config -

  config:
    enable-modsecurity: "true"
    enable-owasp-modsecurity-crs: "true"

Ingress Annotation:

  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/modsecurity-snippet: |
      SecRuleEngine On

[JorgeGuerrero-dev]

Yes, I also confirm Its working on "0.32.0", problem solved! Thanks!!!!
The @junaid18183 works for me!