Owasp-modsecurity-crs: tx.paranoia_level on v3/master ignored

[theMiddleBlue]

Hi,

I've an odd behavior with the last v3/master version (I've just pulled it). It seems that it ignores at all the setvar:tx.paranoia_level=1 and it matches many rules in PL2, 3 and 4.

My configuration:

SecAction \
  "id:900000,\
   phase:1,\
   nolog,\
   pass,\
   t:none,\
   setvar:tx.paranoia_level=1"

And on audit log I've many of these (with my previous version of v3/master it wasn't happen):

{
  "details": {
    "accuracy": "0",
    "rev": "",
    "ver": "",
    "reference": "v1839,1",
    "lineNumber": "18",
    "match": "Matched \"Operator `Eq' with parameter `0' against variable `MULTIPART_UNMATCHED_BOUNDARY' (Value: `2' )",
    "tags": [],
    "ruleId": "200004",
    "maturity": "0",
    "data": "",
    "severity": "0",
    "file": "/usr/local/openresty/nginx/conf/modsecurity_config/http-www.acselspa.it/http-www.acselspa.it"
  },
  "message": "Multipart parser detected a possible unmatched boundary."
},
{
  "details": {
    "accuracy": "9",
    "rev": "2",
    "ver": "OWASP_CRS/3.0.0",
    "reference": "o40,1o41,1o87,1o88,1o89,1o90,1o110,1o111,1o152,1o153,1o201,1o202,1o203,1o204,1o215,1o216,1o257,1o258,1o305,1o306,1o307,1o308,1o313,1o314,1o355,1o356,1o412,1o413,1o414,1o415,1o428,1o429,1o472,1o473,1v1839,474t:urlDecodeUni",
    "lineNumber": "1287",
    "match": "Matched \"Operator `ValidadeByteRange' with parameter `32-36,38-126' against variable `REQUEST_BODY' (Value: `------WebKitFormBoundaryWWc2IeyPuspjWwJ5\\x0d\\x0aContent-Disposition: form-data; name=\"action\"\\x0d\\x0 (476 characters omitted)' )",
    "tags": [
      "application-multi",
      "language-multi",
      "platform-multi",
      "attack-protocol",
      "OWASP_CRS/PROTOCOL_VIOLATION/EVASION",
      "paranoia-level/3"
    ],
    "ruleId": "920272",
    "maturity": "9",
    "data": "",
    "severity": "2",
    "file": "/usr/local/openresty/nginx/conf/owasp-modsecurity-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"
  },
  "message": "Invalid character in request (outside of printable chars below ascii 127)"
},
{
  "details": {
    "accuracy": "9",
    "rev": "2",
    "ver": "OWASP_CRS/3.0.0",
    "reference": "o40,1o41,1o62,1o72,1o73,1o79,1o86,1o87,1o88,1o89,1o90,1o110,1o111,1o152,1o153,1o174,1o184,1o185,1o191,1o200,1o201,1o202,1o203,1o204,1o215,1o216,1o257,1o258,1o279,1o289,1o290,1o296,1o304,1o305,1o306,1o307,1o308,1o313,1o314,1o355,1o356,1o377,1o387,1o388,1o394,1o411,1o412,1o413,1o414,1o415,1o428,1o429,1o472,1o473,1v1839,474t:urlDecodeUni",
    "lineNumber": "1348",
    "match": "Matched \"Operator `ValidadeByteRange' with parameter `38,44-46,48-58,61,65-90,95,97-122' against variable `REQUEST_BODY' (Value: `------WebKitFormBoundaryWWc2IeyPuspjWwJ5\\x0d\\x0aContent-Disposition: form-data; name=\"action\"\\x0d\\x0 (476 characters omitted)' )",
    "tags": [
      "application-multi",
      "language-multi",
      "platform-multi",
      "attack-protocol",
      "OWASP_CRS/PROTOCOL_VIOLATION/EVASION",
      "paranoia-level/4"
    ],
    "ruleId": "920273",
    "maturity": "9",
    "data": "",
    "severity": "2",
    "file": "/usr/local/openresty/nginx/conf/owasp-modsecurity-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"
  },
  "message": "Invalid character in request (outside of very strict set)"
},
{
  "details": {
    "accuracy": "8",
    "rev": "2",
    "ver": "OWASP_CRS/3.0.0",
    "reference": "o2,8o2,8v1931,19t:urlDecodeUni",
    "lineNumber": "1251",
    "match": "Matched \"Operator `Rx' with parameter `((?:[\\~\\!\\@\\#\\$\\%\\^\\&\\*\\(\\)\\-\\+\\=\\{\\}\\[\\]\\|\\:\\;\\\"\\'\\´\\’\\‘\\`\\<\\>][^\\~\\!\\@\\#\\$\\%\\^\\&\\*\\(\\)\\-\\+\\=\\{\\}\\[\\]\\|\\:\\;\\\"\\'\\´\\’\\‘\\`\\<\\>]*?){2})' against variable `ARGS:action' (Value: `wp-remove-post-lock' )",
    "tags": [
      "application-multi",
      "language-multi",
      "platform-multi",
      "attack-sqli",
      "OWASP_CRS/WEB_ATTACK/SQL_INJECTION",
      "WASCTC/WASC-19",
      "OWASP_TOP_10/A1",
      "OWASP_AppSensor/CIE1",
      "PCI/6.5.2",
      "paranoia-level/4"
    ],
    "ruleId": "942432",
    "maturity": "9",
    "data": "Matched Data: -remove- found within ARGS:action: wp-remove-post-lock",
    "severity": "4",
    "file": "/usr/local/openresty/nginx/conf/owasp-modsecurity-crs/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"
  },
  "message": "Restricted SQL Character Anomaly Detection (args): # of special characters exceeded (2)"
}

Any idea? how can I do for pull the previous version of v3/master?

Thanks!

[dune73]

Thank you for reporting @theMiddleBlue. And sorry for the inconvenience.

This sounds annoying indeed. Could you raise the ModSec DebugLogLevel and show us the detailed debug log for 920014, please? There we should see the actual paranoia level and the skipping to the end of the 920xxx rule file. Your alerts indicate this rule not working properly.

[theMiddleBlue]

Hi @dune73,

Following a grep by 920014

[4] (Rule: 920014) Executing operator "Lt" with param "2" against TX:PARANOIA_LEVEL.

and here with 5 lines before and after:

# tail -n 0 -f /var/log/modsec-debug.log | grep -A5 -B5 '920014'
[4] Running [independent] (non-disruptive) action: setvar
[4] Rule returned 1.
[4] Executing chained rule.
[4] (Rule: 0) Executing operator "Within" with param "/proxy/ /lock-token/ /content-range/ /translate/ /if/" Was: "" against TX:regex(^HEADER_NAME_).
[4] Rule returned 0.
[4] (Rule: 920014) Executing operator "Lt" with param "2" against TX:PARANOIA_LEVEL.
[4] Rule returned 0.
[4] (Rule: 920200) Executing operator "Rx" with param "^bytes=((\d+)?\-(\d+)?\s*,?\s*){6}" against REQUEST_HEADERS:Range|REQUEST_HEADERS:Request-Range.
[4] Rule returned 0.
[4] (Rule: 920201) Executing operator "EndsWith" with param ".pdf" against REQUEST_BASENAME.
[4] Rule returned 0.

[csanders-git]

A user on irc reported the same thing. He assumed it was this commit https://github.com/SpiderLabs/ModSecurity/commit/202a15bea8e6e009bb5c03737d565d685f7db9aa

[theMiddleBlue]

I'm using this request for reproduce it:

curl -d 'ContentObjectAttribute_ezstring_data_text_436=Name+Name&ContentObjectAttribute_data_text_437=user%40gmail.com&ContentObjectAttribute_ezstring_data_text_2572=3478710812&ContentObjectAttribute_data_text_438=Message&ActionCollectInformation=Invia&ContentNodeID=78&ContentObjectID=80&ViewMode=full' 'http://127.0.0.1/content/action'

Audit Log:

ModSecurity: Warning. Matched "Operator `Rx' with parameter `^[\d.:]+$' against variable `REQUEST_HEADERS:Host' (Value: `127.0.0.1' ) [file "/usr/local/openresty/nginx/conf/owasp-modsecurity-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "733"] [id "920350"] [rev "2"] [msg "Host header is a numeric IP address"] [data "127.0.0.1"] [severity "4"] [ver "OWASP_CRS/3.0.0"] [maturity "9"] [accuracy "9"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/IP_HOST"] [tag "WASCTC/WASC-21"] [tag "OWASP_TOP_10/A7"] [tag "PCI/6.5.10"] [hostname "172.17.0.1"] [uri "/content/action"] [unique_id "152906589237.129757"] [ref "o0,9v36,9"]
ModSecurity: Warning. Matched "Operator `ValidadeByteRange' with parameter `38,44-46,48-58,61,65-90,95,97-122' against variable `REQUEST_BODY' (Value: `ContentObjectAttribute_ezstring_data_text_436=Name+Name&ContentObjectAttribute_data_text_437=user%40 (193 characters omitted)' ) [file "/usr/local/openresty/nginx/conf/owasp-modsecurity-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "1348"] [id "920273"] [rev "2"] [msg "Invalid character in request (outside of very strict set)"] [data ""] [severity "2"] [ver "OWASP_CRS/3.0.0"] [maturity "9"] [accuracy "9"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/EVASION"] [tag "paranoia-level/4"] [hostname "172.17.0.1"] [uri "/content/action"] [unique_id "152906589237.129757"] [ref "o4,1v197,9t:urlDecodeUnio50,1v151,293t:urlDecodeUni"]

Debug Log:

[4] Running [independent] (non-disruptive) action: setvar
[4] Rule returned 1.
[4] Executing chained rule.
[4] (Rule: 0) Executing operator "Within" with param "/proxy/ /lock-token/ /content-range/ /translate/ /if/" Was: "" against TX:regex(^HEADER_NAME_).
[4] Rule returned 0.
[4] (Rule: 920014) Executing operator "Lt" with param "2" against TX:PARANOIA_LEVEL.
[4] Rule returned 0.
[4] (Rule: 920200) Executing operator "Rx" with param "^bytes=((\d+)?\-(\d+)?\s*,?\s*){6}" against REQUEST_HEADERS:Range|REQUEST_HEADERS:Request-Range.
[4] Rule returned 0.
[4] (Rule: 920201) Executing operator "EndsWith" with param ".pdf" against REQUEST_BASENAME.
[4] Rule returned 0.

[csanders-git]

As this appears in part to be a libmodsec problem @zimmerle would you like me to xpost this?

[theMiddleBlue]

@csanders-git doing some tests, it seems that the latest working commit is SpiderLabs/ModSecurity@f928e44

[theMiddleBlue]

Doing some tests, I've found the problem: v3/master ignore this variable when it's set using lowercase variable name...

Working configuration:

SecAction \
  "id:900000,\
   phase:1,\
   nolog,\
   pass,\
   t:none,\
   setvar:TX.PARANOIA_LEVEL=1"

Wrong configuration:

SecAction \
  "id:900000,\
   phase:1,\
   nolog,\
   pass,\
   t:none,\
   setvar:tx.paranoia_level=1"

Maybe this problem could affects any other SecRule on the CRS that set a lowercase variable name.

[lifeforms]

Hmmmm, that should not happen...

From the ModSecurity handbook:

[theMiddleBlue]

Just tested more:

OK:

SecAction \
  "id:900000,\
   phase:1,\
   nolog,\
   pass,\
   t:none,\
   setvar:TX.PARANOIA_LEVEL=1"

KO:

SecAction \
  "id:900000,\
   phase:1,\
   nolog,\
   pass,\
   t:none,\
   setvar:tx.paranoia_level=1"

KO:

SecAction \
  "id:900000,\
   phase:1,\
   nolog,\
   pass,\
   t:none,\
   setvar:TX.paranoia_level=1"

KO:

SecAction \
  "id:900000,\
   phase:1,\
   nolog,\
   pass,\
   t:none,\
   setvar:TX.parANoia_LEvel=1"

[csanders-git]

So if I had to guess that cfg for variable needs to be case insensitive @zimmerle

[michaelgranzow-avi]

I think this bug has been introduced by https://github.com/SpiderLabs/ModSecurity/commit/892beb53603a44a3b7cdcbfe83eff9e8c51c9307

Here, the lookup was changed from using std::multimap::equal_range with custom hash and equal functions (both case-insensitive) to std::string::compare, which is case-sensitive:

```.c

• auto range = this->equal_range(var);
• for (auto it = range.first; it != range.second; ++it) {
• l->insert(l->begin(), new VariableValue(&m_name, &var, &it->second));
• for (auto &a : *this) {
• if (a.first.compare(0, var.size(), var) == 0) {
• l->insert(l->begin(), new VariableValue(&m_name, &var, &a.second));
• }
  ```

As a side-note, this change also seems to have turned a hash lookup into a linear search, so it may well have performance impact.

[victorhora]

Thanks for the detailed reporting guys.

This is being handled at https://github.com/SpiderLabs/ModSecurity/issues/1808. PR #1810 is up for evaluation to fix this issue. My tests went well and the buildbots are going fine for now. Would appreciate if you folks could test and report as well.

Thanks!

[victorhora]

As pointed out by @zimmerle at https://github.com/SpiderLabs/ModSecurity/issues/1808#issuecomment-400395175, this has been fixed by commit https://github.com/SpiderLabs/ModSecurity/commit/d810de9166794be2bee972fffb0e09cb0b8d6ec5. Thanks for your contribution @michaelgranzow-avi :)

I believe this one can be closed now.