Modsecurity: line termination issue

[jakubbaka]

Hi All,
I have an issue with Windows line termination. I can see ^M signs in audit log:

Content-ID: <http://tempuri.org/0>^M
Content-Transfer-Encoding: 8bit^M
Content-Type: application/xop+xml;charset=utf-8;type="text/xml"^M
^M
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">xxxx</s:Body></s:Envelope>^M

--43619217-F--
HTTP/1.1 403 Forbidden
Content-Length: 235
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1

--43619217-E--

--43619217-H--
Message: XML parser error: XML: Failed parsing document.

do you know how to play with that?
I have MULTIPART_STRICT_ERROR in security.conf. Should I remove that and set MULTIPART_CRLF_LF_LINES ?

[bostrt]

@jakub-pactera I think the might be the text editor you are using to view the audit log. Are you running ModSecurity in Windows and then viewing the log file in Linux?

[jakubbaka]

@bostrt
I can see this char's in vim.
Modsecurity is on Linux, logs are on the same Linux machine, and the strange "^M" occur when Windows clients attempts to connect to my application.

[bostrt]

The ^M is vim's way of rendering a carriage return character (which is part of Windows' newline). The Windows clients are using the ^M as a separator between request Headers (just like Linux uses \n).

[zimmerle]

Closing the issue based on @bostrt comment.

[jakubbaka]

so why mod_security returns me "Message: XML parser error: XML: Failed parsing document." in case I send perfectly valid XML? I assume this is because of ^M. Please re-open the ticket.

[bostrt]

@jakub-pactera ModSecurity uses libxml for validation so you can try dropping the XML into a file and then running xmllint.

First, create the file with just CR ending:

# echo -n -e '<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">xxxx</s:Body></s:Envelope>\x0d' > test.xml
 # file test.xml 
test.xml: ASCII text, with CR line terminators
# hexdump -C test.xml 
00000000  3c 73 3a 45 6e 76 65 6c  6f 70 65 20 78 6d 6c 6e  |<s:Envelope xmln|
00000010  73 3a 73 3d 22 68 74 74  70 3a 2f 2f 73 63 68 65  |s:s="http://sche|
00000020  6d 61 73 2e 78 6d 6c 73  6f 61 70 2e 6f 72 67 2f  |mas.xmlsoap.org/|
00000030  73 6f 61 70 2f 65 6e 76  65 6c 6f 70 65 2f 22 3e  |soap/envelope/">|
00000040  78 78 78 78 3c 2f 73 3a  42 6f 64 79 3e 3c 2f 73  |xxxx</s:Body></s|
00000050  3a 45 6e 76 65 6c 6f 70  65 3e 0d                 |:Envelope>.|
0000005b

Then, run xmllint:

# xmllint test.xml
test.xml:1: parser error : Opening and ending tag mismatch: Envelope line 1 and Body
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">xxxx</s:Body></s
                                                                             ^
test.xml:1: parser error : Extra content at the end of the document
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">xxxx</s:Body></s

xmllint reports that the issue is actually with the lack a <s:Body> tag.

Here's the correctly formatted XML (still with the trailing CR):

# echo -n -e '<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"><s:Body>xxxx</s:Body></s:Envelope>\x0d' > test.xml
# xmllint test.xml
<?xml version="1.0"?>
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"><s:Body>xxxx</s:Body></s:Envelope>

And validation passes.

I think the issue is due to that actual XML issue and not the ^M (carriage return). However, if you intentionally obfuscated the XML when you opened this issue, please do provide the full XML or try validation on your end using xmllint.

Finally, I didn't address your question about using MULTIPART_CRLF_LF_LINES earlier. I do not think that would work since your request is not multipart Content-Type.

[jakubbaka]

@bostrt
thanks for your replay.
I saved my XML to file

$echo -n -e '<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"><s:Body xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">xxxx</s:Body></s:Envelope>\x0d' > my_test.xml

$file my_test.xml
my_test.xml: ASCII text, with very long lines, with CRLF line terminators

$ xmllint my_test.xml
<?xml version="1.0"?>
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"><s:Body xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">xxxxx</s:Body></s:Envelope>

Looks like my XML is perfectly valid. Still don't know why mod_security cannot parse it.
There is only one difference:
your XML: with CR line terminators
my one: with CRLF line terminators

[bostrt]

@jakub-pactera I'm still not able to reproduce that on my side (tried using curl and then netcat to really control the request).

Not sure why I have asked yet, but what version of ModSecurity and libxml are you using?

[jakubbaka]

@bostrt
libxml2-2.9.1-6.el7_2.3.x86_64
mod_security-2.7.3-5.el7.x86_64
mod_security_crs-2.2.9-1.el7.noarch
on Centos 7.3.1611

[bostrt]

@jakub-pactera I definitely should have asked that earlier, 2.7.3 is quite old at this point. Also, given I can't reproduce similar behavior on 2.9.x, you should definitely update to the latest version. FYI, I'm also running libxml2-2.9.3.