Modsecurity: nginx segfault in libmodsecurity.so.3.0.0

[brianp9906]

When attempting to remove a false positive from CRS Rules, Nginx stops responding and generates segmentation faults with libmodsecurity.

file: RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf
add: SecRuleUpdateTargetById 941120 "!REQUEST_HEADERS:Referer"

systemctl reload nginx

Check /var/log/messages:

kernel: nginx[58950]: segfault at 28 ip 00007f59e985af4a sp 00007ffd0657cd80 error 4 in libmodsecurity.so.3.0.0[7f59e9740000+1ed000]

libmodsecurity config output from compiling:

`ModSecurity - v3.0.0-48-ga66aceb for Linux

Mandatory dependencies

• libInjection ....v3.0.0-48-ga66aceb
• SecLang tests ....a66aceb

Optional dependencies

• GeoIP ....found v1.5.0
  -lGeoIP , -I/usr/include/
• LibCURL ....found v7.29.0
  -lcurl , -DWITH_CURL
• YAJL ....found v2.0.4
  -lyajl , -DWITH_YAJL
• LMDB ....not found
• LibXML2 ....found v2.9.1
  -lxml2 -lz -lm -ldl, -I/usr/include/libxml2 -DWITH_LIBXML2
• SSDEEP ....not found
• LUA ....not found

Other Options

• Test Utilities ....enabled
• SecDebugLog ....enabled
• afl fuzzer ....disabled
• library examples ....enabled
• Building parser ....disabled
• Treating pm operations as critical section ....disabled`

[victorhora]

@brianp9906 Are you using any other 3rd party modules or Kubernetes? See ModSecurity-nginx/#74

[brianp9906]

Nope. Just nginx official repo for easy yum installation. Then used the libmodsecurity compile recipe for CentOS 7. I’m using CRS 3.1 and simply putting an exclusion in the file I mentioned.

[zimmerle]

Hi @brianp9906,

Can you share the output of nginx -V on gists?

[brianp9906]

Sure. Nothing special here. Just used Nginx centos7 repo:
[nginx]
name=nginx_repo
baseurl=http://nginx.org/packages/mainline/centos/7/$basearch/
gpgcheck=0
enabled=1

nginx -V output:

nginx version: nginx/1.13.9
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC)
built with OpenSSL 1.0.2k-fips 26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'

[victorhora]

@brianp9906 Thanks for sharing that. Do you know if you have any "conditionals" like "if"/"else"/"then" inside your Nginx.conf? Would mind sharing the conf as well?

Thanks.

[zimmerle]

Running a very simple config with the same environment, I was not able to reproduce the issue.

[LeSwiss]

I guess I have exactly the same issue.

Running nginx plus:

[nginx@stwnetvm01v nginx]# nginx -V
nginx version: nginx/1.13.7 (nginx-plus-r14-p1)
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC)
built with OpenSSL 1.0.1e-fips 11 Feb 2013
TLS SNI support enabled
configure arguments: --build=nginx-plus-r14-p1 --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_jwt_module --with-http_auth_request_module --with-http_dav_module --with-http_f4f_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_hls_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_session_log_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'

As soon as I activate just one SecRuleRemoveByTag, even an example in RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf like "SecRuleUpdateTargetByTag "attack-sqli" "!ARGS:foo"", I receive a ERR_EMPTY_RESPONSE on browsing every domain behind this nginx / WAF instance and some of theses messages:

Apr  7 01:32:33 stwnetvm01v kernel: nginx[108260]: segfault at 28 ip 00007f0c00d2a95a sp 00007ffc145323f0 error 4 in libmodsecurity.so.3.0.0[7f0c00c12000+1f4000]
Apr  7 01:32:33 stwnetvm01v kernel: nginx[108255]: segfault at 28 ip 00007f0c00d2a95a sp 00007ffc145323f0 error 4 in libmodsecurity.so.3.0.0[7f0c00c12000+1f4000]
Apr  7 01:32:33 stwnetvm01v kernel: nginx[108257]: segfault at 28 ip 00007f0c00d2a95a sp 00007ffc145323f0 error 4 in libmodsecurity.so.3.0.0[7f0c00c12000+1f4000]
Apr  7 01:32:33 stwnetvm01v kernel: nginx[108378]: segfault at 28 ip 00007f0c00d2a95a sp 00007ffc14532470 error 4 in libmodsecurity.so.3.0.0[7f0c00c12000+1f4000]
Apr  7 01:32:33 stwnetvm01v kernel: nginx[108262]: segfault at 28 ip 00007f0c00d2a95a sp 00007ffc145323f0 error 4 in libmodsecurity.so.3.0.0[7f0c00c12000+1f4000]
Apr  7 01:32:33 stwnetvm01v kernel: nginx[108379]: segfault at 28 ip 00007f0c00d2a95a sp 00007ffc14532470 error 4 in libmodsecurity.so.3.0.0[7f0c00c12000+1f4000]
Apr  7 01:32:33 stwnetvm01v kernel: nginx[112407]: segfault at 28 ip 00007f0c00d2a95a sp 00007ffc14532470 error 4 in libmodsecurity.so.3.0.0[7f0c00c12000+1f4000]
Apr  7 01:32:39 stwnetvm01v kernel: nginx[112409]: segfault at 28 ip 00007f0c00d2a95a sp 00007ffc14532470 error 4 in libmodsecurity.so.3.0.0[7f0c00c12000+1f4000]
Apr  7 01:33:09 stwnetvm01v kernel: nginx[108253]: segfault at 28 ip 00007f0c00d2a95a sp 00007ffc145323f0 error 4 in libmodsecurity.so.3.0.0[7f0c00c12000+1f4000]

SecRuleRemoveById on the other hand is working as expected.

No if / else used anywhere in my config.

Any help appreciated.
Kind regards.

[LeSwiss]

Nobody can help here?

[victorhora]

@LeSwiss

Can you confirm your version of libModSecurity and nginx-connector?

Maybe ctl:ruleRemovebyTag works for you? It was recently implemented here.

[zimmerle]

@defanator as it is an nginx plus can you help us to try to reproduce the issue? Not able to reproduce in a regular one.

[defanator]

@zimmerle sure thing, I'll take a look. Which "regular" versions of nginx have you tried? I suppose, you've been trying with current v3/master?

[zimmerle]

Not sure. I will check if @victorhora. We managed to reproduce with "ifs" and others conditional configuration tags. Not sure if only happens on the plus version or not, the fact is that your input will be helpful for us.

[defanator]

@zimmerle @victorhora I was able to trigger segfaults with nginx/foss, will try to get minimal reproducible configuration and share it here.

[defanator]

@zimmerle @victorhora - minimal configuration to reproduce (nginx.conf):

user nginx;
worker_processes 1;

load_module /home/test/ngx_http_modsecurity_module-debug.so;

error_log /var/log/nginx/error.log debug;

pid /var/run/nginx.pid;

events { }

http {
    access_log off;

    server {
        listen 80;

        modsecurity on;
        modsecurity_rules_file /etc/nginx/modsec/main.conf;

        error_page   500 502 503 504  /50x.html;
           location = /50x.html {
           root   /etc/nginx/html;
        }

        location / {
          proxy_pass http://nginx.org;
        }
    }
}

/etc/nginx/modsec/main.conf:

include /etc/nginx/modsec/modsecurity.conf
include /etc/nginx/modsec/owasp-crs/crs-setup.conf
include /etc/nginx/modsec/owasp-crs/rules/*.conf
SecRuleUpdateTargetByTag "attack-sqli" "!ARGS:foo"

modsecurity.conf is the default one, owasp-crs/ contains default set of CRS 3.0.2.

Segfault is triggering every single request to the default (/) location. Segfault can be avoided by either of the following:

• removing SecRuleUpdateTargetByTag,
• removing "error_page" + "location = /50x.html" block.

Backtrace:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  modsecurity::Rule::getFinalVars (this=this@entry=0x55a0343bb670, trans=trans@entry=0x55a034311cd0) at rule.cc:441
441         if (a.second->m_isExclusion) {
(gdb) bt
#0  modsecurity::Rule::getFinalVars (this=this@entry=0x55a0343bb670, trans=trans@entry=0x55a034311cd0) at rule.cc:441
#1  0x00007f4db20b5848 in modsecurity::Rule::evaluate (this=0x55a0343bb670, trans=0x55a034311cd0, ruleMessage=std::shared_ptr (count 1, weak 0) 0x55a0350cc7b0) at rule.cc:802
#2  0x00007f4db20a919f in modsecurity::Rules::evaluate (this=0x55a03505d790, phase=phase@entry=3, transaction=transaction@entry=0x55a034311cd0) at rules.cc:247
#3  0x00007f4db209349e in modsecurity::Transaction::processRequestBody (this=0x55a034311cd0) at transaction.cc:825
#4  0x00007f4db20945b5 in modsecurity::msc_process_request_body (transaction=<optimized out>) at transaction.cc:1895
#5  0x00007f4db23b96f6 in ngx_http_modsecurity_pre_access_handler (r=0x55a034370380) at ../ModSecurity-nginx/src/ngx_http_modsecurity_pre_access.c:199
#6  0x000055a0334944d3 in ngx_http_core_generic_phase ()
#7  0x000055a03348fc8d in ngx_http_core_run_phases ()
#8  0x000055a03349bace in ngx_http_process_request ()
#9  0x000055a03349c4e8 in ?? ()
#10 0x000055a033481fb4 in ?? ()
#11 0x000055a033476bba in ngx_process_events_and_timers ()
#12 0x000055a03347fa15 in ?? ()
#13 0x000055a03347de0e in ngx_spawn_process ()
#14 0x000055a03347f0c0 in ?? ()
#15 0x000055a03348075d in ngx_master_process_cycle ()
#16 0x000055a033454d5c in main ()
(gdb) 

Full backtrace:

(gdb) bt full
#0  modsecurity::Rule::getFinalVars (this=this@entry=0x55a0343bb670, trans=trans@entry=0x55a034311cd0) at rule.cc:441
        __for_range = <optimized out>
        exclusions = {<std::__cxx11::_List_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >> = {
            _M_impl = {<std::allocator<std::_List_node<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >> = {<__gnu_cxx::new_allocator<std::_List_node<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >> = {<No data fields>}, <No data fields>}, _M_node = {<std::__detail::_List_node_base> = {_M_next = 0x7ffd9b84efa0, _M_prev = 0x7ffd9b84efa0}, 
                _M_data = 0}}}, <No data fields>}
        exclusions_update_by_tag_remove = {<std::__cxx11::_List_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >> = {
            _M_impl = {<std::allocator<std::_List_node<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >> = {<__gnu_cxx::new_allocator<std::_List_node<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >> = {<No data fields>}, <No data fields>}, _M_node = {<std::__detail::_List_node_base> = {_M_next = 0x7ffd9b84efc0, _M_prev = 0x7ffd9b84efc0}, 
                _M_data = 0}}}, <No data fields>}
        exclusions_update_by_msg_remove = {<std::__cxx11::_List_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >> = {
            _M_impl = {<std::allocator<std::_List_node<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >> = {<__gnu_cxx::new_allocator<std::_List_node<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >> = {<No data fields>}, <No data fields>}, _M_node = {<std::__detail::_List_node_base> = {_M_next = 0x7ffd9b84efe0, _M_prev = 0x7ffd9b84efe0}, 
                _M_data = 0}}}, <No data fields>}
        exclusions_update_by_id_remove = {<std::__cxx11::_List_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >> = {
            _M_impl = {<std::allocator<std::_List_node<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >> = {<__gnu_cxx::new_allocator<std::_List_node<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >> = {<No data fields>}, <No data fields>}, _M_node = {<std::__detail::_List_node_base> = {_M_next = 0x7ffd9b84f000, _M_prev = 0x7ffd9b84f000}, 
                _M_data = 0}}}, <No data fields>}
        variables = std::vector of length 8, capacity 8 = {0x55a0343b83b0, 0x55a0343b8800, 0x55a0343b88e0, 0x55a0343b89c0, 0x55a0343b8af0, 0x55a0343b8bf0, 0x55a0343b8ca0, 0x55a0343b8d50}
        finalVars = std::vector of length 0, capacity 0
#1  0x00007f4db20b5848 in modsecurity::Rule::evaluate (this=0x55a0343bb670, trans=0x55a034311cd0, ruleMessage=std::shared_ptr (count 1, weak 0) 0x55a0350cc7b0) at rule.cc:802
        globalRet = false
        variables = <optimized out>
        recursiveGlobalRet = <optimized out>
        containsDisruptive = false
        finalVars = std::vector of length 0, capacity 0
        eparam = ""
#2  0x00007f4db20a919f in modsecurity::Rules::evaluate (this=0x55a03505d790, phase=phase@entry=3, transaction=transaction@entry=0x55a034311cd0) at rules.cc:247
        remove_rule = <optimized out>
        rule = 0x55a0343bb670
        i = 242
        rules = std::vector of length -11768321627458, capacity -11768321627458 = {0x55a034388ee0, 0x55a03438e6a0, 0x55a03438edf0, 0x55a03438f5f0, 0x55a034405530, 0x55a0343f1500, 0x55a0343f1b70, 
          0x55a0343f2340, 0x55a0343f28c0, 0x55a0343f3130, 0x55a0343f3ac0, 0x55a0343f4090, 0x55a034405c80, 0x55a034406090, 0x55a0344066c0, 0x55a034406d70, 0x55a0344073e0, 0x55a034407d70, 0x55a03440ca50, 
          0x55a03440d1a0, 0x55a03440dad0, 0x55a03440e110, 0x55a03440e810, 0x55a03440eff0, 0x55a034413810, 0x55a034413ed0, 0x55a0344145e0, 0x55a034414bb0, 0x55a0343eda00, 0x55a0343ee0d0, 0x55a0343ee790, 
          0x55a0343efc70, 0x55a0343f0300, 0x55a034415ae0, 0x55a034419a30, 0x55a03441a120, 0x55a03441b330, 0x55a03441c7c0, 0x55a03441ddd0, 0x55a03441f250, 0x55a0344215c0, 0x55a034421ab0, 0x55a034422ff0, 
          0x55a034424800, 0x55a0344262c0, 0x55a034429d60, 0x55a03442c0f0, 0x55a03442d310, 0x55a03442e580, 0x55a034430ba0, 0x55a034431010, 0x55a0344335f0, 0x55a034433ca0, 0x55a0344345d0, 0x55a0343ebc40, 
          0x55a0343e4a70, 0x55a0343e6520, 0x55a034435040, 0x55a034435740, 0x55a0344376b0, 0x55a034438d30, 0x55a03443b470, 0x55a03443dbf0, 0x55a034440380, 0x55a034442900, 0x55a0344431c0, 0x55a034443af0, 
          0x55a034444ae0, 0x55a0344457c0, 0x55a0344464a0, 0x55a034446ad0, 0x55a0343e0c40, 0x55a0343e2e90, 0x55a0343e3a30, 0x55a0344473a0, 0x55a034448100, 0x55a034448770, 0x55a0343df000, 0x55a034451560, 
          0x55a034453bf0, 0x55a034454930, 0x55a034455100, 0x55a034455d20, 0x55a0343d8440, 0x55a0343da8e0, 0x55a0344ece10, 0x55a0344ef600, 0x55a034504700, 0x55a034522980, 0x55a034525140, 0x55a034542b00, 
          0x55a034543770, 0x55a034543d30, 0x55a0343d4470, 0x55a0343d6610, 0x55a034548e40, 0x55a03454ac90, 0x55a03454dd40, 0x55a034550f40, 0x55a0345533c0, 0x55a034555b10, 0x55a034558a00, 0x55a034559960, 
          0x55a03455ba20, 0x55a03455e010, 0x55a034560f90, 0x55a034562db0, 0x55a034564d50, 0x55a034566920, 0x55a034566e60, 0x55a034568180, 0x55a03456cba0, 0x55a03456ff30, 0x55a034570cf0, 0x55a034574030, 
          0x55a034574f50, 0x55a0345770c0, 0x55a0345792f0, 0x55a03457b510, 0x55a03457d6e0, 0x55a03457fda0, 0x55a0345826f0, 0x55a03458a130, 0x55a03458bca0, 0x55a03458ed40, 0x55a034590cc0, 0x55a0345922c0, 
          0x55a034594610, 0x55a0345975e0, 0x55a034598ed0, 0x55a03459d2f0, 0x55a03459f340, 0x55a0345a1410, 0x55a0345a1f90, 0x55a0345a40f0, 0x55a0345a4d50, 0x55a0345a6130, 0x55a0345a9240, 0x55a0345ab540, 
          0x55a0345aded0, 0x55a0345ae540, 0x55a0343d0410, 0x55a0343d2d70, 0x55a0345b04a0, 0x55a0345b30d0, 0x55a0345b9f90, 0x55a0345bc660, 0x55a0345bec70, 0x55a0345c15b0, 0x55a0345c2200, 0x55a0345c4980, 
          0x55a0345c55e0, 0x55a0345c6690, 0x55a0345c7f20, 0x55a0345ca180, 0x55a0345ca730, 0x55a0343cc540, 0x55a0343cf5a0, 0x55a0345cd4c0, 0x55a0345ce520, 0x55a03489adb0, 0x55a0348b3210, 0x55a0348b3e90, 
          0x55a0348b4b10, 0x55a0348b50e0, 0x55a0343c8420, 0x55a0343ca990, 0x55a0348b6cd0, 0x55a0348b92e0, 0x55a0348b9fd0, 0x55a0348bb810, 0x55a0348bd980, 0x55a0348be680, 0x55a0348bec90, 0x55a0343c4410, 
          0x55a0348ca890, 0x55a0348d57c0, 0x55a0348e15e0, 0x55a0348ed7e0, 0x55a0348efa50, 0x55a0349732a0, 0x55a034976480, 0x55a034982d50, 0x55a034985ab0, 0x55a034998480, 0x55a03499af50, 0x55a03499bc60, 
          0x55a03499c960, 0x55a03499d660, 0x55a03499de60, 0x55a0343c0330, 0x55a0343c3120, 0x55a0349a04c0, 0x55a0349a15b0, 0x55a034a52430, 0x55a034a5c1b0, 0x55a034a5f060, 0x55a034a7c150, 0x55a034a7f130, 
          0x55a034a82040...}
#3  0x00007f4db209349e in modsecurity::Transaction::processRequestBody (this=0x55a034311cd0) at transaction.cc:825
        a = std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >> containing 0x0
        fullRequest = "Host: localhost\nUser-Agent: curl/7.47.0\nAccept: */*\n\n\n"
        l = std::vector of length 3, capacity 4 = {0x55a0350a6c60, 0x55a0350a5270, 0x55a0350a53a0}
#4  0x00007f4db20945b5 in modsecurity::msc_process_request_body (transaction=<optimized out>) at transaction.cc:1895
No locals.
#5  0x00007f4db23b96f6 in ngx_http_modsecurity_pre_access_handler (r=0x55a034370380) at ../ModSecurity-nginx/src/ngx_http_modsecurity_pre_access.c:199
        ret = <optimized out>
        already_inspected = <optimized out>
        chain = <optimized out>
        ctx = 0x55a034371030
        cf = <optimized out>
        old_pool = 0x0
        __func__ = "ngx_http_modsecurity_pre_access_handler"
#6  0x000055a0334944d3 in ngx_http_core_generic_phase (r=0x55a034370380, ph=0x55a03437e268) at src/http/ngx_http_core_module.c:880
        rc = <optimized out>
#7  0x000055a03348fc8d in ngx_http_core_run_phases (r=r@entry=0x55a034370380) at src/http/ngx_http_core_module.c:858
        rc = <optimized out>
        ph = 0x55a03437e1d8
        cmcf = <optimized out>
#8  0x000055a03348fd82 in ngx_http_handler (r=r@entry=0x55a034370380) at src/http/ngx_http_core_module.c:841
        cmcf = <optimized out>
#9  0x000055a03349bace in ngx_http_process_request (r=0x55a034370380) at src/http/ngx_http_request.c:1952
        c = 0x55a0350652a0
#10 0x000055a03349c4e8 in ngx_http_process_request_line (rev=0x55a035083190) at src/http/ngx_http_request.c:1052
        n = <optimized out>
        rc = <optimized out>
        rv = <optimized out>
        host = {len = 1, 
          data = 0x55a03349c658 <ngx_http_wait_request_handler+280> "H\205\300H\211\003\017\204\021\377\377\377H\215\005\265\372\377\377H\211\357H\211E\020H\203\304\030[]A\\A]\351\237\372\377\377\017\037\200"}
        c = 0x55a0350652a0
        r = 0x55a034370380
#11 0x000055a033481fb4 in ngx_epoll_process_events (cycle=0x55a03430cd00, timer=<optimized out>, flags=<optimized out>) at src/event/modules/ngx_epoll_module.c:902
        events = 1
        revents = 1
        instance = <optimized out>
        i = 0
        level = <optimized out>
        err = <optimized out>
        rev = 0x55a035083190
        wev = <optimized out>
        queue = <optimized out>
        c = 0x55a0350652a0
#12 0x000055a033476bba in ngx_process_events_and_timers (cycle=cycle@entry=0x55a03430cd00) at src/event/ngx_event.c:242
        flags = <optimized out>
        timer = <optimized out>
        delta = 717629906
#13 0x000055a03347fa15 in ngx_worker_process_cycle (cycle=cycle@entry=0x55a03430cd00, data=data@entry=0x0) at src/os/unix/ngx_process_cycle.c:750
        worker = 0
#14 0x000055a03347de0e in ngx_spawn_process (cycle=cycle@entry=0x55a03430cd00, proc=proc@entry=0x55a03347f9c0 <ngx_worker_process_cycle>, data=data@entry=0x0, 
    name=name@entry=0x55a03353085c "worker process", respawn=respawn@entry=-3) at src/os/unix/ngx_process.c:199
        on = 1
        pid = 0
        s = 0
#15 0x000055a03347f0c0 in ngx_start_worker_processes (cycle=cycle@entry=0x55a03430cd00, n=1, type=type@entry=-3) at src/os/unix/ngx_process_cycle.c:359
        i = 0
        ch = {command = 1, pid = 0, slot = 0, fd = 0}
#16 0x000055a03348075d in ngx_master_process_cycle (cycle=0x55a03430cd00) at src/os/unix/ngx_process_cycle.c:131
        title = 0x55a03437e475 "master process nginx-debug"
        p = <optimized out>
        size = <optimized out>
        i = <optimized out>
        n = <optimized out>
        sigio = <optimized out>
        set = {__val = {0 <repeats 16 times>}}
        itv = {it_interval = {tv_sec = 0, tv_usec = 0}, it_value = {tv_sec = 0, tv_usec = 0}}
        live = <optimized out>
        delay = <optimized out>
        ls = <optimized out>
        ccf = 0x55a03430eae0
#17 0x000055a033454d5c in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:382
        b = <optimized out>
        log = 0x55a033787140 <ngx_log>
        i = <optimized out>
        cycle = 0x55a03430cd00
        init_cycle = {conf_ctx = 0x0, pool = 0x55a03430c110, log = 0x55a033787140 <ngx_log>, new_log = {log_level = 0, file = 0x0, connection = 0, disk_full_time = 0, handler = 0x0, data = 0x0, 
            writer = 0x0, wdata = 0x0, action = 0x0, next = 0x0}, log_use_stderr = 0, files = 0x0, free_connections = 0x0, free_connection_n = 0, modules = 0x0, modules_n = 0, modules_used = 0, 
          reusable_connections_queue = {prev = 0x0, next = 0x0}, reusable_connections_n = 0, listening = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, paths = {elts = 0x0, nelts = 0, size = 0, 
            nalloc = 0, pool = 0x0}, config_dump = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, config_dump_rbtree = {root = 0x0, sentinel = 0x0, insert = 0x0}, config_dump_sentinel = {
            key = 0, left = 0x0, right = 0x0, parent = 0x0, color = 0 '\000', data = 0 '\000'}, open_files = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, 
          shared_memory = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, connection_n = 0, files_n = 0, connections = 0x0, read_events = 0x0, write_events = 0x0, 
          old_cycle = 0x0, conf_file = {len = 21, data = 0x55a03352b950 "/etc/nginx/nginx.conf"}, conf_param = {len = 0, data = 0x0}, conf_prefix = {len = 11, data = 0x55a03352b950 "/etc/nginx/nginx.conf"}, 
          prefix = {len = 11, data = 0x55a03352b944 "/etc/nginx/"}, lock_file = {len = 0, data = 0x0}, hostname = {len = 0, data = 0x0}}
        cd = <optimized out>
        ccf = 0x55a03430eae0
(gdb) 

I used latest v3/master for library and master for nginx connector with nginx/1.13.10 opensource.

[zimmerle]

Thanks for the detailed report @defanator. Did you managed to reproduce without the error conditional?

[defanator]

@zimmerle "without the error conditional" ?

[zimmerle]

        error_page   500 502 503 504  /50x.html;
           location = /50x.html {
           root   /etc/nginx/html;
        }

This is a custom error page that changes the regular nginix flow right? In case of an error, its uses a custom location. Right?

[defanator]

@zimmerle right, see the corresponding doc: http://nginx.org/r/error_page

However, in my case there were no any 5xx errors, segfault is happening during the initial request processing (note the "test location" and "using configuration" in the below debug log):

2018/04/22 21:36:17 [debug] 11969#11969: epoll: fd:8 ev:0001 d:00005574259B10C0
2018/04/22 21:36:17 [debug] 11969#11969: accept on 0.0.0.0:80, ready: 0
2018/04/22 21:36:17 [debug] 11969#11969: posix_memalign: 0000557424C58110:512 @16
2018/04/22 21:36:17 [debug] 11969#11969: *1 accept: 127.0.0.1:1098 fd:3
2018/04/22 21:36:17 [debug] 11969#11969: *1 event timer add: 3: 60000:1031631337
2018/04/22 21:36:17 [debug] 11969#11969: *1 reusable connection: 1
2018/04/22 21:36:17 [debug] 11969#11969: *1 epoll add event: fd:3 op:1 ev:80002001
2018/04/22 21:36:17 [debug] 11969#11969: timer delta: 28343
2018/04/22 21:36:17 [debug] 11969#11969: worker cycle
2018/04/22 21:36:17 [debug] 11969#11969: epoll timer: 60000
2018/04/22 21:36:17 [debug] 11969#11969: epoll: fd:3 ev:0001 d:00005574259B12A0
2018/04/22 21:36:17 [debug] 11969#11969: *1 http wait request handler
2018/04/22 21:36:17 [debug] 11969#11969: *1 malloc: 0000557424CCCC00:1024
2018/04/22 21:36:17 [debug] 11969#11969: *1 recv: eof:0, avail:1
2018/04/22 21:36:17 [debug] 11969#11969: *1 recv: fd:3 73 of 1024
2018/04/22 21:36:17 [debug] 11969#11969: *1 reusable connection: 0
2018/04/22 21:36:17 [debug] 11969#11969: *1 posix_memalign: 0000557424CBC330:4096 @16
2018/04/22 21:36:17 [debug] 11969#11969: *1 http process request line
2018/04/22 21:36:17 [debug] 11969#11969: *1 http request line: "GET / HTTP/1.1"
2018/04/22 21:36:17 [debug] 11969#11969: *1 http uri: "/"
2018/04/22 21:36:17 [debug] 11969#11969: *1 http args: ""
2018/04/22 21:36:17 [debug] 11969#11969: *1 http exten: ""
2018/04/22 21:36:17 [debug] 11969#11969: *1 posix_memalign: 0000557424C5CCC0:4096 @16
2018/04/22 21:36:17 [debug] 11969#11969: *1 http process request header line
2018/04/22 21:36:17 [debug] 11969#11969: *1 http header: "Host: localhost"
2018/04/22 21:36:17 [debug] 11969#11969: *1 http header: "User-Agent: curl/7.47.0"
2018/04/22 21:36:17 [debug] 11969#11969: *1 http header: "Accept: */*"
2018/04/22 21:36:17 [debug] 11969#11969: *1 http header done
2018/04/22 21:36:17 [debug] 11969#11969: *1 event timer del: 3: 1031631337
2018/04/22 21:36:17 [debug] 11969#11969: *1 generic phase: 0
2018/04/22 21:36:17 [debug] 11969#11969: *1 rewrite phase: 1
2018/04/22 21:36:17 [debug] 11969#11969: *1 test location: "/"
2018/04/22 21:36:17 [debug] 11969#11969: *1 using configuration "/"
2018/04/22 21:36:17 [debug] 11969#11969: *1 http cl:-1 max:1048576
2018/04/22 21:36:17 [debug] 11969#11969: *1 rewrite phase: 3
modsec *** ngx_http_modsecurity_rewrite_handler: catching a new _rewrite_ phase handler at ../ModSecurity-nginx/src/ngx_http_modsecurity_rewrite.c line 45.
modsec *** ngx_http_modsecurity_rewrite_handler: recovering ctx: (nil) at ../ModSecurity-nginx/src/ngx_http_modsecurity_rewrite.c line 49.
modsec *** ngx_http_modsecurity_create_ctx: creating transaction with the following rules: '0x5574259a9790' -- ms: '0x557424cadef0' at ../ModSecurity-nginx/src/ngx_http_modsecurity_module.c line 251.
modsec *** ngx_http_modsecurity_create_ctx: transaction created at ../ModSecurity-nginx/src/ngx_http_modsecurity_module.c line 255.
2018/04/22 21:36:17 [debug] 11969#11969: *1 add cleanup: 0000557424CBD058
modsec *** ngx_http_modsecurity_rewrite_handler: ctx was NULL, creating new context: 0x557424cbd030 at ../ModSecurity-nginx/src/ngx_http_modsecurity_rewrite.c line 64.
modsec *** ngx_http_modsecurity_rewrite_handler: Processing intervention with the connection information filled in at ../ModSecurity-nginx/src/ngx_http_modsecurity_rewrite.c line 103.
modsec *** ngx_http_modsecurity_process_intervention: processing intervention at ../ModSecurity-nginx/src/ngx_http_modsecurity_module.c line 142.
modsec *** ngx_http_modsecurity_process_intervention: nothing to do at ../ModSecurity-nginx/src/ngx_http_modsecurity_module.c line 145.
modsec *** ngx_http_modsecurity_rewrite_handler: Processing intervention with the transaction information filled in (uri, method and version) at ../ModSecurity-nginx/src/ngx_http_modsecurity_rewrite.c line 139.
modsec *** ngx_http_modsecurity_process_intervention: processing intervention at ../ModSecurity-nginx/src/ngx_http_modsecurity_module.c line 142.
modsec *** ngx_http_modsecurity_process_intervention: nothing to do at ../ModSecurity-nginx/src/ngx_http_modsecurity_module.c line 145.
modsec *** ngx_http_modsecurity_rewrite_handler: Adding request header: Host with value localhost at ../ModSecurity-nginx/src/ngx_http_modsecurity_rewrite.c line 171.
modsec *** ngx_http_modsecurity_rewrite_handler: Adding request header: User-Agent with value curl/7.47.0 at ../ModSecurity-nginx/src/ngx_http_modsecurity_rewrite.c line 171.
modsec *** ngx_http_modsecurity_rewrite_handler: Adding request header: Accept with value */* at ../ModSecurity-nginx/src/ngx_http_modsecurity_rewrite.c line 171.
modsec *** ngx_http_modsecurity_rewrite_handler: Processing intervention with the request headers information filled in at ../ModSecurity-nginx/src/ngx_http_modsecurity_rewrite.c line 187.
modsec *** ngx_http_modsecurity_process_intervention: processing intervention at ../ModSecurity-nginx/src/ngx_http_modsecurity_module.c line 142.
modsec *** ngx_http_modsecurity_process_intervention: nothing to do at ../ModSecurity-nginx/src/ngx_http_modsecurity_module.c line 145.
2018/04/22 21:36:17 [debug] 11969#11969: *1 rewrite phase: 4
2018/04/22 21:36:17 [debug] 11969#11969: *1 post rewrite phase: 5
2018/04/22 21:36:17 [debug] 11969#11969: *1 generic phase: 6
modsec *** ngx_http_modsecurity_pre_access_handler: catching a new _preaccess_ phase handler at ../ModSecurity-nginx/src/ngx_http_modsecurity_pre_access.c line 50.
modsec *** ngx_http_modsecurity_pre_access_handler: recovering ctx: 0x557424cbd030 at ../ModSecurity-nginx/src/ngx_http_modsecurity_pre_access.c line 72.
modsec *** ngx_http_modsecurity_pre_access_handler: asking for the request body, if any. Count: 1 at ../ModSecurity-nginx/src/ngx_http_modsecurity_pre_access.c line 95.
modsec *** ngx_http_modsecurity_pre_access_handler: request body is ready to be processed at ../ModSecurity-nginx/src/ngx_http_modsecurity_pre_access.c line 132.
modsec *** ngx_http_modsecurity_pre_access_handler: inspection request body in memory. at ../ModSecurity-nginx/src/ngx_http_modsecurity_pre_access.c line 161.
2018/04/22 21:36:17 [debug] 11969#11969: *1 malloc: 00005574259F6DF0:4096
2018/04/22 21:36:17 [debug] 11969#11969: *1 malloc: 00005574259F7E00:4096
2018/04/22 21:36:17 [debug] 11969#11969: *1 free: 00005574259F6DF0
2018/04/22 21:36:17 [debug] 11969#11969: *1 free: 00005574259F7E00
2018/04/22 21:36:17 [debug] 11969#11969: *1 malloc: 0000557425A098E0:4096
2018/04/22 21:36:17 [debug] 11969#11969: *1 malloc: 0000557425A0A8F0:4096
2018/04/22 21:36:17 [debug] 11969#11969: *1 free: 0000557425A098E0
2018/04/22 21:36:17 [debug] 11969#11969: *1 free: 0000557425A0A8F0
2018/04/22 21:36:17 [notice] 11968#11968: signal 17 (SIGCHLD) received from 11969
2018/04/22 21:36:17 [alert] 11968#11968: worker process 11969 exited on signal 11 (core dumped)
2018/04/22 21:36:17 [debug] 11968#11968: shmtx forced unlock
2018/04/22 21:36:17 [debug] 11968#11968: wake up, sigio 0
2018/04/22 21:36:17 [debug] 11968#11968: reap children
2018/04/22 21:36:17 [debug] 11968#11968: child: 0 11969 e:0 t:1 d:0 r:1 j:0
2018/04/22 21:36:17 [debug] 11968#11968: channel 3:10
2018/04/22 21:36:17 [notice] 11968#11968: start worker process 11976
2018/04/22 21:36:17 [debug] 11968#11968: sigsuspend
2018/04/22 21:36:17 [debug] 11976#11976: add cleanup: 0000557424CCA490
2018/04/22 21:36:17 [debug] 11976#11976: malloc: 00005574256C4BA0:8
2018/04/22 21:36:17 [debug] 11976#11976: notify eventfd: 12
2018/04/22 21:36:17 [debug] 11976#11976: eventfd: 13
2018/04/22 21:36:17 [debug] 11976#11976: testing the EPOLLRDHUP flag: success
2018/04/22 21:36:17 [debug] 11976#11976: malloc: 0000557424CBAB20:6144
2018/04/22 21:36:17 [debug] 11976#11976: malloc: 00005574259B10C0:122880
2018/04/22 21:36:17 [debug] 11976#11976: malloc: 00005574259CF0D0:49152
2018/04/22 21:36:17 [debug] 11976#11976: malloc: 00005574259DB0E0:49152
2018/04/22 21:36:17 [debug] 11976#11976: epoll add event: fd:8 op:1 ev:00002001
2018/04/22 21:36:17 [debug] 11976#11976: epoll add event: fd:10 op:1 ev:00002001
2018/04/22 21:36:17 [debug] 11976#11976: setproctitle: "nginx: worker process"
2018/04/22 21:36:17 [debug] 11976#11976: worker cycle
2018/04/22 21:36:17 [debug] 11976#11976: epoll timer: -1

[defanator]

JFTR, here's the ASAN output for the same case of segfault:

root@vagrant:/home/test# cat /var/log/nginx/error.log
2018/04/22 23:46:28 [notice] 8796#8796: ModSecurity-nginx v1.0.0
2018/04/22 23:46:28 [notice] 8796#8796: using the "epoll" event method
2018/04/22 23:46:28 [notice] 8796#8796: nginx/1.13.10
2018/04/22 23:46:28 [notice] 8796#8796: built by gcc 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9) 
2018/04/22 23:46:28 [notice] 8796#8796: OS: Linux 4.4.0-116-generic
2018/04/22 23:46:28 [notice] 8796#8796: getrlimit(RLIMIT_NOFILE): 131072:131072
ASAN:SIGSEGV
=================================================================
==8797==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000058 (pc 0x7fecb9f75bfc bp 0x60600008bfe0 sp 0x7ffd52802b30 T0)
    #0 0x7fecb9f75bfb in modsecurity::Rule::getFinalVars(modsecurity::Transaction*) /home/test/ModSecurity/src/rule.cc:441
    #1 0x7fecb9f7c834 in modsecurity::Rule::evaluate(modsecurity::Transaction*, std::shared_ptr<modsecurity::RuleMessage>) /home/test/ModSecurity/src/rule.cc:802
    #2 0x7fecb9f5d97a in modsecurity::Rules::evaluate(int, modsecurity::Transaction*) /home/test/ModSecurity/src/rules.cc:247
    #3 0x7fecb9f30924 in modsecurity::Transaction::processRequestBody() /home/test/ModSecurity/src/transaction.cc:825
    #4 0x5558c11fbbe9 in ngx_http_modsecurity_pre_access_handler ../ModSecurity-nginx/src/ngx_http_modsecurity_pre_access.c:199
    #5 0x5558c10620b9 in ngx_http_core_generic_phase src/http/ngx_http_core_module.c:880
    #6 0x5558c1055d64 in ngx_http_core_run_phases src/http/ngx_http_core_module.c:858
    #7 0x5558c10756a5 in ngx_http_process_request src/http/ngx_http_request.c:1952
    #8 0x5558c10770b8 in ngx_http_process_request_line src/http/ngx_http_request.c:1052
    #9 0x5558c10344fd in ngx_epoll_process_events src/event/modules/ngx_epoll_module.c:902
    #10 0x5558c101a8c8 in ngx_process_events_and_timers src/event/ngx_event.c:242
    #11 0x5558c1032abf in ngx_single_process_cycle src/os/unix/ngx_process_cycle.c:310
    #12 0x5558c0fc28ed in main src/core/nginx.c:379
    #13 0x7fecb8df882f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #14 0x5558c0fc4098 in _start (/home/test/nginx-static+0x8b098)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /home/test/ModSecurity/src/rule.cc:441 modsecurity::Rule::getFinalVars(modsecurity::Transaction*)
==8797==ABORTING

[defanator]

Also, the error_page directive does not have any effect - only additional location block matters.

[defanator]

@zimmerle @victorhora were you able to take a look at this one? Let me know if I could provide anything else here. TIA!

[defanator]

@zimmerle @victorhora ping :)

[victorhora]

Sorry @defanator, getting back to this now. Will provide my the result of my testing here as well. Thanks!

[zimmerle]

fixed in current v3/master.

[defanator]

@zimmerle thanks, I can confirm that this one is not reproduced anymore.