Haproxy: [2.0.3] segfault with lua res_set_header

[jbonjean]

Output of haproxy -vv and uname -a

HA-Proxy version 2.0.3 2019/07/23 - https://haproxy.org/
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-format-truncation -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wno-stringop-overflow -Wno-cast-function-type -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
  OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1

Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE -PCRE_JIT +PCRE2 +PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED -REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=8).
Built with OpenSSL version : OpenSSL 1.1.1c  28 May 2019
Running on OpenSSL version : OpenSSL 1.1.1c  28 May 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE2 version : 10.32 2018-09-10
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with the Prometheus exporter as a service

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTX        side=FE|BE     mux=H2
              h2 : mode=HTTP       side=FE        mux=H2
       <default> : mode=HTX        side=FE|BE     mux=H1
       <default> : mode=TCP|HTTP   side=FE|BE     mux=PASS

Available services :
    prometheus-exporter

Available filters :
    [SPOE] spoe
    [COMP] compression
    [CACHE] cache
    [TRACE] trace

Linux 18e77738f5d9 4.19.0-5-amd64 #1 SMP Debian 4.19.37-5 (2019-06-19) x86_64 GNU/Linux

What's the configuration?

• haproxy.cfg

global
    log /dev/log local0
    lua-load /sample.lua
    set-dumpable

defaults
    mode http
    log global
    timeout connect 10s
    timeout client 30s
    timeout server 30s

frontend default
    bind *:80
    http-request lua.segfault
    http-request deny deny_status 200

• sample.lua

core.register_action("segfault", { "http-req" }, function(txn)
    txn.http:res_set_header("x-custom", "segfault")
end)

Steps to reproduce the behavior

Just curl a few times (usually less than 10).

curl localhost

Actual behavior

Segfault.

• HAProxy stdout:

[ALERT] 205/175204 (1) : Current worker #1 (6) exited with code 139 (Segmentation fault)
[ALERT] 205/175204 (1) : exit-on-failure: killing every processes with SIGTERM
[WARNING] 205/175204 (1) : All workers exited. Exiting... (139)

• dmesg:

[102440.131923] haproxy[25332]: segfault at 561c0ced4000 ip 0000561c0b44cefb sp 00007ffeee280450 error 4 in haproxy[561c0b301000+164000]
[102440.131929] Code: ff 0f 84 c8 00 00 00 39 c6 0f 84 c0 00 00 00 83 ee 01 48 89 d0 39 73 2c 74 67 8b 13 8b 08 c1 ea 03 83 ea 01 29 f2 49 8d 14 d0 <33> 0a 8b 7a 04 89 08 33 0a 89 0a c1 ef 1c 31 08 8b 48 04 33 4a 04

• Stack trace extracted from core dump:

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `haproxy -W -db -f /usr/local/etc/haproxy/haproxy.cfg'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  http_add_header (htx=0x561c0cec3360, n=..., v=...) at include/common/htx.h:355
355 include/common/htx.h: No such file or directory.
[Current thread is 1 (Thread 0x7f071795e040 (LWP 6))]
(gdb) bt
#0  http_add_header (htx=0x561c0cec3360, n=..., v=...) at include/common/htx.h:355
#1  0x0000561c0b323cf6 in hlua_http_add_hdr (htxn=0x561c0cec88c8, msg=0x561c0cec3070, L=0x561c0cec7f78) at include/common/ist.h:183
#2  hlua_http_res_set_hdr (L=0x561c0cec7f78) at src/hlua.c:5558
#3  0x00007f0717d3d585 in ?? () from /usr/lib/x86_64-linux-gnu/liblua5.3.so.0
#4  0x00007f0717d4a365 in ?? () from /usr/lib/x86_64-linux-gnu/liblua5.3.so.0
#5  0x00007f0717d3cd82 in ?? () from /usr/lib/x86_64-linux-gnu/liblua5.3.so.0
#6  0x00007f0717d3da35 in lua_resume () from /usr/lib/x86_64-linux-gnu/liblua5.3.so.0
#7  0x0000561c0b322492 in hlua_ctx_resume (yield_allowed=<optimized out>, lua=<optimized out>) at src/hlua.c:1107
#8  hlua_ctx_resume (lua=0x561c0cebd0f0, yield_allowed=1) at src/hlua.c:1075
#9  0x0000561c0b329e4e in hlua_action (rule=0x561c0ce73870, px=0x561c0ce74860, sess=<optimized out>, s=0x561c0cec2ac0, flags=2) at src/hlua.c:6794
#10 0x0000561c0b3534be in htx_req_get_intercept_rule (px=px@entry=0x561c0ce74860, rules=rules@entry=0x561c0ce748b0, s=s@entry=0x561c0cec2ac0, deny_status=deny_status@entry=0x7ffeee280a0c)
    at src/proto_htx.c:3091
#11 0x0000561c0b3569cf in htx_process_req_common (s=s@entry=0x561c0cec2ac0, req=req@entry=0x561c0cec2ad0, an_bit=an_bit@entry=16, px=px@entry=0x561c0ce74860) at src/proto_htx.c:500
#12 0x0000561c0b33bd23 in http_process_req_common (s=s@entry=0x561c0cec2ac0, req=req@entry=0x561c0cec2ad0, an_bit=an_bit@entry=16, px=0x561c0ce74860) at src/proto_http.c:2526
#13 0x0000561c0b368268 in process_stream (t=<optimized out>, context=0x561c0cec2ac0, state=<optimized out>) at src/stream.c:2089
#14 0x0000561c0b43a5f6 in process_runnable_tasks () at src/task.c:412
#15 0x0000561c0b3a6244 in run_poll_loop () at src/haproxy.c:2516
#16 run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:2637
#17 0x0000561c0b3045ac in main (argc=<optimized out>, argv=<optimized out>) at src/haproxy.c:3314

Expected behavior

No segfault.

Do you have any idea what may have caused this?

Nope, only it is related to res_set_header call. Pretty sure that it is not a valid statement in this context (should probably be in http-res), but I think it shouldn't trigger a segfault.

Do you have an idea how to solve the issue?

No.

Additional information

This is the simplest environment I could think of to reproduce the bug. And I used the official Docker image (haproxy:2.0.3).

[capflam]

There is indeed a bug. I'm able to reproduce it. Of courses, a segfault should never happen. But it is unexpected and unsupported to set headers on the response from http-request rules. Unfortunately, in the lua code, there is no check to forbid this kind of actions.

So, to be clear, txn.http:res_* must not be used when the request is evaluated and txn.http:req_* must not be used when the response is evaluated. I will add some checks to prevent any segfault.

By reading your configuration, I guess you want to add a custom header in the deny response sent by HAProxy. A solution to dynamically customize such responses is to use a lua service. This way, you will be able to send from lua a fully customized response. For instance

• my_deny_service.lua

core.register_service("my_deny", "http", function(applet)
    applet:set_status(200)
    applet:add_header("x-custom", "it works")
    applet:start_response()
    applet:send("no segfault expected")
end)

• haproxy.cfg:

global
    ...
    lua-load .../my_deny_service.lua

frontend
    ...
    http-request use-service lua.my_deny

However, I'll try to see with Thierry if it is possible/desirable to have a lua function to send a response from the lua code. Something like http_reply_and_close. So, it would be possible to have action returning custom response on conditions and to let the request go to the server otherwise.

[jbonjean]

Thanks for the detailed explanation, it makes sense. FYI, I also tried with an older version (1.8 I think) and I wasn't able to make it segfault.

Your guess is right, I was fiddling around with the configuration to try to inject specific headers in a deny response sent by HAProxy. The problem with the service is that it doesn't have access to txn, where I store the variables during the action (where I decide if I should deny the request, based on an external service response). I probably could forward the data by injecting it as request headers when the request is denied so it can be accessed in the service, but it feels hacky, and I would still have to use txn for regular backends.

Something like http_reply_and_close would be perfect I think, and make things a lot simpler and more flexible.

[capflam]

You can access variables from a lua applet using the function applet:get_var(). In fact, set_var(), get_var() and unset_var() functions exist in the class TXN but also in lua applets. Of course variables set from the lua are also accessible from HAProxy configuration and vice versa.

So if you set a variable in the scope txn, for instance from a lua action :

   txn.set_var("txn.my-var", "my-value")

you can retrieve it from the lua service:

  applet:get_var("txn.my-var")

[jbonjean]

OK, I don't know why I thought that variables in the transaction scope were not available in services. Thank you very much for taking the time to show me I was wrong!