Haproxy: "http-request set-src" affects connection, not transaction or request

[lukastribus]

Michael Brown (@supermathie) reported this, further researched it and provided a workaround on discourse:
https://discourse.haproxy.org/t/cloudflare-haproxy-is-using-the-wrong-ip-for-http-requests-after-the-first/3769

Output of haproxy -vv and uname -a

lukas@dev:~/haproxy$ ./haproxy -vv
HA-Proxy version 2.0-dev2-a48237-261 2019/05/02 - https://haproxy.org/
Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits
  OPTIONS = USE_GETADDRINFO=1

Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE -PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED -REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO -OPENSSL -LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 -ZLIB -SLZ +CPU_AFFINITY -TFO -NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=2).
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built without compression support (neither USE_ZLIB nor USE_SLZ are set).
Compression algorithms supported : identity("identity")
Built without PCRE or PCRE2 support (using libc's regex instead)
Encrypted password support via crypt(3): yes

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTX        side=FE|BE
              h2 : mode=HTTP       side=FE
       <default> : mode=HTX        side=FE|BE
       <default> : mode=TCP|HTTP   side=FE|BE

Available services : none

Available filters :
        [SPOE] spoe
        [COMP] compression
        [CACHE] cache
        [TRACE] trace

lukas@dev:~/haproxy$ uname -a
Linux dev 4.4.0-109-generic #132-Ubuntu SMP Tue Jan 9 19:52:39 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
lukas@dev:~/haproxy$

Not a regression though, from the initial commit 2fbcafc9ce 4 years ago until latest HEAD the behavior is the same.

What's the configuration?

global
 maxconn 100
 log 10.0.0.4 syslog debug

defaults
 mode http
 option httplog
 log global
 timeout connect 10s
 timeout client 120s
 timeout server 120s
 option http-keep-alive

frontend myfrontend
 bind :80
 http-request set-src hdr(x-forwarded-for) #if { src 10.0.0.4 }
 default_backend be_ip

backend be_ip
 server www 10.0.0.33:80

Steps to reproduce the behavior

In a keep-alived client connection, issue multiple subsequent requests with different x-forwarded-for values (certain CDNs or if another frontend haproxy layer has http-reuse enabled)

1. Opening a telnet window to haproxy and paste something like this:

GET / HTTP/1.1
Host: localhost
X-Forwarded-For: 192.168.0.1

GET / HTTP/1.1
Host: localhost
X-Forwarded-For: 192.168.0.2

GET / HTTP/1.1
Host: localhost
X-Forwarded-For: 192.168.0.3

GET / HTTP/1.1
Host: localhost

Actual behavior 1

When set-src is unconditionally called (no src based ACL), the first three request will show up fine in haproxy logs. The forth request will keep the src from the last set-src call (192.168.0.3)

Actual behavior 2

When set-src is conditioned by an ACL (only trust X-Forwarded-For from known trusted proxies or CDNs), then the behavior is worse, because after the first set-src call (192.168.0.1), the ACL stops matching, because we replaced the actual low level connection IP.

Expected behavior

Expected behavior would be to only set the source IP for this specific request/transaction, http-request implies exactly that, without impacting subsequent transactions on the same connection.

Do you have any idea what may have caused this?

set-src works on the connection layer.

Do you have an idea how to solve the issue?

Either we are able to fix the underlying technical issue (I assume this is complex), or we document this behavior clearly along with some workarounds (already provided on discourse by Michael).

[TimWolla]

@lukastribus I guess this should at least be a “severity: medium”, because: “For a bug, it generally indicates something odd which requires changing the configuration in an undesired way to work around the issue.”. The work around is clearly an undesired change. It could even qualify as “severity: major”, because passing incorrect IP addresses could introduce “severe reliability issues” if one uses this to restrict access to trusted IP addresses.

[lukastribus]

Agreed for severity: medium.

But it's not something that crashes and blocks the application at 02 am in the morning after running fine for days. That would be major.

[wtarreau]

I see why this is happening. In the past we didn't have such a feature and the only way to have a request element dictate the source was by using the "source ... usesrc hdr_ip()". There are specific provisions for this in the connection setup code. But here we're facing a new class of problems that was not envisioned previously. I think we need to change the way it works to flag the connection and mention that its source (or possibly destination) was set from L7 and should be rechecked once assigned differently. It's in fact a bit more complicated than this but it should give the idea :-)

[capflam]

1.9 is EOL

[lukastribus]

Just documenting the workaround from @Supermathie with a two changes:

• replaced X-F-F with cf-connecting-ip (X-F-F can contain multiple IPs)
• denying direct access to this domain

# allow cloudflare src ranges (https://www.cloudflare.com/ips-v4 + https://www.cloudflare.com/ips-v6)
acl is_cloudflare src -f /etc/cloudflare/ips-v4
acl is_cloudflare src -f /etc/cloudflare/ips-v6

# WORKAROUND: https://github.com/haproxy/haproxy/issues/90
acl is_cloudflare var(sess.cloudflare) -m found
http-request set-var(sess.cloudflare) always_true if { http_first_req } is_cloudflare

# deny cloudflare bypass for this domain
http-request deny if { hdr(host) -i example.org } ! is_cloudflare

# set true source if cloudflare
http-request set-src hdr(cf-connecting-ip) if is_cloudflare

And for automatic and atomic updates of the file, I call wget from cron, save to temporary file and rename:

wget "https://www.cloudflare.com/ips-v4" -qO /etc/cloudflare/ips-v4.tmp && mv /etc/cloudflare/ips-v4.tmp /etc/cloudflare/ips-v4; wget "https://www.cloudflare.com/ips-v6" -qO /etc/cloudflare/ips-v6.tmp && mv /etc/cloudflare/ips-v6.tmp /etc/cloudflare/ips-v6