Haproxy: Slow file uploads when using HTTP/2

[yoshz]

Since we enabled HTTP/2 we noticed a huge speed difference while uploading big files.
A 10MB file takes 12 seconds when HTTP/2 is enabled, and when it is disabled 1.5 seconds.

I have stripped the configuration file to bare basics and tried to tweak all kinds of settings, but without any improvements.

Output of haproxy -vv and uname -a

HA-Proxy version 2.0.6 2019/09/13 - https://haproxy.org/
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits
  OPTIONS = USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1

Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER +PCRE -PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED -REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL -LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=16).
Built with OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTX        side=FE|BE     mux=H2
              h2 : mode=HTTP       side=FE        mux=H2
       <default> : mode=HTX        side=FE|BE     mux=H1
       <default> : mode=TCP|HTTP   side=FE|BE     mux=PASS

Available services : none

Available filters :
    [SPOE] spoe
    [COMP] compression
    [CACHE] cache
    [TRACE] trace

Linux haproxy 3.10.0-957.10.1.el7.x86_64 #1 SMP Mon Mar 18 15:06:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

What's the configuration?

global
  chroot  /var/lib/haproxy
  daemon
  group  haproxy
  pidfile  /var/run/haproxy.pid
  ssl-default-bind-ciphers  AES128+EECDH:AES128+EDH
  ssl-default-bind-options  ssl-min-ver TLSv1.2 no-tls-tickets
  tune.ssl.default-dh-param  2048
  tune.ssl.cachesize 1000000
  user  haproxy

defaults
  timeout client 2m
  timeout connect 5s
  timeout http-request 2m
  timeout server 2m

frontend http
  bind :443 ssl crt /etc/haproxy/certs/server.pem alpn h2,http/1.1
  bind :80
  mode http
  default_backend web

backend web
  mode http
  server web 10.36.1.106:30177

Steps to reproduce the behavior

Using a multipart form that sends a POST request to the backend with 1 file of 10 MB.

Actual behavior

The response is correct but the POST request takes 12 seconds over HTTP/2.

Expected behavior

I expect the request to take 1.5 seconds or less, similar to when HTTP/2 is disabled.

Do you have any idea what may have caused this?

No, we have this issue also in version 1.8, so HTTP/2 in Haproxy has never worked for us.
Installing nginx on the same server with HTTP/2 doesn't have this problem.

Do you have an idea how to solve the issue?

No

[wtarreau]

I suspect you're testing over a moderately high latency network and are
simply experiencing the default stream window size which is 65535 bytes
in flight per stream by default in the spec. You can adjust it by adding
this in the global section:

  tune.h2.initial-window-size 1048576

This will multiply the default window by 16 which should be enough to cover
the 1:8 ratio you're seeing. Please keep in mind that using too large windows
can make some fast streams unfair to each other on the sender's side, which
is why the HTTP standard recommended to set it to 64k by default.

[yoshz]

This setting indeed fixes the upload speed. Thank you a lot!
You are right about the latency, this problem only happened in our overseas datacenter but not our nearby datacenter.
What I am curious about is, what is the difference here with nginx? I use the default configuration in nginx, which also uses the 64k initial window size.

[wtarreau]

Great! Regarding NGINX, it's primarily a web server (I'd say an application
server to be more exact), and I think that it mostly deals with single
streams over a connection so it's very likely that they change the
setting at some point (maybe they increase it once a POST transfer starts).
In our case we're purely a load balancer and cannot afford to make wild
guesses on all the streams that pass through and are spread over multiple
backend servers, as that could result in a big loss of interactivity, for
example if you want to read notifications during an attachment upload in
a webmail.

[yoshz]

Thank you for your reply. This makes it easier for me for to troubleshoot these kind of problems in the future. Hopefully this ticket also helps other people with similar problems.