Haproxy: ssl_ckch.c: cli_parse_commit_cert - Segmentation fault/Race condition

[Manawyrm]

Hi,

this is a strange bug and it only happens on the internal control socket, so it shouldn't be security relevant.

Output of haproxy -vv and uname -a

# uname -a
Linux haproxy 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1+deb10u1 (2020-04-27) x86_64 GNU/Linux
# haproxy -vv
HA-Proxy version 2.2-dev10 2020/06/19 - https://haproxy.org/
Status: development branch - not safe for use in production.
Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open
Running on: Linux 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1+deb10u1 (2020-04-27) x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = native
  CC      = gcc
  CFLAGS  = -m64 -march=x86-64 -O2 -march=native -g -Wall -Wextra -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-stringop-overflow -Wno-cast-function-type -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
  OPTIONS = USE_PCRE=1 USE_OPENSSL=1 USE_SYSTEMD=1

Feature list : +EPOLL -KQUEUE +NETFILTER +PCRE -PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H +GETADDRINFO +OPENSSL -LUA +FUTEX +ACCEPT4 -ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=1).
Built with OpenSSL version : OpenSSL 1.1.1d  10 Sep 2019
Running on OpenSSL version : OpenSSL 1.1.1d  10 Sep 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with network namespace support.
Built without compression support (neither USE_ZLIB nor USE_SLZ are set).
Compression algorithms supported : identity("identity")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 8.3.0

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
            fcgi : mode=HTTP       side=BE        mux=FCGI
       <default> : mode=HTTP       side=FE|BE     mux=H1
              h2 : mode=HTTP       side=FE|BE     mux=H2
       <default> : mode=TCP        side=FE|BE     mux=PASS

Available services : none

Available filters :
    [SPOE] spoe
    [COMP] compression
    [TRACE] trace
    [CACHE] cache
    [FCGI] fcgi-app

What's the configuration?

global
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats socket [::1]:2342 mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

    # Default SSL material locations
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

    ssl-dh-param-file /opt/ssl/dhparams-4096.pem

    # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

    ssl-default-server-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
    ssl-default-server-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

defaults
    log global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5000
    timeout client  50000
    timeout server  50000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

backend web_servers
    balance roundrobin
    server server1 [::1]:8080 check maxconn 500

frontend https-in:
    bind [::]:80 v4v6
    bind [::]:443 v4v6 alpn h2,http/1.1 ssl crt /etc/haproxy/ssl-static/dummy.pem crt /etc/haproxy/ssl-static/api.pem crt /etc/haproxy/ssl/
    default_backend web_servers
    http-request set-header X-Forwarded-Proto https if { ssl_fc }

Steps to reproduce the behavior

1. Requires php-cli, sorry about that :frowning_face:
2. Download https://gist.github.com/Manawyrm/316e4c891f8ca191f64e3ca0593bebca (don't worry, the certificate is self-signed)
3. Run HAProxy
4. Run watch -n 0.1 php segfault.php

This will connect to [::1]:2342 (the control socket) and repeatedly delete and add a certificate (with private key).

Actual behavior

https://gist.github.com/Manawyrm/1d3576fecc1063c954af074b1dc14909
https://gist.github.com/Manawyrm/af3d025b49a1263ee83a0f4909ff7fa1
https://gist.github.com/Manawyrm/afd51a198607365acaef2ff01ec407ac

HAProxy will randomly segfault in the first couple of seconds. I've seen crashes in 2 different locations and have put backtraces for both in the 3 files above.

Expected behavior

No segfault. :smile:

Do you have any idea what may have caused this?

No, sorry.

Do you have an idea how to solve the issue?

Not really, either.

[wlallemand]

Thanks for the reproducer, I'm able to reproduce your fist trace instantly with it.

[wlallemand]

The problem is provoked by the fact that the certificate format in your script is buggy. Indeed, when using the CLI with the payload delimiter "<<", the delimiter for the end of the payload is an empty line... which you have in your certificate, so that won't work!
So what's happening is that we try to check if the private key match the public key, even if the private key wasn't uploaded, which could provoke a crash.
The 1st and 3rd trace are the same, and 2nd is probably a side effect of a process which didn't segfault during the commit.

I'm fixing this.

[wlallemand]

The problem is fixed with the patch above.

Be careful to always remove the empty lines in your scripts. The haproxy output of a command is also finished by an empty line.

[TimWolla]

I've adjusted the labels to add "status: fixed" and "2.1".

[Manawyrm]

I tested it as best as I could and couldn't crash HAProxy anymore.

Thanks for fixing the bug! :+1:

[wlallemand]

I made a mistake, it is 2.2-dev only, 2.1 is not affected at all by this issue.