Haproxy: External check for a backend affects other service connections.

[aminancelot]

Output of haproxy -vv and uname -a

HA-Proxy version 1.6.3 2015/12/25
Copyright 2000-2015 Willy Tarreau <[email protected]>

Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
  OPTIONS = USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016
Running on OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.41 2017-07-05
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with Lua version : Lua 5.3.1
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Linux REDACTED-02 4.15.0-60-generic #67~16.04.1-Ubuntu SMP Mon Aug 26 08:57:33 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

What's the configuration?

# To check current status of connections, etc.., run the following line.
# echo "show stat" | socat unix-connect:/var/run/haproxy/haproxy.sock stdio | awk -F ',' '{print $1 "|" $2 "|" $18 "|" $37}' | sed 's/^#//' | column -t -s '|'
#

global
    # Do not use chroot. That would break external-check
    log /dev/log    local0
    log /dev/log    local1 notice
    external-check
    user haproxy
    group haproxy
    daemon
    stats socket /var/run/haproxy/haproxy.sock mode 660 level admin
    stats timeout 2m


defaults
    log global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5s
    timeout client  120s
    timeout server  120s
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

#------------------------------------------------

frontend mariadb-single-connection
   mode tcp
   bind 127.0.0.1:3308
   default_backend mariadb-master

backend mariadb-master
    mode tcp
    timeout server 3s
        server bx-sa-preprod-db01 172.17.60.97:3306 check rise 3
        server bx-sb-preprod-db01 172.17.60.98:3306 check rise 3 backup
        server bx-sc-preprod-db01 172.17.60.99:3306 check rise 3 backup

#------------------------------------------------

frontend mariadb
   mode tcp
   bind 127.0.0.1:3306
   default_backend mariadb-masters

backend mariadb-masters
    mode tcp
    balance roundrobin


    # To test the custom script for each server, execute the command in the comment and if the output is not simply 0, check is considered DOWN.
    # You can copy and edit the script to uncomment the "set -x" line, it will give you debug information about what's happening.
    # You may also want to check the first comment of this file.

    # $ /etc/haproxy/check_mysql.sh NOT_USED NOT_USED 172.17.60.97 3306; echo $?
    server master_1 172.17.60.97:3306 check

    # $ /etc/haproxy/check_mysql.sh NOT_USED NOT_USED 172.17.60.98 3306; echo $?
    server master_2 172.17.60.98:3306 check

    # $ /etc/haproxy/check_mysql.sh NOT_USED NOT_USED 172.17.60.99 3306; echo $?
    server master_3 172.17.60.99:3306 check


#------------------------------------------------

frontend memcache
    mode tcp
    bind 127.0.0.1:11211
    timeout client  21600s
    default_backend memcache-masters

backend memcache-masters
    mode tcp
    timeout server  21600s
    server master_1 172.17.60.97:11211 check
    server master_2 172.17.60.99:11211 check backup

#------------------------------------------------

frontend redis
    mode tcp
    bind 127.0.0.1:6381
    default_backend redis-masters

backend redis-masters
    mode tcp
    server master_0 172.17.60.61:6379 check

#------------------------------------------------

Steps to reproduce the behavior

1. Enable external checks for mariadb-masters backend
2. Observe connectivity issues in memcache.

Actual behavior

As it is described above, behaviour works perfectly.
But if I change

backend mariadb-masters
    mode tcp
    balance roundrobin

towards

backend mariadb-masters
    mode tcp
    balance roundrobin

    option external-check
    external-check path "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    external-check command /etc/haproxy/check_mysql.sh

We start to see memcached related errors.
"socket was unexpectedly closed"

Observed patterns are very clear. If external checks are enabled for mariadb-masters backend, issues appear for memcached clients using memcache frontend.

Expected behavior

Enabling external checks on a backend shouldn't affect another service.

Do you have an idea how to solve the issue?

Right now, we disabled external checks.
I tried to see if not using the same server for mariadb and memcache would help, it didn't.
Not sure how to go from here.

[TimWolla]

HA-Proxy version 1.6.3 2015/12/25

Upgrade your HAProxy, please. The 1.6.x branch is already in the “critical fixes only” state and you are not even using the latest version within that branch, but a version with a total of 349 known bugs (three of those having external-checks in the subject line, with possibly more of them affecting external-checks): https://www.haproxy.org/bugs/bugs-1.6.3.html

At the very least update to 1.6.15. The upgrade to 1.8.25 should be fairly smooth as well and will give you 2 additional years of support.

[TimWolla]

This commit looks like it could be a fix for the issues you are seeing: https://git.haproxy.org/?p=haproxy-1.6.git;a=commitdiff;h=ebcbf88

[aminancelot]

Thank you. I will attempt this.

[aminancelot]

@TimWolla This worked for me. Thank you.