Haproxy: 51degress device detection seems to be broken with htx enabled

[albix]

Output of haproxy -vv and uname -a

HA-Proxy version 2.0.12 2019/12/21 - https://haproxy.org/
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits
  OPTIONS = USE_PCRE_JIT=1 USE_THREAD=1 USE_PTHREAD_PSHARED=1 USE_STATIC_PCRE=1 USE_LINUX_TPROXY=1 USE_VSYSCALL=1 USE_OPENSSL=1 USE_SLZ=1 USE_TFO=1 USE_NS=1 USE_51DEGREES=1 USE_SYSTEMD=1

Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE +PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD +PTHREAD_PSHARED -REGPARM +STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H +VSYSCALL +GETADDRINFO +OPENSSL -LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 -ZLIB +SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS +51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=16).
Built with OpenSSL version : OpenSSL 1.1.1d  10 Sep 2019
Running on OpenSSL version : OpenSSL 1.1.1d  10 Sep 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with 51Degrees Trie support.
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.43 2019-02-23
Running on PCRE version : 8.43 2019-02-23
PCRE library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with the Prometheus exporter as a service

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTX        side=FE|BE     mux=H2
              h2 : mode=HTTP       side=FE        mux=H2
       <default> : mode=HTX        side=FE|BE     mux=H1
       <default> : mode=TCP|HTTP   side=FE|BE     mux=PASS

Available services :
    prometheus-exporter

Available filters :
    [SPOE] spoe
    [COMP] compression
    [CACHE] cache
    [TRACE] trace


[root@lb06 ~]# uname -a
Linux lb06 5.4.11-1.el7.elrepo.x86_64 #1 SMP Sun Jan 12 09:52:48 EST 2020 x86_64 x86_64 x86_64 GNU/Linux

What's the configuration?

global
  chroot /tmp/haproxy
  user haproxy
  group haproxy

  51degrees-data-file /tmp/haproxy/51Degrees-PremiumV3.4.trie
  51degrees-property-name-list IsMobile LayoutEngine ScreenPixelsHeight ScreenPixelsWidth DeviceType HardwareFamily HardwareName PlatformName JavascriptVersion BrowserName BrowserVersion IsCrawler

defaults
  mode http
  #no option http-use-htx


frontend foo
  bind *:10000
  http-request set-header X-51D-DeviceType %[51d.all(DeviceType)]
  http-request set-header X-51D-HardwareFamily %[51d.all(HardwareFamily)]
  http-request set-header X-51D-PlatformName %[51d.all(PlatformName)]
  http-request set-header X-51D-LayoutEngine %[51d.all(LayoutEngine)]
  http-request set-header X-51D-JavascriptVersion %[51d.all(JavascriptVersion)]
  http-request set-header X-51D-BrowserName %[51d.all(BrowserName)]
  http-request set-header X-51D-BrowserVersion %[51d.all(BrowserVersion)]
  http-request set-header X-51D-IsCrawler %[51d.all(IsCrawler)]

  http-request set-header Host httpbin.org
  default_backend bar


backend bar
  # httpbin.org
  server s1 34.193.212.251:80
  server s2 54.172.95.6:80

Steps to reproduce the behavior

1. Run haproxy with 51degrees and htx enabled
2. Run haproxy with 51degrees and htx disabled

Actual behavior

After upgrading to haproxy 2.0.12 from 1.9.10 the 51degrees device detection seems to be broken. Instead of proper values I only get "1" reported (X-51d headers). When I disable htx (no option http-use-htx) the device detection works as expected.

$ curl -H "accept: application/json" http://127.1:10000/anything
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "application/json", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.29.0", 
    "X-51D-Browsername": "1", 
    "X-51D-Browserversion": "1", 
    "X-51D-Devicetype": "1", 
    "X-51D-Hardwarefamily": "1", 
    "X-51D-Iscrawler": "1", 
    "X-51D-Javascriptversion": "1", 
    "X-51D-Layoutengine": "1", 
    "X-51D-Platformname": "1"
  }, 
  "json": null, 
  "method": "GET", 
  "origin": "<deducted>", 
  "url": "https://httpbin.org/anything"
}

Expected behavior

curl -H "accept: application/json" http://127.1:10000/anything
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "application/json", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.29.0", 
    "X-51D-Browsername": "Unknown Crawler", 
    "X-51D-Browserversion": "Unknown", 
    "X-51D-Devicetype": "Desktop", 
    "X-51D-Hardwarefamily": "Emulator", 
    "X-51D-Iscrawler": "True", 
    "X-51D-Javascriptversion": "Unknown", 
    "X-51D-Layoutengine": "Unknown", 
    "X-51D-Platformname": "Unknown"
  }, 
  "json": null, 
  "method": "GET", 
  "origin": "<deducted>", 
  "url": "https://httpbin.org/anything"
}

Do you have any idea what may have caused this?

As mentioned above, there seems to be something wrong between 51degress and htx

Do you have an idea how to solve the issue?

unfortunately not

[fgtham]

I had the same issue and switched from 51d.all to 51d.single as workaround:

http-request set-header X-51D-BrowserName %[req.fhdr(User-Agent),51d.single(BrowserName)]

[Steve51D]

Hi, Just confirming that 51Degrees are looking at this from our end as well. We'll update this issue once we know more.

[capflam]

Type (SMP_T_STR) and flags (SMP_F_CONST) of the sample fetch are set too early in _51d_fetch() function. Every call to smp_prefetch_htx() alters the sample fetch. So, you must set these info at the end or don't call smp_prefetch_htx() in sub-functions. An alternative is to use another sample fetch in _51d_set_device_offsets() and _51d_set_headers(). But honestly there is no reason to not pass the htx message instead of the sample fetch as parameter to these both functions.

[ben51degrees]

Hi @albix,
I have now submitted a patch which fixes this issue. I will update and close this issue once it has been applied to master and backported to 2.0.
And thanks @capflam for the input. You certainly made tracking down the cause much quicker.

[albix]

Great news! Thank you all!

[capflam]

Backported now. Thanks !