Icinga2: segfault with sha1_block_data_order_avx of libcrypto

[lazyfrosch]

Icinga 2 seems to have a segfault issues with libcrypto, so far I only noticed it with one user on RHEL 7.5 (and he says even before with 7.4)

So far we (@gunnarbeutner and me) think it is a race condition with ApiUser and password_hash generation.

• User has multiple ApiUser which all still had a simple password
• Icinga segfaults during config object activiation
• When debugging through breakpoints on icinga::CreateHashedPasswordString it works fine (slowing down execution)
• When switching all ApiUser to password_hash

The segfaults seem to be happen during sha1_block_data_order_avx in libcrypto

Your Environment

• Version used (icinga2 --version): 2.8.4
• Operating System and version: RHEL 7.5
• OpenSSL version: 1.0.2k-12.el7.x86_64
• Config validation (icinga2 daemon -C): see below

[lazyfrosch]

icinga2 daemon -C

# icinga2 daemon -C
information/cli: Icinga application loader (version: r2.8.4-1)
information/cli: Loading configuration file(s).
information/ConfigItem: Committing config item(s).
warning/Zone: The Zone object 'checker' has more than two endpoints. Due to a known issue this type of configuration is strongly discouraged and may cause Icinga to use excessive amounts of CPU time.
warning/ApiListener: Attribute 'key_path' for object 'api' of type 'ApiListener' is deprecated and should not be used.
warning/ApiListener: Attribute 'ca_path' for object 'api' of type 'ApiListener' is deprecated and should not be used.
warning/ApiListener: Attribute 'cert_path' for object 'api' of type 'ApiListener' is deprecated and should not be used.
warning/ApiListener: Please read the upgrading documentation for v2.8: https://www.icinga.com/docs/icinga2/latest/doc/16-upgrading-icinga-2/
information/ApiListener: My API identity: xxx
information/ConfigItem: Instantiated 1 ApiListener.
information/ConfigItem: Instantiated 5 Zones.
information/ConfigItem: Instantiated 9 Endpoints.
information/ConfigItem: Instantiated 1 FileLogger.
information/ConfigItem: Instantiated 9 ApiUsers.
information/ConfigItem: Instantiated 3 NotificationCommands.
information/ConfigItem: Instantiated 269 CheckCommands.
information/ConfigItem: Instantiated 1 IcingaApplication.
information/ConfigItem: Instantiated 5256 Hosts.
information/ConfigItem: Instantiated 340 HostGroups.
information/ConfigItem: Instantiated 597 Downtimes.
information/ConfigItem: Instantiated 12383 Services.
information/ConfigItem: Instantiated 1 NotificationComponent.
information/ConfigItem: Instantiated 1 CheckerComponent.
information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
information/cli: Finished validating the configuration file(s).

gdb

[root@icinga2 icinga2]# gdb --args /usr/lib64/icinga2/sbin/icinga2 daemon
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-110.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/lib64/icinga2/sbin/icinga2...Reading symbols from /usr/lib/debug/usr/lib64/icinga2/sbin/icinga2.debug...done.
done.
(gdb) r
Starting program: /usr/lib64/icinga2/sbin/icinga2 daemon
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7ffff0f3d700 (LWP 13612)]
[New Thread 0x7fffebfff700 (LWP 13613)]
[New Thread 0x7fffeb7fe700 (LWP 13614)]
[New Thread 0x7fffeaffd700 (LWP 13615)]
[New Thread 0x7fffea7fc700 (LWP 13616)]
[New Thread 0x7fffe9ffb700 (LWP 13617)]
Detaching after fork from child process 13618.
Detaching after fork from child process 13619.
Detaching after fork from child process 13620.
Detaching after fork from child process 13621.
Detaching after fork from child process 13622.
Detaching after fork from child process 13623.
Detaching after fork from child process 13625.
Detaching after fork from child process 13627.
[Thread 0x7fffe9ffb700 (LWP 13617) exited]
[Thread 0x7fffea7fc700 (LWP 13616) exited]
[Thread 0x7fffeaffd700 (LWP 13615) exited]
[Thread 0x7fffeb7fe700 (LWP 13614) exited]
[Thread 0x7fffebfff700 (LWP 13613) exited]
[Thread 0x7ffff0f3d700 (LWP 13612) exited]
process 13608 is executing new program: /usr/lib64/icinga2/sbin/icinga2
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7ffff7fd7700 (LWP 13629)]
[New Thread 0x7ffff7f96700 (LWP 13630)]
[New Thread 0x7ffff7f55700 (LWP 13631)]
[New Thread 0x7ffff7f14700 (LWP 13632)]
[New Thread 0x7ffff7ed3700 (LWP 13633)]
[New Thread 0x7ffff7e92700 (LWP 13634)]
Detaching after fork from child process 13635.
Detaching after fork from child process 13636.
Detaching after fork from child process 13637.
Detaching after fork from child process 13638.
Detaching after fork from child process 13639.
Detaching after fork from child process 13640.
Detaching after fork from child process 13642.
Detaching after fork from child process 13644.
Detaching after fork from child process 13646.
[2018-05-02 13:57:48 +0200] information/cli: Icinga application loader (version: r2.8.4-1)
[2018-05-02 13:57:48 +0200] information/cli: Loading configuration file(s).
[2018-05-02 13:57:49 +0200] information/ConfigItem: Committing config item(s).
[New Thread 0x7ffff7e51700 (LWP 13648)]
[New Thread 0x7fffdbfff700 (LWP 13649)]
[New Thread 0x7fffdbfbe700 (LWP 13650)]
[New Thread 0x7fffdbf7d700 (LWP 13651)]
[New Thread 0x7fffdbf3c700 (LWP 13652)]
[New Thread 0x7fffdbefb700 (LWP 13653)]
[New Thread 0x7fffdbeba700 (LWP 13654)]
[New Thread 0x7fffdbe79700 (LWP 13655)]
[2018-05-02 13:57:49 +0200] warning/Zone: The Zone object 'checker' has more than two endpoints. Due to a known issue this type of configuration is strongly discouraged and may cause Icinga to use excessive amounts of CPU time.
[2018-05-02 13:57:49 +0200] warning/ApiListener: Attribute 'key_path' for object 'api' of type 'ApiListener' is deprecated and should not be used.
[2018-05-02 13:57:49 +0200] warning/ApiListener: Attribute 'ca_path' for object 'api' of type 'ApiListener' is deprecated and should not be used.
[2018-05-02 13:57:49 +0200] warning/ApiListener: Attribute 'cert_path' for object 'api' of type 'ApiListener' is deprecated and should not be used.
[2018-05-02 13:57:49 +0200] warning/ApiListener: Please read the upgrading documentation for v2.8: https://www.icinga.com/docs/icinga2/latest/doc/16-upgrading-icinga-2/
[2018-05-02 13:57:49 +0200] information/ApiListener: My API identity: xxx

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffdbe79700 (LWP 13655)]
sha1_block_data_order_avx () at sha1-x86_64.s:3458
3458            vmovdqu 32(%r9),%xmm2
(gdb) bt
#0  sha1_block_data_order_avx () at sha1-x86_64.s:3458
#1  0xf52cc4018a669336 in ?? ()
#2  0xca66bed55063f6b4 in ?? ()
#3  0x366569fe767730c2 in ?? ()
#4  0xca8bb43231b9dcfd in ?? ()
#5  0xe1223a7ed5245833 in ?? ()
#6  0x8a62f1e51e551325 in ?? ()
#7  0xf4c2ca06a9ff1378 in ?? ()
#8  0xb560042b643e7ea6 in ?? ()
#9  0x0000000000000000 in ?? ()
(gdb) i thr
  Id   Target Id         Frame
* 21   Thread 0x7fffdbe79700 (LWP 13655) "icinga2" sha1_block_data_order_avx () at sha1-x86_64.s:3458
  20   Thread 0x7fffdbeba700 (LWP 13654) "icinga2" sha256_block_data_order_avx () at sha256-x86_64.s:3659
  19   Thread 0x7fffdbefb700 (LWP 13653) "icinga2" getrn (lh=lh@entry=0x7fffb8008460, data=data@entry=0x7ffff639cf70 <X509V3_str_functs+656>, rhash=rhash@entry=0x7fffdbef8d60) at lhash.c:401
  18   Thread 0x7fffdbf3c700 (LWP 13652) "icinga2" _int_free (av=0x7fffc4000020, p=0x7fffc4002820, have_lock=0) at malloc.c:3906
  17   Thread 0x7fffdbf7d700 (LWP 13651) "icinga2" sha1_block_data_order_avx () at sha1-x86_64.s:3458
  16   Thread 0x7fffdbfbe700 (LWP 13650) "icinga2" sha1_block_data_order_avx () at sha1-x86_64.s:3456
  15   Thread 0x7fffdbfff700 (LWP 13649) "icinga2" sha1_block_data_order_avx () at sha1-x86_64.s:3457
  14   Thread 0x7ffff7e51700 (LWP 13648) "icinga2" sha1_block_data_order_avx () at sha1-x86_64.s:3459
  13   Thread 0x7ffff7e92700 (LWP 13634) "icinga2" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
  12   Thread 0x7ffff7ed3700 (LWP 13633) "icinga2" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
  11   Thread 0x7ffff7f14700 (LWP 13632) "icinga2" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  10   Thread 0x7ffff7f55700 (LWP 13631) "icinga2" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  9    Thread 0x7ffff7f96700 (LWP 13630) "icinga2" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  8    Thread 0x7ffff7fd7700 (LWP 13629) "icinga2" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  1    Thread 0x7ffff7fdb880 (LWP 13608) "icinga2" 0x00007ffff71af14c in icinga::WorkQueue::Enqueue(boost::function<void ()>&&, icinga::WorkQueuePriority, bool) (this=this@entry=0x7fffffffa740, function=...,
    priority=priority@entry=icinga::PriorityNormal, allowInterleaved=allowInterleaved@entry=false) at /usr/src/debug/icinga2-2.8.4/lib/base/workqueue.cpp:94
(gdb)




Breakpoint 1, sha1_block_data_order_avx () at sha1-x86_64.s:2589
2589            movq    %rsp,%rax
(gdb) bt
#0  sha1_block_data_order_avx () at sha1-x86_64.s:2589
#1  0x00007ffff5fbb0af in SHA1_Update (c=0x7fffd0002c20, data_=<optimized out>, len=<optimized out>) at ../md32_common.h:329
#2  0x00007ffff606ad08 in ssleay_rand_add (buf=0x7ffff7e4fca0, num=48, add=48) at md_rand.c:288
#3  0x00007ffff606c4a9 in RAND_poll () at rand_unix.c:405
#4  0x00007ffff606b07d in ssleay_rand_bytes (buf=0x7fffd0002630 "\300\020", num=8, pseudo=0, lock=1) at md_rand.c:398
#5  0x00007ffff71ced50 in icinga::RandomString (length=length@entry=8) at /usr/src/debug/icinga2-2.8.4/lib/base/tlsutil
icinga::RandomString (length=length@entry=8) at /usr/src/debug/icinga2-2.8.4/lib/base/tlsutility.cpp:721
#6  0x00007ffff68c4792 in icinga::ApiUser::OnConfigLoaded() (this=0x7fffd00029a0) at /usr/src/debug/icinga2-2.8.4/lib/remote/apiuser.cpp:35
#7  0x00007ffff6e77676 in icinga::ConfigItem::Commit (this=<optimized out>, discard=<optimized out>) at /usr/src/debug/icinga2-2.8.4/lib/config/configitem.cpp:274
#8  0x00007ffff6e78806 in operator() (__closure=<optimized out>) at /usr/src/debug/icinga2-2.8.4/lib/config/configitem.cpp:434
#9  boost::detail::function::void_function_obj_invoker0<icinga::ConfigItem::CommitNewItems(boost::intrusive_ptr<icinga::ActivationContext> const&, icinga::WorkQueue&, std::vector<boost::intrusive_ptr<icinga::ConfigItem>, std::allocator<boost::intrusive_ptr<icinga::ConfigItem> > >&)::{lambda()#1}, void>::invoke(boost::detail::function::function_buffer&) [clone .65497] (function_obj_ptr=...)
    at /usr/include/boost/function/function_template.hpp:153
#10 0x00007ffff716be8d in operator() (this=0x7ffff7e50b10) at /usr/include/boost/function/function_template.hpp:767
#11 icinga::WorkQueue::WorkerThreadProc (this=0x7fffffffa740) at /usr/src/debug/icinga2-2.8.4/lib/base/workqueue.cpp:263
#12 0x00007ffff7bd127a in boost::(anonymous namespace)::thread_proxy (param=<optimized out>) at libs/thread/src/pthread/thread.cpp:165
#13 0x00007ffff489bdd5 in start_thread (arg=0x7ffff7e51700) at pthread_create.c:308
#14 0x00007ffff45c5b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

(gdb) i thr
  Id   Target Id         Frame
  21   Thread 0x7fffdbe79700 (LWP 13785) "icinga2" sha1_block_data_order_avx () at sha1-x86_64.s:2589
  20   Thread 0x7fffdbeba700 (LWP 13784) "icinga2" elf_machine_fixup_plt (map=<optimized out>, t=<optimized out>, reloc=<optimized out>, value=140737320263472, reloc_addr=0x7ffff66108a0)
    at ../sysdeps/x86_64/dl-machine.h:232
  19   Thread 0x7fffdbefb700 (LWP 13783) "icinga2" 0x00007ffff48a280d in close () at ../sysdeps/unix/syscall-template.S:81
  18   Thread 0x7fffdbf3c700 (LWP 13782) "icinga2" sha1_block_data_order_avx () at sha1-x86_64.s:2589
  17   Thread 0x7fffdbf7d700 (LWP 13781) "icinga2" __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
  16   Thread 0x7fffdbfbe700 (LWP 13780) "icinga2" 0x00007ffff48a27ad in read () at ../sysdeps/unix/syscall-template.S:81
  15   Thread 0x7fffdbfff700 (LWP 13779) "icinga2" sha1_block_data_order_avx () at sha1-x86_64.s:2589
* 14   Thread 0x7ffff7e51700 (LWP 13778) "icinga2" sha1_block_data_order_avx () at sha1-x86_64.s:2589
  13   Thread 0x7ffff7e92700 (LWP 13765) "icinga2" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
  12   Thread 0x7ffff7ed3700 (LWP 13764) "icinga2" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
  11   Thread 0x7ffff7f14700 (LWP 13763) "icinga2" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  10   Thread 0x7ffff7f55700 (LWP 13762) "icinga2" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  9    Thread 0x7ffff7f96700 (LWP 13761) "icinga2" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  8    Thread 0x7ffff7fd7700 (LWP 13760) "icinga2" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  1    Thread 0x7ffff7fdb880 (LWP 13742) "icinga2" 0x00007ffff713bc78 in operator+= (__n=725, this=<synthetic pointer>) at /usr/include/c++/4.8.2/bits/stl_deque.h:200

Seems like the segfaults happens during icinga::RandomString and ssleay_rand_bytes

[lazyfrosch]

So far I was not able to re-create the issue with a bunch of ApiUser, on the user system this happens every time during daemon startup.

I tested on CentOS 7 and RHEL 7 within Docker, but only the demo config + 100 ApiUser

[lazyfrosch]

This looks to me like an internal issue in OpenSSL, like a memory corruption between sha1_block_data_order_avx and sha256_block_data_order_avx

(gdb) i thr
  Id   Target Id         Frame
* 21   Thread 0x7fffdbe79700 (LWP 18645) "icinga2" sha1_block_data_order_avx () at sha1-x86_64.s:3457
  20   Thread 0x7fffdbeba700 (LWP 18644) "icinga2" sha256_block_data_order_avx () at sha256-x86_64.s:3341
  19   Thread 0x7fffdbefb700 (LWP 18643) "icinga2" 0x00007ffff454c822 in __GI___libc_malloc (bytes=24) at malloc.c:2897
  18   Thread 0x7fffdbf3c700 (LWP 18642) "icinga2" sha256_block_data_order_avx () at sha256-x86_64.s:3405
  17   Thread 0x7fffdbf7d700 (LWP 18641) "icinga2" sha1_block_data_order_avx () at sha1-x86_64.s:3458
  16   Thread 0x7fffdbfbe700 (LWP 18640) "icinga2" sha1_block_data_order_avx () at sha1-x86_64.s:3458
  15   Thread 0x7fffdbfff700 (LWP 18639) "icinga2" OPENSSL_cleanse () at x86_64cpuid.s:193
  14   Thread 0x7ffff7e51700 (LWP 18638) "icinga2" 0x00007ffff45496f6 in _int_malloc (av=av@entry=0x7fffd0000020, bytes=bytes@entry=120) at malloc.c:3382
  13   Thread 0x7ffff7e92700 (LWP 18624) "icinga2" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
  12   Thread 0x7ffff7ed3700 (LWP 18623) "icinga2" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
  11   Thread 0x7ffff7f14700 (LWP 18622) "icinga2" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  10   Thread 0x7ffff7f55700 (LWP 18621) "icinga2" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  9    Thread 0x7ffff7f96700 (LWP 18620) "icinga2" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  8    Thread 0x7ffff7fd7700 (LWP 18619) "icinga2" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  1    Thread 0x7ffff7fdb880 (LWP 18598) "icinga2" boost::detail::get_once_per_thread_epoch () at libs/thread/src/pthread/once.cpp:62


(gdb) thr 15
[Switching to thread 15 (Thread 0x7fffdbfff700 (LWP 18639))]
#0  OPENSSL_cleanse () at x86_64cpuid.s:193
193             leaq    -8(%rsi),%rsi
(gdb) bt
#0  OPENSSL_cleanse () at x86_64cpuid.s:193
#1  0x00007ffff606f040 in EVP_MD_CTX_cleanup (ctx=ctx@entry=0x7fffdbffddd8) at digest.c:423
#2  0x00007ffff5fda146 in HMAC_CTX_cleanup (ctx=0x7fffdbffdd70) at hmac.c:222
#3  0x00007ffff607c919 in PKCS5_PBKDF2_HMAC (pass=<optimized out>, passlen=<optimized out>, salt=0x7fffc8001f08 "a8ac6bca0700a4c2", saltlen=16, iter=1000, digest=<optimized out>, keylen=32,
    out=0x7fffdbffdfa0 "\376EQ\213^\252a\302P\022\067\376'\234A\253\061\355\006\276:T\t}Xq\a6\270\231\207\030") at p5_crpt2.c:143
#4  0x00007ffff7157ea9 in icinga::PBKDF2_SHA256(icinga::String const&, icinga::String const&, int) (password=..., salt=..., iterations=iterations@entry=1000)
    at /usr/src/debug/icinga2-2.8.4/lib/base/tlsutility.cpp:631
#5  0x00007ffff7157fb8 in icinga::CreateHashedPasswordString(icinga::String const&, icinga::String const&, int) (password=..., salt=..., algorithm=algorithm@entry=5)
    at /usr/src/debug/icinga2-2.8.4/lib/base/tlsutility.cpp:791
#6  0x00007ffff68c47b0 in icinga::ApiUser::OnConfigLoaded() (this=0x7fffc8002ad0) at /usr/src/debug/icinga2-2.8.4/lib/remote/apiuser.cpp:35
#7  0x00007ffff6e77676 in icinga::ConfigItem::Commit (this=<optimized out>, discard=<optimized out>) at /usr/src/debug/icinga2-2.8.4/lib/config/configitem.cpp:274
#8  0x00007ffff6e78806 in operator() (__closure=<optimized out>) at /usr/src/debug/icinga2-2.8.4/lib/config/configitem.cpp:434
#9  boost::detail::function::void_function_obj_invoker0<icinga::ConfigItem::CommitNewItems(boost::intrusive_ptr<icinga::ActivationContext> const&, icinga::WorkQueue&, std::vector<boost::intrusive_ptr<icinga::ConfigItem>, std::allocator<boost::intrusive_ptr<icinga::ConfigItem> > >&)::{lambda()#1}, void>::invoke(boost::detail::function::function_buffer&) [clone .65497] (function_obj_ptr=...)
    at /usr/include/boost/function/function_template.hpp:153
#10 0x00007ffff716be8d in operator() (this=0x7fffdbffeb10) at /usr/include/boost/function/function_template.hpp:767
#11 icinga::WorkQueue::WorkerThreadProc (this=0x7fffffffa740) at /usr/src/debug/icinga2-2.8.4/lib/base/workqueue.cpp:263
#12 0x00007ffff7bd127a in boost::(anonymous namespace)::thread_proxy (param=<optimized out>) at libs/thread/src/pthread/thread.cpp:165
#13 0x00007ffff489bdd5 in start_thread (arg=0x7fffdbfff700) at pthread_create.c:308
#14 0x00007ffff45c5b3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

[Crunsher]

Sounds like a race condition, googling found someone with a similar problem and a possible solution

This problem seems to have been fixed in openssl 1.1.0

[dnsmichi]

@Crunsher please start with this issue in CW21.

[dnsmichi]

Doesn't happen on macOS with OpenSSL 1.0.2o btw.

Simple test config:

```
const users = 10000;

for (u in range(users)) {
object ApiUser u use (u) {
password = "threadsafetywhoneedsthat?" + u
}
}
``´

[dnsmichi]

The simplest fix which comes to mind - use a local lock in RandomString() in lib/base/tlsutility.cpp. The other thread initialization thing from OpenSSL is far too generic to waste time on it. libssh2 suffers from that as well.

[dnsmichi]

I'm also wondering why the password hash generator was implemented into OnConfigLoaded() inside the config item workqueue which indeed runs async. A different fix would be to move this generation after all the configuration objects have been compiled.

[N-o-X]

I'm currently not able to reproduce this issue. Tried it on the following setups:

• CentOS 7.4 - OpenSSL 1.0.2k-12.el7
• Ubuntu 16.04 - OpenSSL 1.0.2g
• macOS 10.13.4 - OpenSSL 1.0.2o (again)

I used the config @dnsmichi posted and tested it like this:
i=0; while icinga2 daemon -C; do echo $((++i)); done

[Crunsher]

Tested this with a centos 7.5 and openssl-1.0.2k with Michis config. Startup takes noticeably longer but no crash

[Crunsher]

This happens in RandomString(), which we call once for every password (And we should not change that). The password hashing should happen as early as possible and if it wasn't for the crash, it being async would have helped with performance :D
To propose a solution: We lock during the password hashing using the openssl crypto_threads. This way we do not have to worry about thread safety.

Edit:
@dnsmichi

The other thread initialization thing from OpenSSL is far too generic to waste time on it. libssh2 suffers from that as well.

I read that just now, I'm going to take another look whether the openssl libs are the right tool.

[dnsmichi]

References:
http://openssl.6102.n7.nabble.com/Self-initialization-of-locking-threadid-callbacks-and-auto-detection-of-features-td46931.html
https://github.com/openssl/openssl/issues/2165
https://www.openssl.org/blog/blog/2017/02/21/threads/

TL;DR - when using OpenSSL < 1.1.0 you cannot assume that CRYPTO_lock ensures thread-safety for all OpenSSL operations. Application side workarounds always include a local lock like done in the referenced PR, I would also just use that for 1.1.0 linked versions for better safety.