Krypto-trading-bot: --free-version Deathlock in ws communication

[piezza]

I have a deathlock after running K for about 2-10 minutes. I am not an experienced C++ developer to analyze this in deep but at least I can give you some gdb backtraces:

(gdb) info threads
  Id   Target Id         Frame
* 1    Thread 0x7ffff7fc5780 (LWP 12339) "K" 0x00007ffff76b6dbd in __GI___pthread_mutex_lock (mutex=0xc7b440 <K::kxMutex>)
    at ../nptl/pthread_mutex_lock.c:80
  4    Thread 0x7ffff72e2700 (LWP 12362) "K" __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
  5    Thread 0x7ffff62aa700 (LWP 12363) "K" 0x00007ffff73e0593 in select () at ../sysdeps/unix/syscall-template.S:84

[ctubio]

many thanks for the report, hope i can reproduce or something this time :S will investigate soOn''

[ctubio]

can i ask what exchange/currency pairs/operating system are you using (some $ uname -a maybe? or a link to download your very same distro?)?

thanks''

[piezza]

I am using coinbase and trade BTC/EUR

edit: and I use Ubuntu xenial x64

[ctubio]

please let me know if you still have issues after the next commit, is supposed that now websockets cannot be locked continuosly (still i was not able to reproduce the issue)

[piezza]

The issue is more seldom now, but still there:

(gdb) info threads
  Id   Target Id         Frame
* 1    Thread 0x7fa51e4ee780 (LWP 8046) "K" __lll_lock_wait ()
    at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
  2    Thread 0x7fa51d807700 (LWP 8070) "K" __lll_lock_wait ()
    at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
  3    Thread 0x7fa517fff700 (LWP 8071) "K" 0x00007fa51d905593 in select ()
    at ../sysdeps/unix/syscall-template.S:84

(gdb) bt
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007fa51dbdbdbd in __GI___pthread_mutex_lock (mutex=0xc68440 <K::kxMutex>)
    at ../nptl/pthread_mutex_lock.c:80
#2  0x00000000005764d0 in __gthread_mutex_lock (__mutex=0xc68440 <K::kxMutex>)
    at /usr/include/x86_64-linux-gnu/c++/6/bits/gthr-default.h:748
#3  std::mutex::lock (this=0xc68440 <K::kxMutex>) at /usr/include/c++/6/bits/std_mutex.h:103
#4  K::<lambda(uWS::WebSocket<false>*, char*, size_t, uWS::OpCode)>::operator()(uWS::WebSocket<false> *, char *, size_t, uWS::OpCode) (w=0x247a860, message=<optimized out>, length=<optimized out>, opCode=<optimized out>,
    __closure=<optimized out>) at build/K.msg:7003
#5  0x000000000055f688 in std::function<void (uWS::WebSocket<false>*, char*, unsigned long, uWS::OpCode)>::operator()(uWS::WebSocket<false>*, char*, unsigned long, uWS::OpCode) const (__args#3=<optimized out>,
    __args#2=<optimized out>, __args#1=<optimized out>, __args#0=<optimized out>, this=0x24b7958)
    at /usr/include/c++/6/functional:2127
#6  uWS::WebSocket<false>::handleFragment (
    data=data@entry=0x7fa51e4a2021 "{\"type\":\"job\",\"params\":{\"job_id\":\"911154918465763\",\"blob\":\"0606f694a6d105f21a26300f921400f611638055ad96a31ae4d2671d9c7a6547d95fc41ba61fc800000000b58d0696c593d8388344eb9f5c2f6803961c3c2e6c0ea3e7a01969b"..., length=<optimized out>, length@entry=234, remainingBytes=remainingBytes@entry=0, opCode=1,
    fin=<optimized out>, webSocketState=webSocketState@entry=0x247a8a0)
    at build-x86_64-linux-gnu/local/include/uWS/WebSocket.cpp:316
#7  0x00000000005601e6 in uWS::WebSocketProtocol<false, uWS::WebSocket<false> >::consumeMessage<4u, unsigned short>
    (wState=0x247a8a0, length=<synthetischer Zeiger>, src=<synthetischer Zeiger>, payLength=234)
    at build-x86_64-linux-gnu/local/include/uWS/WebSocketProtocol.h:135
#8  uWS::WebSocketProtocol<false, uWS::WebSocket<false> >::consume (wState=0x247a8a0, length=238,
    src=0x7fa51e4a201d "\201~") at build-x86_64-linux-gnu/local/include/uWS/WebSocketProtocol.h:353
#9  uWS::WebSocket<false>::onData (s=s@entry=0x247a860, data=<optimized out>, length=<optimized out>)
    at build-x86_64-linux-gnu/local/include/uWS/WebSocket.cpp:180
#10 0x000000000054de37 in uS::Socket::sslIoHandler<uWS::WebSocket<false> > (p=0x247a860, status=<optimized out>,
    events=<optimized out>) at build-x86_64-linux-gnu/local/include/uWS/Socket.h:207
#11 0x0000000000546083 in Loop::run (this=0x2430710) at build-x86_64-linux-gnu/local/include/uWS/Epoll.cpp:35
#12 0x00000000004f796d in main (argc=<optimized out>, argv=0x7ffd48d5e178) at src/server/K.cxx:76



md5-9afef4eee6f93f9f585539eac3a19e2e



(gdb) print __mutex.__data.__owner
$1 = 8070



md5-9afef4eee6f93f9f585539eac3a19e2e



(gdb) thread 2
[Switching to thread 2 (Thread 0x7fa51d807700 (LWP 8070))]
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
135     ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: Datei oder Verzeichnis nicht gefunden.
(gdb) bt
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007fa51dbdbe42 in __GI___pthread_mutex_lock (mutex=0x2430000) at ../nptl/pthread_mutex_lock.c:115
#2  0x000000000054d830 in __gthread_mutex_lock (__mutex=0x2430000)
    at /usr/include/x86_64-linux-gnu/c++/6/bits/gthr-default.h:748
#3  __gthread_recursive_mutex_lock (__mutex=0x2430000)
    at /usr/include/x86_64-linux-gnu/c++/6/bits/gthr-default.h:810
#4  std::recursive_mutex::lock (this=0x2430000) at /usr/include/c++/6/mutex:105
#5  std::lock_guard<std::recursive_mutex>::lock_guard (__m=..., this=<synthetischer Zeiger>)
    at /usr/include/c++/6/bits/std_mutex.h:162
#6  uWS::Group<false>::broadcast (this=0x24b7890,
    message=0x7fa510000b90 "{\"type\":\"submit\",\"params\":{\"nonce\":\"45e70eb9\",\"result\":\"da8649dfc799879d0a8b1bc905fafc98986928b0fa35fbcc5e20629011fd6e00\",\"job_id\":\"530295938113704\"}}", length=150,
    opCode=opCode@entry=uWS::TEXT) at build-x86_64-linux-gnu/local/include/uWS/Group.cpp:230
#7  0x0000000000574206 in K::kxx (kxM=0xc68440 <K::kxMutex>) at build/K.msg:6943
#8  K::<lambda()>::operator() (__closure=<optimized out>) at build/K.msg:6965
#9  std::_Bind_simple<K::kA(K::Gw*, uWS::Hub*)::<lambda()>()>::_M_invoke<> (this=<optimized out>)
    at /usr/include/c++/6/functional:1391
#10 std::_Bind_simple<K::kA(K::Gw*, uWS::Hub*)::<lambda()>()>::operator() (this=<optimized out>)
    at /usr/include/c++/6/functional:1380
#11 std::thread::_State_impl<std::_Bind_simple<K::kA(K::Gw*, uWS::Hub*)::<lambda()>()> >::_M_run(void) (
    this=<optimized out>) at /usr/include/c++/6/thread:197
#12 0x000000000084bbff in execute_native_thread_routine ()
#13 0x00007fa51dbd96ba in start_thread (arg=0x7fa51d807700) at pthread_create.c:333
#14 0x00007fa51d90f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109



md5-9afef4eee6f93f9f585539eac3a19e2e



(gdb) print __mutex.__data.__owner
$2 = 8046

[ctubio]

and when this happens the app just stops without doing nothing? or what is the consequence?

[piezza]

Yes, it stops then without doing nothing. On the webinterface I only get this Disconnected html side then. Also it doesn't update the orders (what is the really bad part about that ;-) )

Today I updated to your "Coinbase libquickfix" commit. Until now I had no more coinbase deathlocks I think. Same issue occures on Bitfinex also on my system. So this seems to be a more general problem. When I tested this in a Vmware VM which was times slower as my other system, it ran without errors (I tested for half a day). I know another guy who is evaluating on a slower embedded system who also had no problem with Bitfinex in the last two days so far.

In this shoutbox in the K webinterface one other guy also complained that the bot was very unstable. It sounded somehow as the same problem I have.

By the way, thanks for making this bot public and for this very fast support :-)

[ctubio]

i keep this issue closed cos i cannot really reproduce this no matter how long i run coinbase or bitfinex (cos since is closed maybe others will open new with more info); anyway i will continue cleaning up all possible mutex to get rid of them on next commits kxMutex will be the last i think :( but someday

[piezza]

I have some more informations here: this bug seems only to be there in the free version
As far as I understand the two threads conflicting each other are from submitting the results/receiving the next XMR mining jobs
I didn't check against current version with fixes but this version with above mentioned "Coinbase libquickfix" did not solve the problem. K still crashed after some hours.

[ctubio]

thanks'' you are right; that is the last thread; will try to avoid it too soOn

lets reopen now that you clearly identified the issue and a solution is possible (thanks again'¡)