Conky: Conky cores with a lot of docker runs.

Created on 5 Aug 2018 · 7Comments · Source: brndnmtthws/conky

Hi,
When I run a lot of docker runs (a bunch of batch tests, kicked off from the CLI, just docker run ..., an output from tests to stdout, however, a quite a few of runs in a while), conky cores from time to time.

I've managed to catch some info as follows:

conky cored:

$ gdb /usr/bin/conky core
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/conky...(no debugging symbols found)...done.
[New LWP 8850]
[New LWP 8841]
[New LWP 8844]
[New LWP 8845]
[New LWP 8846]
[New LWP 8847]
[New LWP 8848]
[New LWP 8849]
[New LWP 8843]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `conky -d -p 5 -c .../.conkyrc'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f0579a6d84d in ?? () from /usr/lib/x86_64-linux-gnu/liblua5.1.so.0
[Current thread is 1 (Thread 0x7f055e7fc700 (LWP 8850))]
(gdb) bt
#0  0x00007f0579a6d84d in ?? () from /usr/lib/x86_64-linux-gnu/liblua5.1.so.0
#1  0x00007f0579a6e1f2 in ?? () from /usr/lib/x86_64-linux-gnu/liblua5.1.so.0
#2  0x00007f0579a6e73d in ?? () from /usr/lib/x86_64-linux-gnu/liblua5.1.so.0
#3  0x00007f0579a6ba5e in ?? () from /usr/lib/x86_64-linux-gnu/liblua5.1.so.0
#4  0x00007f0579a6c8eb in ?? () from /usr/lib/x86_64-linux-gnu/liblua5.1.so.0
#5  0x00007f0579a683a8 in lua_pcall () from /usr/lib/x86_64-linux-gnu/liblua5.1.so.0
#6  0x0000000000462042 in lua::state::call(int, int, int) ()
#7  0x0000000000466436 in conky::cleanup_config_settings(lua::state&) ()
#8  0x0000000000437887 in clean_up_without_threads(void*, void*) ()
#9  0x000000000045313f in get_net_stat(char const*, void*, void*) ()
#10 0x000000000046d6de in update_net_stats() ()
#11 0x00000000004687a3 in conky::priv::callback_base::start_routine() ()
#12 0x00007f057814bc80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007f057ae536ba in start_thread (arg=0x7f055e7fc700) at pthread_create.c:333
#14 0x00007f05778b141d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb) q
$

conky failed (no core file):

$ conky: too many interfaces used (limit is 64)
conky: /build/conky-bbQ61s/conky-1.10.1/src/luamm.hh:322: lua::stack_sentry::stack_sentry(lua::state&, int): Assertion `n >= 0' failed.
conky: too many interfaces used (limit is 64)
terminate called after throwing an instance of 'std::system_error'
    what():  Resource deadlock avoided
$

System Info:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.5 LTS
Release:        16.04
Codename:       xenial
$
$ uname -a
Linux hawran-lin 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:53:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$

$ dpkg -l '*conky*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                                       Version                            Architecture                       Description
+++-==========================================================-==================================-==================================-==========================================================================================================================
un  conky                                                      <none>                             <none>                             (no description available)
ii  conky-all                                                  1.10.1-3                           amd64                              highly configurable system monitor (all features enabled)
un  conky-cli                                                  <none>                             <none>                             (no description available)
un  conky-std                                                  <none>                             <none>                             (no description available)
$

What's the problem?

Regards,
haw

Source

hawran

❤1

All 7 comments

Try again with (gdb) bt full as it should be more helpful.

lasers on 5 Aug 2018

❤1

Finally I've managed to reproduce the issue (really a bunch of runs):
```(gdb) bt full

0 0x00007fde3d08b428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54

    resultvar = 0
    pid = 24613
    selftid = 24618

1 0x00007fde3d08d02a in __GI_abort () at abort.c:89

    save_stage = 2
    act = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, sa_mask = {__val = {140592484242153, 0, 140592483724287, 140592487183680, 1, 140592487183811, 140728283854367, 140592331617912, 140592484242141, 1, 26994800, 140592197674288, 140592487188336, 140592331618048, 
          140592483731483, 140592197677800}}, sa_flags = 1033681024, sa_restorer = 0x7fde2c0032e8}
    sigs = {__val = {32, 0 <repeats 15 times>}}

2 0x00007fde3d9ce84d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

No symbol table info available.

3 0x00007fde3d9cc6b6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

No symbol table info available.

4 0x00007fde3d9cc701 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

No symbol table info available.

5 0x00007fde3d9f7d38 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

No symbol table info available.

6 0x00007fde406ff6ba in start_thread (arg=0x7fde33fbf700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7fde33fbf700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140592331618048, 4262622693945145704, 0, 140728283854367, 140592331618752, 0, -4279678352096331416, -4279721336489346712}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

7 0x00007fde3d15d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

No locals.
```
I hope it helps.

hawran on 6 Aug 2018

❤1

Seems like upgrading and patching lua may solve this issue - https://github.com/brndnmtthws/conky/issues/609#issuecomment-412869982

su8 on 31 Aug 2018

❤1

Hi, today I've experienced the issue again, even though I'd upgraded my linux some time ago:

gdb:

(gdb) bt full
#0  0x00007fadb368fc29 in  () at /usr/lib/x86_64-linux-gnu/liblua5.1.so.0
#1  0x00007fadb3683dc7 in lua_rawgeti () at /usr/lib/x86_64-linux-gnu/liblua5.1.so.0
#2  0x00007fadb3694971 in luaL_ref () at /usr/lib/x86_64-linux-gnu/liblua5.1.so.0
#3  0x000055873c8aa24a in lua::exception::exception(lua::state*) ()
#4  0x000055873c8aa497 in lua::state::call(int, int, int) ()
#5  0x000055873c8ae8ad in conky::cleanup_config_settings(lua::state&) ()
#6  0x000055873c87da92 in clean_up_without_threads(void*, void*) ()
#7  0x000055873c89a275 in get_net_stat(char const*, void*, void*) ()
#8  0x000055873c8b7130 in update_net_stats() ()
#9  0x000055873c8b0ef3 in conky::priv::callback_base::start_routine() ()
#10 0x00007fadb18f757f in  () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#11 0x00007fadb4a6e6db in start_thread (arg=0x7fada18a0700) at pthread_create.c:463
        pd = 0x7fada18a0700
        now = <optimized out>
        unwind_buf = 
              {cancel_jmp_buf = {{jmp_buf = {140383716247296, 4100317550910363677, 140383716244864, 0, 94039337790176, 140725203813232, -4054363342266968035, -4054321191672096739}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#12 0x00007fadb0fb488f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

SysInfo:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.1 LTS
Release:        18.04
Codename:       bionic

$ uname -a
Linux hawran-lin 4.15.0-45-generic #48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

$ dpkg -l '*conky*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                       Version            Architecture       Description
+++-==========================-==================-==================-=========================================================
un  conky                      <none>             <none>             (no description available)
ii  conky-all                  1.10.8-1           amd64              highly configurable system monitor (all features enabled)
un  conky-cli                  <none>             <none>             (no description available)
un  conky-std                  <none>             <none>             (no description available)

$ dpkg -l '*lua*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                       Version            Architecture       Description
+++-==========================-==================-==================-=========================================================
un  dnsmasq-base-lua           <none>             <none>             (no description available)
ii  liblua5.1-0:amd64          5.1.5-8.1build2    amd64              Shared library for the Lua interpreter version 5.1
ii  liblua5.2-0:amd64          5.2.4-1.1build1    amd64              Shared library for the Lua interpreter version 5.2
ii  liblua5.3-0:amd64          5.3.3-1            amd64              Shared library for the Lua interpreter version 5.3
ii  libluajit-5.1-2:amd64      2.1.0~beta3+dfsg-5 amd64              Just in time compiler for Lua - library version
ii  libluajit-5.1-common       2.1.0~beta3+dfsg-5 all                Just in time compiler for Lua - common files
ii  libtexlua52:amd64          2017.20170613.4457 amd64              TeX Live: Lua 5.2, modified for use with LuaTeX
ii  libtexluajit2:amd64        2017.20170613.4457 amd64              TeX Live: LuaJIT, modified for use with LuaJITTeX
un  luasseq                    <none>             <none>             (no description available)
un  luatex                     <none>             <none>             (no description available)
un  texlive-luatex             <none>             <none>             (no description available)
un  vim-lua                    <none>             <none>             (no description available)

hawran on 5 Feb 2019

For what it's worth, I experienced this same problem (same error message, same coincident docker usage), and upgrading from 1.10.6 to 1.11.2 seemed to make the problem go away.

In case it might be helpful, here are the exception messages I saw:

conky: too many interfaces used (limit is 64)
terminate called after throwing an instance of 'lua::exception'
  what():  attempt to concatenate a nil value
Aborted

conky: too many interfaces used (limit is 64)
Segmentation fault

conky: too many interfaces used (limit is 64)
terminate called after throwing an instance of 'std::system_error'
  what():  Resource deadlock avoided
Aborted

I suggest closing the issue.

Edit: I see now that Lua version matters. My package manager (dpkg) says I've got shared libraries installed for versions 5.1.5, 5.2.4, and 5.3.3.

feuGeneA on 2 Apr 2019

Ah, in that case you probably need to increase the limit on interfaces. It can be modified at build time with cmake -DMAX_NET_INTERFACES=256 .., for example.

I will also make the limit a bit higher, because 64 is probably no longer a sufficient default.