Janus-gateway: crash keeps happening since version 0.1.1 and also on latest git checkout

[bhakimi]

let me know if you need anything else, this is the core dump

[New LWP 22784]
[New LWP 34672]
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: generic error
warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
        add-auto-load-safe-path /lib64/libthread_db-1.0.so
line to your configuration file "/root/.gdbinit".
To completely disable this security protection add
        set auto-load safe-path /
line to your configuration file "/root/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
        info "(gdb)Auto-loading safe path"
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: generic error
warning: File "/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Core was generated by `/usr/bin/janus -F/etc/janus -N -L /var/log/janus.log -p /var/run/janus.pid'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fc705ccc2d0 in sha1_update () from /usr/lib64/libsrtp.so.1
[Current thread is 1 (LWP 12108)]

[lminiero]

The gdb trace is incomplete, you should provide (in gist/pastebin, not pasted here) the full stacktrace. Make sure your libsrtp and openssl are up-to-date, libsrtp 1.4.x is known to give issues.

[bhakimi]

we use net-libs/libsrtp-1.5.2-r1:0 , what command do i run to get the complete output

[lminiero]

bt is usually enough
https://sourceware.org/gdb/onlinedocs/gdb/Backtrace.html

[lminiero]

Any info on the backtrace?

[bhakimi]

i reverted back to 0.0.5 because the crashed kept happening every other hour
vs other day, if you like message me on skype and i'll run whatever
commands i can to give you the data, i wasnt able to get what you needed
even if the commands you provided

On Wed, Feb 8, 2017 at 8:50 AM, Lorenzo Miniero [e-mail]
wrote:

Any info on the backtrace?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/meetecho/janus-gateway/issues/751#issuecomment-278386338,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADQXbQ1tU3rkSvmt_N-hP2k_uZLQscjnks5rafJngaJpZM4LmTCi
.

[bhakimi]

i ran gdb /usr/bin/janus core and this is what i got, does this help ?

warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/bin/janus -F/etc/janus.1 -N -L /var/log/janus.1.log -p /var/run/janus.1.pi'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  janus_ice_cb_nice_recv (agent=<optimized out>, stream_id=<optimized out>, component_id=<optimized out>, len=<optimized out>, buf=<optimized out>, ice=<optimized out>) at ice.c:2009
2009    ice.c: No such file or directory.
[Current thread is 1 (Thread 0x7f0cb7fff700 (LWP 12577))]
(gdb)

bt, we use libnice 0.1.13

(gdb) bt
#0  janus_ice_cb_nice_recv (agent=<optimized out>, stream_id=<optimized out>, component_id=<optimized out>, len=<optimized out>, buf=<optimized out>, ice=<optimized out>) at ice.c:2009
#1  0x00007f0f3f37a321 in ?? () from /usr/lib64/libnice.so.10
#2  0x00007f0f3f384d2d in ?? () from /usr/lib64/libnice.so.10
#3  0x00007f0f3d034e90 in ?? () from /usr/lib64/libgio-2.0.so.0
#4  0x00007f0f3ee36c4d in g_main_context_dispatch () from /usr/lib64/libglib-2.0.so.0
#5  0x00007f0f3ee37028 in ?? () from /usr/lib64/libglib-2.0.so.0
#6  0x00007f0f3ee374c2 in g_main_loop_run () from /usr/lib64/libglib-2.0.so.0
#7  0x00007f0f3f811558 in janus_ice_thread (data=0x7f0e20004180) at ice.c:2232
#8  0x00007f0f3ee65d78 in ?? () from /usr/lib64/libglib-2.0.so.0
#9  0x00007f0f3d93061c in start_thread () from /lib64/libpthread.so.0
#10 0x00007f0f3d66f21d in clone () from /lib64/libc.so.6
(gdb)

additionally i see this in dmesg, we use libsrtp version 1.5.2

[37594.428939] iceloop 6079931[21903]: segfault at 0 ip 00007f66472ab2d2 sp 00007f62f27fbbb0 error 4 in libsrtp.so.1[7f66472a5000+17000]
[40227.281339] icesend 5547222[9802]: segfault at 7f8775500cd3 ip 00007f88538f92dd sp 00007f87dcff6ba0 error 6 in libsrtp.so.1[7f88538eb000+17000]
[40227.281352] icesend 5547222[58971]: segfault at 7f8775500cd3 ip 00007f88538f92dd sp 00007f8751ff8ba0 error 6

[bhakimi]

here is another crash that just happened now

warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/bin/janus -F/etc/janus.1 -N -L /var/log/janus.1.log -p /var/run/janus.1.pi'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f34be9ecb80 in srtp_sha1_update () from /usr/lib64/libsrtp2.so.1
[Current thread is 1 (Thread 0x7f3462ffd700 (LWP 24603))]
(gdb) bt
#0  0x00007f34be9ecb80 in srtp_sha1_update () from /usr/lib64/libsrtp2.so.1
#1  0x00007f34be9ec3dd in ?? () from /usr/lib64/libsrtp2.so.1
#2  0x00007f34be9e7752 in srtp_protect_rtcp () from /usr/lib64/libsrtp2.so.1
#3  0x00007f34c043fffc in janus_ice_send_thread (data=0x7f344c05fe90) at ice.c:3499
#4  0x00007f34bfa82fa8 in ?? () from /usr/lib64/libglib-2.0.so.0
#5  0x00007f34be557604 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f34be290bbd in clone () from /lib64/libc.so.6
(gdb)

[bhakimi]

should i provide more of these or is this enough ? i get about 2 crashes a day

[bhakimi]

i like to pay a one time support fee to fix this, please contact me directly if this is possible

[cmacq2]

If you run gdb /usr/bin/janus gdb will just load the binary. It won't actually begin executing Janus gateway, to start that you have to enter start at which point Jaus will run into the first (default) break point set by gdb just before main(). Then you can type continue to continue running Janus from there.

So if you just want to run Janus in gdb and don't have a particular idea of what you are looking for you'd do something like this:

gdb /usr/bin/janus # add arguments to janus as needed
start
continue

If you do have an idea what you are looking for you would set up breakpoints in advance before typing start to start Janus.

[bhakimi]

this is a production system, i cant run janus in gdb and mess with it, we have a 24/7 operation, do the results above not help?

[jackmjones25]

I think I have the same problem, guys....

My janus gw crashes everyday at leats once on peak time (40+ users).

No error on console (debug_level: 5)
OS: fedora 25
Janus v0.2.2
libnice-0.1.13-7.fc25.x86_64

0 0x0000000000430a3e in janus_ice_cb_nice_recv (agent=, stream_id=, component_id=, len=, buf=, ice=) at ice.c:2009

BT: http://pastebin.com/bc0MPj0C

Any ideas ? How else can I help, @lminiero ?
(should I open a new issue?)

[lminiero]

Are you on the SIP plugin too? If so, 40 users cannot really be considered a peak, of course depending on the resources the machine provides (hardware, network). Try installing those separate debuginfos it suggests via dnf. Besides, can you check via gdb what the value of cur_seq is?

[jackmjones25]

Hi @lminiero,

Yes, I'm using the sip plugin. The server never goes high on resources, but it always crashes during his peak times.

(gdb) print cur_seq
$1 = (seq_info_t *) 0x0

after the debuginfos + bt full:
http://pastebin.com/rX9ext7Y

(some warnings about the debug/lib version, but they seem to be right)

[lminiero]

Apparently in your case going through the sequence number in the for loop results in a NULL pointer that we then access. Fixing it should be as simple as replacing the for (;;) { a few lines before with a while(cur_seq != NULL) {. Let me know if that fixes your specific crash, I'll commit to master in the meanwhile. If still an issue and unrelated to this issue here (where the problem is different) please open another issue.

[jackmjones25]

I'll do that. Thank you, sir.

[bhakimi]

this wont fix my issue does it?

[lminiero]

Updating won't harm. Both of you should try the reference counters branch too, which is more conservative with memory and will be merged in the next few weeks anyway (April or May).

[bhakimi]

i just updated to master, if master wont fix it i will try that branch

On Thu, Mar 16, 2017 at 5:17 AM, Lorenzo Miniero [e-mail]
wrote:

Updating won't harm. Both of you should try the reference counters branch
too, which is more conservative with memory and will be merged in the next
few weeks anyway (April or May).

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/meetecho/janus-gateway/issues/751#issuecomment-287039788,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADQXbf4sHD_2n9JqQN-EoWVJXuIGmEqjks5rmShXgaJpZM4LmTCi
.

[bhakimi]

@lminiero tried latest trunk and the refcounter branch, both still cause crashes for me

[jackmjones25]

@lminiero as requested: Mine seems to be fixed. No crashes since the update. Thanks!

[lminiero]

Fixed in #887.