Janus-gateway: Segmentation fault in janus_websocket_send_message

[peffis]

Hi, I'm using the websocket interface to Janus and attaching to the audiobridge plugin. I use the same janus.js in the browser and very similar javascript code as in the audiobridge test demo. Occasionally (it is not too uncommon actually) Janus crashes when I refresh the page (which means I had a an attached session to audiobridge running with two users in the room and then I just hit refresh in one of the browsers).

I have a core file from the crash and the stack trace looks like this:

Core was generated by `/opt/janus/bin/janus'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f27633d44b8 in lws_callback_on_writable () from /usr/lib/libwebsockets.so.12
[Current thread is 1 (Thread 0x7f2725ff3700 (LWP 92))]
(gdb) bt
#0  0x00007f27633d44b8 in lws_callback_on_writable () from /usr/lib/libwebsockets.so.12
#1  0x00007f27635fae34 in janus_websockets_send_message (transport=0x7f2748001620, request_id=<optimized out>, admin=<optimized out>, message=0x7f2734001a90) at transports/janus_websockets.c:807
#2  0x000000000043dd88 in janus_process_success (request=request@entry=0x7f2748000c30, payload=payload@entry=0x7f2734001a90) at janus.c:2384
#3  0x0000000000440808 in janus_process_incoming_request (request=0x7f2748000c30) at janus.c:901
#4  0x0000000000447b4d in janus_transport_task (data=0x7f2748000c30, user_data=<optimized out>) at janus.c:2718
#5  0x00007f277d85954e in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#6  0x00007f277d858bb5 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#7  0x00007f277c32b6ba in start_thread (arg=0x7f2725ff3700) at pthread_create.c:333
#8  0x00007f277c0613dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

I have used Janus in the past but using the http interface for the API and I have never experienced it as unstable as this before. Is it a known issue that the websocket interface is not very stable perhaps?

[peffis]

This happens during detach from the old session while creating the new one. I suppose it could mean that it is trying to send some message on a websocket that probably has already closed or has had some part of its resources freed, but I cannot tell if the crash is related to the old session terminating or the new session being setup.

[lminiero]

As I've said tons of times (and I'm getting tired of this) logs don't belong inline. If they're small use quotes, or they'll mess up issues with non-existing references to other issues (like yours did).

[lminiero]

Make sure you're on master, we fixed several things lately.

[peffis]

(I apologize for the earlier inlining of the logs)

Yes, this was on master (built 3 hours ago - 7th of December). So I should have all your commits.

[peffis]

Allow me to correct myself. I noticed that the build process had cached the source since three days back when I did the build so I did not have your latest commits. After having updated I don't get the crash with the websocket connection but instead, when I reload the page, it does not seem to be able to reconnect to the websocket interface. The connection hangs for a while before giving up. So it feels like instead of the crash we might now have some kind of dead-lock which invalidates the websocket interface after the disconnect of the first successful session.

janus.js:52 WebSocket connection to 'ws://localhost:8188/' failed: Connection closed before receiving a handshake response

I see nothing in the janus log when I now try to connect to the websocket but a message saying that it is "Destroying WebSocket client".

[lminiero]

I thought it could be related to a WebSockets deadlock we fixed in the refcount branch, lately, but what caused it there is not in master, so it's a different thing. To see why it deadlocks and where, you'll have to enable the locking debug via the Admin API, either in the admin.html demo or with this curl-based one-liner:

curl -X POST -H "Content-Type: application/json" -d '{"janus": "set_locking_debug", "debug": true, "transaction": "123", "admin_secret": "janusoverlord"}' http://localhost:7088/admin/

Once you get it to deadlock, pass the log to this simple nodejs application we wrote to spot deadlocks, and that we used in an issue exactly to address websockets freezes.

You'll probably want to disable all plugins except the ones you're using (audiobridge + websockets) or you may get the log spammed by mutexes belonging to plugins you don't care about.

[lminiero]

PS: you may want to check if you're getting the same behaviour in #403 too.

[peffis]

It doesn't seem like my Janus installation has any admin interface on port 7088. Perhaps it is on some other port or perhaps I need to enable it somehow?

[lminiero]

https://github.com/meetecho/janus-gateway/blob/master/conf/janus.transport.http.cfg.sample.in#L39

[peffis]

Thanks. I did the tests above, got the log and ran your node program on it. It says Locks still taken: {} so I guess that means that this is NOT a dead-lock issue. But it is very repeatable for me. Two clients join through websocket interface, both attach and share their audio in audiobridge room. One user hits reload - the connection to the websocket fails. The websocket interface remains broken until I restart janus.

[lminiero]

Tried replicating locally, refreshing tons of times for several minutes, and couldn't. Did you build libwebsockets with -DLWS_MAX_SMP=1 as suggested in the README?

[peffis]

No, I'm afraid I missed that one (or rather, our build script is one that we used about one year back where that instruction was probably not there, and I did not read the build instructions this time, sorry :( ). I will try and rebuild libwebsocket and try again....

[peffis]

I tried now with the LWS_MAX_SMP=1 and still the same result. The websocket interface become unreachable after the first session. I'm going to try the "stable libwebsocket" version also to see if this changes the behaviour. I should also mention that I'm running this Janus instance inside a Docker container using a turn server. I don't see how that could affect the websocket interface though. We'll see what happens with a different version of libwebsocket...

[peffis]

With a line like the one commented in the README ("git checkout v2.4-stable")
and also with LWS_MAX_SMP=1, it seems to actually work. It does no longer hang the interface after the first session and so far I have not managed to crash it either. So I think you can close this issue.

If anyone is reading this and also struggling with the websocket interface, the final build command for libwebsockets that made the interface work for me is the following:

git clone https://github.com/warmcat/libwebsockets.git /tmp/libwebsockets \
  && cd /tmp/libwebsockets \
  && git checkout v2.4-stable \
  && [ -e build ] || mkdir build \
  && cd build \
  && cmake -DLWS_MAX_SMP=1 -DCMAKE_INSTALL_PREFIX:PATH=/usr -DCMAKE_C_FLAGS="-fpic" .. \
  && make \
  && make install

[atoppi]

Thanks for confirming it's working now, closing then.