Janus-gateway: videoroom deadlocks with room destroy

[tugtugtug]

With the latest rev (7a1f8e5db8df40e7d6be2a01d8aad0ca0508b7ae) of master, videoroom deadlocks consistently when room is destroyed.

Issue is due to janus_videoroom_handle_message locks the sessions_mutex and calls gateway->end_session(p->session->handle), which then calls back janus_videoroom_destroy_session.

I know that all our examples do not create/destroy rooms, but we probably should have some test cases that run regularly covering these cases.

backtrace at:
https://pastebin.com/FFxLJXQS

but the line numbers probably will be off by a little bit due to changes locally.

If you want to open an issue, please make sure that:

1. you actually found a bug: for generic questions, use the meetecho-janus group instead;
2. nobody opened the same bug already (do a search);
3. the issue wasn't already solved;
4. you don't paste a huge amount of text in the issue or comments: use a service like pastebin or similar;
5. you provide a GDB stacktrace if Janus crashed on you, and/or output from AddressSanitizer or Valgrind (more details availablehere).

If so, just remove this text and share your findings!

[tugtugtug]

Also deadlocks with kick, basically these commands that result in a gateway action that calls back with the sessions_mutex locked in the janus_videoroom_handle_message function.

https://pastebin.com/LeMTfWZi

[tugtugtug]

Audiobrdige has the exact same bug when calling gateway->close_pc, just worse, it locks both sessions_mutex and rooms_mutex, which would deadlock 2 twice.

https://pastebin.com/iq8PAQAE

[lminiero]

Thanks for the info, we never test the room destruction much. Yep, it's a known issue that close_pc actually gets back in hangup_media right away (it's done on the same thread), but we thought we had that sorted down. I guess in part that this could be actually considered a core issue (close_pc should originate an hangup_media asynchronously) more than something plugins should worry about, but we'll think of the best way to fix this.

[lminiero]

I went for the core fix: now both close_pc and end_session are truly asynchronous, meaning the core schedules what they ask to do instead of doing it right away. As such, they return right away, and there's no risk of deadlock when you get a destroy_session and hangup_media callback anymore. Notice that this is only in master, right now, so if you were testing the refcount branch (or others) you'll need to wait for me to merge the changes there too.

[lminiero]

Updated the refcount branch too. Please let me know if you're still encountering issues.

[tugtugtug]

@lminiero master fix looks good so far, will look at the refcount branch next week.

[lminiero]

Closing then. Please let me know if you find out it's still an issue in master and/or refcount and I'll reopen.