Mumble: Mumble crashes when clients join too fast

[wfjsw]

Describe the bug
So after Murmur fails for me (#3878, #3893) I decided to write a custom server based on mumble-voip/grumble and I finally get rid of that retarded handlerLoop routine to achieve full asynchoronous authentication process. It works nicely until I restart the server, and all these disconnected clients flood in at once. Reports said mumble clients freeze and crash, and the bug reports window appears on next startup. Meanwhile server encounters no exception. Other reports also highlight the crash affect not only Mumble itself but also games and whole operation system, forcing them to restart the computer.

Steps to Reproduce
Steps to reproduce the behavior:
Have 500 toons join server in about 3 seconds.

Expected behavior
No crash is expected in this case. At least the crash should be limited to Mumble itself, not other random things.

Desktop (please complete the following information):

• OS: Windows 10
• Version: Mumble 1.3.0

Additional context
Server-side code is the same as https://github.com/mumble-voip/grumble/blob/master/cmd/grumble/server.go#L448-L685 with these two function merged into one and run in a seperated goroutine.

[Krzmbrzl]

Is this reproduceable? Aka can you get Mumble to crash every time this happens?

Also is there any kind of error message that might pop-up when this happens (or in the bug-report screen afterwards)?

Are all of your clients using Windows as their OS or do you have Mac/Linux user on this server as well. If so, do they experience the same crash/freeze?

Could you log which kind of messages are sent to the clients when this happens (which type of messages, which order are they in and how many of them are being sent)?

[wfjsw]

1. Yes. If numbers is sufficient, nearly 70% of the clients will crash. Won't get that if number is below 50.
2. No. I'll solve this problem on my own if such message or report is available.
3. It is windows according to reports I get.
4. It crashes after displaying channel list and (increasing) user list so I assume the only possible message is UserState.

[Krzmbrzl]

It crashes after displaying channel list and (increasing) user list so I assume the only possible message is UserState.

Hm okay. Then I assume that the client is somewhat flooded with UserState messages which probably causes some sort of race-condition in here
https://github.com/mumble-voip/mumble/blob/ec51a41cfe7a348d08c40468bc49aca0dbd3e9d8/src/mumble/Messages.cpp#L327-L349
Probably somewhere in that UserModel::addUser function.
I have the feeling that this is the kind of bug that might be fixed with #3901 - that's a big one though.

It would be really helpful to know where and why exactly the client is crashing though. If one of oyur clients was on Linux I'd suggest starting Mumble via gdb, provocate that crash and print out the backtrace and the error message of that crash. I don't have a clue how one does something like this on Windows though...

[wfjsw]

I can't afford to keep this bug on production server, so it would be really helpful if there is a load simulator to generate a few clients. Multiboxing 500 clients is too difficult for my computer.

[Krzmbrzl]

I can't afford to keep this bug on production server

To me this doesn't sound like a bug in the server. I'd assume this to be a bug in the client.

it would be really helpful if there is a load simulator to generate a few clients

Indeed. I don't know if we have something like this though :thinking:
@davidebeatrici do you know of something like this?

[wfjsw]

But if it is a bug that will crash every client then it is a bug in server implemention too. (At least Murmur won't cause this.)
I temporarily fixed this by adding a mutex to the whole authenticate function.

[Krzmbrzl]

That depends. As long as you are not violating the Mumble protocol, your server implementation might simply handle the thing differently than murmur and thus Mumble didn't encounter that bug before. Technically speaking the bug would still be in the client and in the client only.

I temporarily fixed this by adding a mutex to the whole authenticate function.

In your server implementation? So that only one client at a time can authenticate?

[wfjsw]

Yes. That literally slows down the authentication process and is receiving a positive impact.

[wfjsw]

I managed to get a console.txt after crash:

<W>2020-03-25 00:29:34.076 msgUserState(): unknown channel.
<W>2020-03-25 00:29:34.133 msgUserState(): unknown channel.
<W>2020-03-25 00:29:34.217 msgUserState(): unknown channel.
<W>2020-03-25 00:29:34.294 msgUserState(): unknown channel.
<W>2020-03-25 00:29:34.370 msgUserState(): unknown channel.
<W>2020-03-25 00:29:34.453 msgUserState(): unknown channel.
<W>2020-03-25 00:29:34.532 msgUserState(): unknown channel.
<W>2020-03-25 00:29:34.567 MainWindow: Message for nonexistent victim 121.
<W>2020-03-25 00:29:34.624 msgUserState(): unknown channel.
<W>2020-03-25 00:29:34.699 MainWindow: Message for nonexistent victim 132.
<W>2020-03-25 00:29:34.718 msgUserState(): unknown channel.
<W>2020-03-25 00:29:34.813 msgUserState(): unknown channel.
<W>2020-03-25 00:29:34.919 msgUserState(): unknown channel.
<W>2020-03-25 00:29:35.044 msgUserState(): unknown channel.
<W>2020-03-25 00:29:35.114 msgUserState(): unknown channel.
<W>2020-03-25 00:29:35.195 MainWindow: Message for nonexistent victim 217.
<W>2020-03-25 00:29:35.196 MainWindow: Message for nonexistent victim 5.
<W>2020-03-25 00:29:35.213 msgUserState(): unknown channel.
<W>2020-03-25 00:29:35.423 msgUserState(): unknown channel.
<W>2020-03-25 00:29:35.424 msgUserState(): unknown channel.
<W>2020-03-25 00:29:36.094 msgUserState(): unknown channel.
<W>2020-03-25 00:29:36.095 msgUserState(): unknown channel.
<W>2020-03-25 00:29:36.097 MainWindow: Message for nonexistent victim 200.
<W>2020-03-25 00:29:36.097 msgUserState(): unknown channel.
<W>2020-03-25 00:29:36.099 msgUserState(): unknown channel.
<W>2020-03-25 00:29:36.100 msgUserState(): unknown channel.
<W>2020-03-25 00:29:36.102 msgUserState(): unknown channel.
<W>2020-03-25 00:29:36.103 msgUserState(): unknown channel.
<W>2020-03-25 00:29:36.178 msgUserState(): unknown channel.
<W>2020-03-25 00:29:36.275 msgUserState(): unknown channel.
<W>2020-03-25 00:29:36.368 msgUserState(): unknown channel.
<W>2020-03-25 00:29:36.476 msgUserState(): unknown channel.
<W>2020-03-25 00:29:36.577 msgUserState(): unknown channel.
<W>2020-03-25 00:29:36.667 msgUserState(): unknown channel.
<W>2020-03-25 00:29:36.766 msgUserState(): unknown channel.
<W>2020-03-25 00:29:36.871 msgUserState(): unknown channel.
<W>2020-03-25 00:29:36.955 msgUserState(): unknown channel.
<W>2020-03-25 00:29:37.044 msgUserState(): unknown channel.
<W>2020-03-25 00:29:37.133 msgUserState(): unknown channel.
<W>2020-03-25 00:29:37.171 MainWindow: Message for nonexistent victim 188.
<W>2020-03-25 00:29:37.234 msgUserState(): unknown channel.
<W>2020-03-25 00:29:37.310 msgUserState(): unknown channel.
<W>2020-03-25 00:29:37.396 msgUserState(): unknown channel.
<W>2020-03-25 00:29:37.492 msgUserState(): unknown channel.
<W>2020-03-25 00:29:37.577 msgUserState(): unknown channel.
<W>2020-03-25 00:29:37.658 msgUserState(): unknown channel.
<W>2020-03-25 00:29:37.748 msgUserState(): unknown channel.
<W>2020-03-25 00:29:37.836 msgUserState(): unknown channel.
<W>2020-03-25 00:29:37.920 msgUserState(): unknown channel.
<W>2020-03-25 00:29:38.003 msgUserState(): unknown channel.
<W>2020-03-25 00:29:38.093 msgUserState(): unknown channel.
<W>2020-03-25 00:29:38.178 msgUserState(): unknown channel.
<W>2020-03-25 00:29:38.262 msgUserState(): unknown channel.
<W>2020-03-25 00:29:38.337 msgUserState(): unknown channel.
<W>2020-03-25 00:29:38.416 msgUserState(): unknown channel.
<W>2020-03-25 00:29:38.496 msgUserState(): unknown channel.
<W>2020-03-25 00:29:38.575 msgUserState(): unknown channel.
<W>2020-03-25 00:29:38.649 msgUserState(): unknown channel.
<W>2020-03-25 00:29:38.729 msgUserState(): unknown channel.
<W>2020-03-25 00:29:38.801 msgUserState(): unknown channel.
<W>2020-03-25 00:29:38.878 msgUserState(): unknown channel.
<W>2020-03-25 00:29:38.948 msgUserState(): unknown channel.
<W>2020-03-25 00:29:39.025 msgUserState(): unknown channel.
<W>2020-03-25 00:29:39.095 msgUserState(): unknown channel.
<W>2020-03-25 00:29:39.162 msgUserState(): unknown channel.
<W>2020-03-25 00:29:39.230 msgUserState(): unknown channel.
<W>2020-03-25 00:29:39.301 msgUserState(): unknown channel.
<W>2020-03-25 00:29:39.369 msgUserState(): unknown channel.
<W>2020-03-25 00:29:39.439 msgUserState(): unknown channel.
<W>2020-03-25 00:29:39.506 msgUserState(): unknown channel.
<W>2020-03-25 00:29:39.572 msgUserState(): unknown channel.
<W>2020-03-25 00:29:39.644 msgUserState(): unknown channel.
<W>2020-03-25 00:29:39.716 msgUserState(): unknown channel.
<W>2020-03-25 00:29:39.787 msgUserState(): unknown channel.
<W>2020-03-25 00:29:39.858 msgUserState(): unknown channel.
<W>2020-03-25 00:29:39.925 msgUserState(): unknown channel.
<W>2020-03-25 00:29:39.991 msgUserState(): unknown channel.
<W>2020-03-25 00:29:40.056 msgUserState(): unknown channel.
<W>2020-03-25 00:29:40.119 msgUserState(): unknown channel.
<W>2020-03-25 00:29:40.188 msgUserState(): unknown channel.
<W>2020-03-25 00:29:40.245 msgUserState(): unknown channel.
<W>2020-03-25 00:29:40.330 msgUserState(): unknown channel.
<W>2020-03-25 00:29:40.396 msgUserState(): unknown channel.
<W>2020-03-25 00:29:40.464 msgUserState(): unknown channel.
<W>2020-03-25 00:29:40.527 msgUserState(): unknown channel.
<W>2020-03-25 00:29:40.594 msgUserState(): unknown channel.
<W>2020-03-25 00:29:40.661 msgUserState(): unknown channel.
<W>2020-03-25 00:29:40.722 msgUserState(): unknown channel.
<W>2020-03-25 00:29:40.788 msgUserState(): unknown channel.
<W>2020-03-25 00:29:40.853 msgUserState(): unknown channel.
<W>2020-03-25 00:29:40.919 msgUserState(): unknown channel.
<W>2020-03-25 00:29:40.980 msgUserState(): unknown channel.
<W>2020-03-25 00:29:41.042 msgUserState(): unknown channel.
<W>2020-03-25 00:29:41.108 msgUserState(): unknown channel.
<W>2020-03-25 00:29:41.172 msgUserState(): unknown channel.
<W>2020-03-25 00:29:41.236 msgUserState(): unknown channel.
<W>2020-03-25 00:29:41.300 msgUserState(): unknown channel.
<W>2020-03-25 00:29:41.373 msgUserState(): unknown channel.
<W>2020-03-25 00:29:41.429 msgUserState(): unknown channel.
<W>2020-03-25 00:29:41.493 msgUserState(): unknown channel.
<W>2020-03-25 00:29:41.557 msgUserState(): unknown channel.
<W>2020-03-25 00:29:41.620 msgUserState(): unknown channel.
<W>2020-03-25 00:29:41.685 msgUserState(): unknown channel.
<W>2020-03-25 00:29:41.748 msgUserState(): unknown channel.
<W>2020-03-25 00:29:41.811 msgUserState(): unknown channel.
<W>2020-03-25 00:29:41.967 msgUserState(): unknown channel.
<W>2020-03-25 00:29:42.017 msgUserState(): unknown channel.
<W>2020-03-25 00:29:42.075 msgUserState(): unknown channel.
<W>2020-03-25 00:29:42.138 msgUserState(): unknown channel.
<W>2020-03-25 00:29:42.207 msgUserState(): unknown channel.
<W>2020-03-25 00:29:42.278 msgUserState(): unknown channel.
<W>2020-03-25 00:29:42.342 msgUserState(): unknown channel.
<W>2020-03-25 00:29:42.406 msgUserState(): unknown channel.
<W>2020-03-25 00:29:42.469 msgUserState(): unknown channel.
<W>2020-03-25 00:29:42.531 msgUserState(): unknown channel.
<W>2020-03-25 00:29:42.599 msgUserState(): unknown channel.
<W>2020-03-25 00:29:42.668 msgUserState(): unknown channel.
<W>2020-03-25 00:29:42.732 msgUserState(): unknown channel.
<W>2020-03-25 00:29:42.796 msgUserState(): unknown channel.
<W>2020-03-25 00:29:42.865 msgUserState(): unknown channel.
<W>2020-03-25 00:29:42.936 msgUserState(): unknown channel.
<W>2020-03-25 00:29:43.002 msgUserState(): unknown channel.
<W>2020-03-25 00:29:43.069 msgUserState(): unknown channel.
<W>2020-03-25 00:29:43.135 msgUserState(): unknown channel.
<W>2020-03-25 00:29:43.199 msgUserState(): unknown channel.
<W>2020-03-25 00:29:43.266 msgUserState(): unknown channel.
<W>2020-03-25 00:29:43.329 msgUserState(): unknown channel.
<W>2020-03-25 00:29:43.418 msgUserState(): unknown channel.
<W>2020-03-25 00:29:43.491 msgUserState(): unknown channel.
<W>2020-03-25 00:29:43.560 msgUserState(): unknown channel.
<W>2020-03-25 00:29:58.401 Qt has caught an exception thrown from an event handler. Throwing
exceptions from an event handler is not supported in Qt.
You must not let any exception whatsoever propagate through Qt code.
If that is not possible, in Qt 5 you must at least reimplement
QCoreApplication::notify() and catch all exceptions there.

<W>2020-03-25 00:29:58.401 Qt has caught an exception thrown from an event handler. Throwing
exceptions from an event handler is not supported in Qt.
You must not let any exception whatsoever propagate through Qt code.
If that is not possible, in Qt 5 you must at least reimplement
QCoreApplication::notify() and catch all exceptions there.

[Krzmbrzl]

msgUserState(): unknown channel. is coming from
https://github.com/mumble-voip/mumble/blob/310497a40054babeec7b39cfd9c1415b6cb9423d/src/mumble/Messages.cpp#L320-L323

This might be an indication for the server starting to send joining clients to Mumble before having synchronized all channels first. Therefore clients can claim to be in a channel, the local Mumble instance doesn't know anything about (yet).

Similarly MainWindow: Message for nonexistent victim 188. from
https://github.com/mumble-voip/mumble/blob/310497a40054babeec7b39cfd9c1415b6cb9423d/src/mumble/Messages.cpp#L46-L49
indicates that messages that act on users are sent before all users have been reported to Mumble. Hence it doesn't know what to do with those messages (after all their target doesn't exist (yet)).

[wfjsw]

Actually I got a copy of coredump:
Mumble.zip
Trying to inspect it with windbg now...

[wfjsw]

0:000> .excr
rax=0000000000000000 rbx=0000000000000006 rcx=0000000000000001
rdx=000001289f1a7428 rsi=000001289d043ea0 rdi=000001289ccce780
rip=00007ffba2149f9f rsp=0000000e68569ec0 rbp=0000000e68569f89
 r8=0000000000000083  r9=0000000055c117e8 r10=0000000e68569f10
r11=0000000e68569de0 r12=00007ffba36ef8a8 r13=00007ffba38628b0
r14=0000000e6856a200 r15=000001289a3dec00
iopl=0         nv up ei pl nz na pe nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202
mumble_app!MainWindow::findDesiredChannel+0x36f:
00007ffb`a2149f9f 4c8b4048        mov     r8,qword ptr [rax+48h] ds:00000000`00000048=????????????????

  *** Stack trace for last set context - .thread/.cxr resets it
 # Child-SP          RetAddr           Call Site
00 0000000e`68569ec0 00007ffb`a21ee12a mumble_app!MainWindow::findDesiredChannel+0x36f [d:\jenkins\workspace\mumble_windows_x64\src\mumble\mainwindow.cpp @ 965] 
01 0000000e`68569ff0 00007ffb`a21495d5 mumble_app!MainWindow::msgServerSync+0x28a [d:\jenkins\workspace\mumble_windows_x64\src\mumble\messages.cpp @ 121] 
02 0000000e`6856a100 00007ffb`a2c95fb8 mumble_app!MainWindow::customEvent+0x2a5 [d:\jenkins\workspace\mumble_windows_x64\src\mumble\mainwindow.cpp @ 3187] 
03 0000000e`6856a7e0 00007ffb`a26f84b9 mumble_app!QObject::event+0x2f8 [c:\mumblebuild\win64-static-1.3.x-2017-04-14-c6f166d-858.build\qt-everywhere-opensource-src-5.6.2\qtbase\src\corelib\kernel\qobject.cpp @ 1283] 
04 0000000e`6856a980 00007ffb`a27f7c49 mumble_app!QWidget::event+0x1079 [c:\mumblebuild\win64-static-1.3.x-2017-04-14-c6f166d-858.build\qt-everywhere-opensource-src-5.6.2\qtbase\src\widgets\kernel\qwidget.cpp @ 9181] 
05 0000000e`6856abb0 00007ffb`a277b444 mumble_app!QMainWindow::event+0x5d9 [c:\mumblebuild\win64-static-1.3.x-2017-04-14-c6f166d-858.build\qt-everywhere-opensource-src-5.6.2\qtbase\src\widgets\widgets\qmainwindow.cpp @ 1544] 
06 0000000e`6856ac90 00007ffb`a277d1ef mumble_app!QApplicationPrivate::notify_helper+0x154 [c:\mumblebuild\win64-static-1.3.x-2017-04-14-c6f166d-858.build\qt-everywhere-opensource-src-5.6.2\qtbase\src\widgets\kernel\qapplication.cpp @ 3807] 
07 0000000e`6856acc0 00007ffb`a2d06c87 mumble_app!QApplication::notify+0x1d7f [c:\mumblebuild\win64-static-1.3.x-2017-04-14-c6f166d-858.build\qt-everywhere-opensource-src-5.6.2\qtbase\src\widgets\kernel\qapplication.cpp @ 3646] 
08 0000000e`6856b270 00007ffb`a2d04042 mumble_app!QCoreApplication::notifyInternal2+0xb7 [c:\mumblebuild\win64-static-1.3.x-2017-04-14-c6f166d-858.build\qt-everywhere-opensource-src-5.6.2\qtbase\src\corelib\kernel\qcoreapplication.cpp @ 1015] 
09 (Inline Function) --------`-------- mumble_app!QCoreApplication::sendEvent+0x15 [c:\mumblebuild\win64-static-1.3.x-2017-04-14-c6f166d-858.build\qt-everywhere-opensource-src-5.6.2\qtbase\src\corelib\kernel\qcoreapplication.h @ 225] 
0a 0000000e`6856b2f0 00007ffb`a2a921fa mumble_app!QCoreApplicationPrivate::sendPostedEvents+0x272 [c:\mumblebuild\win64-static-1.3.x-2017-04-14-c6f166d-858.build\qt-everywhere-opensource-src-5.6.2\qtbase\src\corelib\kernel\qcoreapplication.cpp @ 1650] 
0b (Inline Function) --------`-------- mumble_app!QEventDispatcherWin32::sendPostedEvents+0x14 [c:\mumblebuild\win64-static-1.3.x-2017-04-14-c6f166d-858.build\qt-everywhere-opensource-src-5.6.2\qtbase\src\corelib\kernel\qeventdispatcher_win.cpp @ 1287] 
0c 0000000e`6856b3c0 00007ffb`a2da382c mumble_app!QWindowsGuiEventDispatcher::sendPostedEvents+0x1a [c:\mumblebuild\win64-static-1.3.x-2017-04-14-c6f166d-858.build\qt-everywhere-opensource-src-5.6.2\qtbase\src\platformsupport\eventdispatchers\qwindowsguieventdispatcher.cpp @ 76] 
0d 0000000e`6856b3f0 00007ffc`16c35c0d mumble_app!qt_internal_proc+0x4dc [c:\mumblebuild\win64-static-1.3.x-2017-04-14-c6f166d-858.build\qt-everywhere-opensource-src-5.6.2\qtbase\src\corelib\kernel\qeventdispatcher_win.cpp @ 438] 
0e 0000000e`6856b530 00007ffc`16c35602 user32!CallWindowProcW+0x3bd
0f 0000000e`6856b6c0 00007ffb`a2da456f user32!DispatchMessageW+0x1f2
10 0000000e`6856b740 00007ffb`a2a92228 mumble_app!QEventDispatcherWin32::processEvents+0x52f [c:\mumblebuild\win64-static-1.3.x-2017-04-14-c6f166d-858.build\qt-everywhere-opensource-src-5.6.2\qtbase\src\corelib\kernel\qeventdispatcher_win.cpp @ 838] 
11 0000000e`6856eab0 00007ffb`a2d6450e mumble_app!QWindowsGuiEventDispatcher::processEvents+0x18 [c:\mumblebuild\win64-static-1.3.x-2017-04-14-c6f166d-858.build\qt-everywhere-opensource-src-5.6.2\qtbase\src\platformsupport\eventdispatchers\qwindowsguieventdispatcher.cpp @ 69] 
12 (Inline Function) --------`-------- mumble_app!QEventLoop::processEvents+0x17 [c:\mumblebuild\win64-static-1.3.x-2017-04-14-c6f166d-858.build\qt-everywhere-opensource-src-5.6.2\qtbase\src\corelib\kernel\qeventloop.cpp @ 128] 
13 0000000e`6856eae0 00007ffb`a2d0619e mumble_app!QEventLoop::exec+0xfe [c:\mumblebuild\win64-static-1.3.x-2017-04-14-c6f166d-858.build\qt-everywhere-opensource-src-5.6.2\qtbase\src\corelib\kernel\qeventloop.cpp @ 205] 
14 0000000e`6856eb50 00007ffb`a2140f69 mumble_app!QCoreApplication::exec+0x14e [c:\mumblebuild\win64-static-1.3.x-2017-04-14-c6f166d-858.build\qt-everywhere-opensource-src-5.6.2\qtbase\src\corelib\kernel\qcoreapplication.cpp @ 1285] 
15 0000000e`6856ebc0 00007ffb`a2141be4 mumble_app!main+0x3409 [d:\jenkins\workspace\mumble_windows_x64\src\mumble\main.cpp @ 559] 
16 0000000e`6856f9f0 00007ff6`b28c122c mumble_app!MumbleMain+0x394 [d:\jenkins\workspace\mumble_windows_x64\src\mumble\main.cpp @ 701] 
17 0000000e`6856fa80 00007ff6`b28c2953 mumble!wWinMain+0x22c [d:\jenkins\workspace\mumble_windows_x64\src\mumble_exe\mumble_exe.cpp @ 219] 
18 (Inline Function) --------`-------- mumble!invoke_main+0x21 [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 113] 
19 0000000e`6856fb80 00007ffc`15cc7bd4 mumble!__scrt_common_main_seh+0x117 [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 253] 
1a 0000000e`6856fbc0 00007ffc`17c0ced1 kernel32!BaseThreadInitThunk+0x14
1b 0000000e`6856fbf0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

[wfjsw]

Guess I have a clue now... I did move ServerSync from before ServerConfig to after it...

[wfjsw]

https://github.com/mumble-voip/mumble/blob/48e793e381390f224a6d2d5b4506ed0b71fab98a/src/mumble/MainWindow.cpp#L965

This line is crashing. What does it do?

[Krzmbrzl]

qtvUsers is of type UserView which inherits from QTreeView. setCurrentIndex sets the current selection to the given index.

Inidces are used to represent entries in the User-view (either channels or users). Thus this line will select the channel the local user is currently in.
(ClientUser::get(g.uiSession) gets the local user)

[wfjsw]

I made a simple stress test by sending 800 UserState simulating user joining the server at the same time but crash does not happen. I assume the problem lays somewhere else. Closing this issue.

NOTE: If ServerSync was sent without UserState beforehand a crash happen immediately. This is probably the cause of this issue.

[Krzmbrzl]

Thanks for reporting your findings :+1:

NOTE: If ServerSync was sent without UserState beforehand a crash happen immediately. This is probably the cause of this issue.

Yeah I guess that makes sense given that UserState is used to transmit information about the different users whereas ServerSync is sent to indicate that the client has all information it needs and thus can start acting with it (e.g. showing the UI, etc.).

[wfjsw]

Actually in my own server implemention weirds thing often happens, for example, client user list sometimes go desync from the server and I have no idea how to find them out or resync it to a correct state. It is not causing apparent issue currently (as it is often failure to remove disconnected client).