Meshcentral: Remote desktop fails, KVM child process exits

[changchichung]

meshcentral server OS: ubuntu 18.04.3 x64
npm: 3.5.2
meshcentral version: 0.4.3-b

client OS: ubuntu 18.04 x64

error

I have no idea why this happened . I can connect to the client couple days ago . then the error happened this morning , how do I provide more detail log ?

[changchichung]

meshagent.log

cat meshagent.log 

[2019-10-25 10:11:31 AM] ** CRASH **
[/usr/local/mesh/meshagent_E0AC4772DF2BD5E9]
/lib/x86_64-linux-gnu/libc.so.6(+0x3ef20) [0x7feec2515f20]
/usr/lib/x86_64-linux-gnu/libX11.so.6(XQueryExtension+0x2d) [0x7feec19d26ad]
/usr/local/mesh/meshagent() [0x48f6cf]
/usr/local/mesh/meshagent() [0x48fc48]
/usr/local/mesh/meshagent() [0x490866]
/usr/local/mesh/meshagent() [0x490908]
/usr/local/mesh/meshagent() [0x494f81]
/usr/local/mesh/meshagent() [0x436c60]
/usr/local/mesh/meshagent() [0x462900]
/usr/local/mesh/meshagent() [0x43591e]
/usr/local/mesh/meshagent() [0x436c1f]
/usr/local/mesh/meshagent() [0x436e4b]
/usr/local/mesh/meshagent() [0x43c1a6]
/usr/local/mesh/meshagent() [0x43c4d8]
/usr/local/mesh/meshagent() [0x47622c]
/usr/local/mesh/meshagent() [0x436c60]
/usr/local/mesh/meshagent() [0x436e4b]
/usr/local/mesh/meshagent() [0x43c1a6]
/usr/local/mesh/meshagent() [0x43c4d8]
/usr/local/mesh/meshagent() [0x46a240]
/usr/local/mesh/meshagent() [0x48546b]
/usr/local/mesh/meshagent() [0x46a06e]
/usr/local/mesh/meshagent() [0x466f4b]
/usr/local/mesh/meshagent() [0x40f413]
/usr/local/mesh/meshagent() [0x40fb1a]
/usr/local/mesh/meshagent() [0x4140cb]
/usr/local/mesh/meshagent() [0x4969ac]
/usr/local/mesh/meshagent() [0x4984be]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7feec24f8b97]
/usr/local/mesh/meshagent() [0x40d895]

[krayon007]

Are any exceptions showing up on the console tab?

In the next agent update, I'll modify the child kvm process on Linux to support core dumps if the parent has it enabled. It should make this type of thing easier to debug. But usually if the child kvm process crashes, it's an X11 error. Just today, in clear linux i found and issue where there are 2 x11 libraries installed in the platform, but one of them uses processer instructions that my PC doesn't have, so if it gets used, X crashes with the same message you are getting.

I'm working on a fix for that. If on your client machine, you do:

ldconfig -p | grep libX11

Do you see more than one library, where one has hwcap specified?

[changchichung]

Error trying to determine XAUTHORITY/DISPLAY: TypeError: cannot read property 'split' of null 

[krayon007]

Nevermind, looking at the path: /usr/lib/x86_64-linux-gnu/libX11.so.6 , that looks like the regular generic version... Not sure why it's crashing at XQueryExtension. If you set, "coreDumpEnabled=1" in the .msh file, does it generate a core file? If that crash handler got invoked, then if the above flag is set, it should generate a Core file, on crash, which will make it easier to debug.

[changchichung]

2019-10-25 10:33:33 [minion@hqpc074 ~]$ sudo ldconfig -p|grep libX11
    libX11.so.6 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libX11.so.6
    libX11-xcb.so.1 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libX11-xcb.so.1
2019-10-25 10:33:46 [minion@hqpc074 ~]$ 

[changchichung]

Nevermind, looking at the path: /usr/lib/x86_64-linux-gnu/libX11.so.6 , that looks like the regular generic version... Not sure why it's crashing at XQueryExtension. If you set, "coreDumpEnabled=1" in the .msh file, does it generate a core file? If that crash handler got invoked, then if the above flag is set, it should generate a Core file, on crash, which will make it easier to debug.

where should I find the core dump file ?
there are 3 files in /usr/local/mesh

2019-10-25 10:50:34 [minion@hqdc027 log]$ ls /usr/local/mesh
meshagent  meshagent.db  meshagent.msh
2019-10-25 10:50:38 [minion@hqdc027 log]$

[krayon007]

You can upload it to Google drive or Dropbox, or similar. Or even on your meshcentral server. Then you can email me the link, and I can take a look at it.

Ubuntu names the file 'core'

[changchichung]

well , there is no such file in that folder .

[changchichung]

when I try to connect remote desktop and get the error , there are something in /var/log/apport.log like this

ERROR: apport (pid 4431) Fri Oct 25 11:02:13 2019: executable does not belong to a package, ignoring
ERROR: apport (pid 5984) Fri Oct 25 11:04:01 2019: called for pid 5953, signal 11, core limit 18446744073709551615, dump mode 2
ERROR: apport (pid 5984) Fri Oct 25 11:04:01 2019: ignoring implausibly big core limit, treating as unlimited
ERROR: apport (pid 5984) Fri Oct 25 11:04:01 2019: not creating core for pid with dump mode of 2
ERROR: apport (pid 5984) Fri Oct 25 11:04:01 2019: executable: /usr/local/mesh/meshagent (command line "/usr/local/mesh/meshagent")
ERROR: apport (pid 5984) Fri Oct 25 11:04:01 2019: executable does not belong to a package, ignoring

and /var/log/syslog

Oct 25 11:04:01 hqdc027 kernel: [ 3568.938048] meshagent[5953]: segfault at 968 ip 00007f0a0f88e6ad sp 00007ffcbae69270 error 4 in libX11.so.6.3.0[7f0a0f85a000+133000]

is that helpful ?

[krayon007]

I'll have to do some research, as I've never seen that error before, not even on my Ubuntu 18.04 system
It looks like that error says it is not creating the dump file...

[changchichung]

well , I found that kvem child process exists issue only occur when nobody login to desktop environment.

that means , if I reboot a machine , do nothing in login screen , then open remote desktop from meshcentral will cause the issue , but if I login with any account , then I can open remote desktop.

not sure if this helps.

[krayon007]

I should have looked more closely at that you said. This is normal behavior with Ubuntu 18 and newer, because of a new security policy they implemented that prevents attaching an X client when no users are logged in.

I'll do some testing, but I thought I had it return an error message instead of spawning a child process in this case.

[changchichung]

please let us know how to help to test.

[krayon007]

I did some testing, it actually does output the error message I mentioned, you just don't see it, because the Child Disconnect message is also sent, which hides the previous message. I'll talk to Ylian, to see if we can have it concatenate messages, instead of only showing the last message.

[changchichung]

I think this issue might be related about proxy.
I try to install a new client with multiple user accounts , some of the users connect to internet via proxy , which managed by gsettings. when login with non-internet users , I can connect to remote desktop , and with internet users , I cant do the same.
so , I add a user level proxy settings named ignore-hosts , add meshcentral server ip address into it, then remote desktop works .

[changchichung]

@krayon007

Can you please try that if add Environment="NO_PROXY=192.168.0.0/16" in /etc/systemd/system/multi-user.target.wants/meshagent.service works or not ?
I think pass no_proxy in meshagent's systemd service file seems not work

[krayon007]

Well I know NO_PROXY is not going to work, because the environment variable is never queried... I will add support for this environment variable... However, I think this issue is different than the original issue in this thread, as Ubuntu 18.04, will not allow a X connection when no users are logged in, which is why the child process exits.

If the agent is trying to use a proxy when none is required, that would prevent the websocket tunnel connection from succeeding, which should result in the UX saying "connecting..." without completing. Or saying connected, but presented with a black screen, which would happen if the browser connection completes, but the agent connection fails. If you are actually getting the error messages displayed on the screen, that means the tunnel connection by the agent was successful, because the agent is the one that sends that message over the tunnel.

[jmbabazi]

I had the same issue on my Ubuntu 18.04 and to solve the issue, I had to enable auto login for a user.

[alyatis85]

Hi Ylian,

I tested this with an Ubuntu 19.10 client and get the same KVM message : "kvm child process has unexpectedly exited",

also i've got the followings message in the terminal :
uncaughtException1: Error: => EventEmitter.emit(): Event dispatch for 'end' on 'memoryStream' threw an exception: TypeError: undefined not callable (property 'trim' of [object Function]) in method '()'
uncaughtException1: Error: child_process.subProcess.exit(): => EventEmitter.emit(): Event dispatch for 'exit' on 'childProcess.subProcess' threw an exception: TypeError: undefined not callable (property 'indexOf' of [object Function]) in method '()'

Is that sounds to you ? my version of MC is the 0.5.1-a

NB : thanks for the very usefull function to give rights to a specific machine for a given users. It works great.

Have a nice day, without covid19 !

[edlins]

I also saw this error when I rebooted Ubuntu and the display was screen saved blank. Moving the mouse to wake up the display fixed it.
EDIT: If it happens again I'll try xdotool in the terminal to send mouse movements.

[krayon007]

Interesting. I'll see if I can recreate this scenario with the blank screen saver...

[krayon007]

I tried this with Ubuntu 18.04 LTS, and when the screen blanks, the KVM still works. It connects to a back screen, but moving the mouse on the kvm, caused the remote screen to turn on...

I'll build a 19.10 system, and see if the results are different...

[krayon007]

Ok, I just built a Ubuntu 19.10 system, and it still behaves the same as my 18.04 system. Are you using the built in power saver, to blank the screen after a fixed amount of time, or are you using a 3rd party screen saver?