Arduino-pro-ide: Ubuntu 18.04 - clangd process keeps running after closing IDE and terminal.

[F-K-F-F]

Hi

I noticed my CPU was running a little more than usual and it turns out clangd is using up to 7%, after i closed the IDE and the terminal.

Is this related to Arduino-IDE?

[jankeromnes]

I'm not sure if this is the same problem, but on Ubuntu Disco 19.04, running & closing Arduino Pro seems to leave a bunch of zombie & orphaned processes behind, including clangd.

Specifically, these processes were not there before running the IDE, and won't go away after I quit the IDE:

$ ps axjf
   PPID     PID    PGID     SID TTY        TPGID STAT   UID   TIME COMMAND
[...]
      1    2349    2347    1419 pts/1       1419 Z    33333   0:00 [arduino.pro.ide] <defunct>
      1    2393    2347    1419 pts/1       1419 Z    33333   0:02 [arduino.pro.ide] <defunct>
      1    2440    2347    1419 pts/1       1419 Z    33333   0:00 [sh] <defunct>
      1    2446    2446    2446 ?             -1 Ssl  33333   0:00 /workspace/arduino-editor/Arduino Pro IDE-v0.0.3-linux/arduino.pro.ide /workspace/arduino-editor/Arduino Pro IDE-v0.0.3-linux/resources/app/node_modules/arduino-ide-extension/lib/node/daemon-watcher.js 2393 2440
      1    2499    2347    1419 pts/1       1419 Sl   33333   0:01 /workspace/arduino-editor/Arduino Pro IDE-v0.0.3-linux/arduino.pro.ide /workspace/arduino-editor/Arduino Pro IDE-v0.0.3-linux/resources/app/node_modules/@theia/core/lib/node/messaging/ipc-bootstrap.js
      1    2564    2347    1419 pts/1       1419 Z    33333   0:00 [arduino-languag] <defunct>
      1    2569    2347    1419 pts/1       1419 Z    33333   0:00 [clangd.main] <defunct>

Maybe we should at least make sure that no processes are left behind after exit?

EDIT: FYI, this is what the process tree looks like while the IDE is running:

ps axjf

   PPID     PID    PGID     SID TTY        TPGID STAT   UID   TIME COMMAND
[...]
   1419    3998    3998    1419 pts/1       3998 Sl+  33333   0:01  |   \_ ./arduino.pro.ide
   3998    4000    3998    1419 pts/1       3998 S+   33333   0:00  |       \_ /workspace/arduino-editor/Arduino Pro IDE-v0.0.3-linux/arduino.pro.ide --type=zygote --no-sandbox
   4000    4138    3998    1419 pts/1       3998 Sl+  33333   0:06  |       |   \_ /workspace/arduino-editor/Arduino Pro IDE-v0.0.3-linux/arduino.pro.ide --type=renderer --no-sandbox --enable-features=SharedArrayBuffer --disable-gpu-compositing --service-pipe-token=4245785956076169936 --lang=en-US --app-path=/workspace/arduino-editor/Arduino Pro IDE-v0.0.3-linux/resources/app --user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) arduino.
   3998    4025    3998    1419 pts/1       3998 Sl+  33333   0:00  |       \_ /workspace/arduino-editor/Arduino Pro IDE-v0.0.3-linux/arduino.pro.ide --type=gpu-process --enable-features=SharedArrayBuffer --no-sandbox --gpu-preferences=KAAAAAAAAACAAABAAQAAAAAAAAAAAGAAAAAAAAAAAAAIAAAAAAAAAAgAAAAAAAAA --service-request-channel-token=17899716977927950879
   3998    4029    3998    1419 pts/1       3998 Sl+  33333   0:02  |       \_ /workspace/arduino-editor/Arduino Pro IDE-v0.0.3-linux/arduino.pro.ide /workspace/arduino-editor/Arduino Pro IDE-v0.0.3-linux/resources/app/src-gen/backend/main
   4029    4076    3998    1419 pts/1       3998 S+   33333   0:00  |           \_ /bin/sh -c "/workspace/arduino-editor/Arduino Pro IDE-v0.0.3-linux/resources/app/node_modules/arduino-ide-extension/build/arduino-cli" daemon -v --log-level info --format json --log-format json
   4076    4077    3998    1419 pts/1       3998 Sl+  33333   0:01  |           |   \_ /workspace/arduino-editor/Arduino Pro IDE-v0.0.3-linux/resources/app/node_modules/arduino-ide-extension/build/arduino-cli daemon -v --log-level info --format json --log-format json
   4029    4080    4080    4080 ?             -1 Ssl  33333   0:00  |           \_ /workspace/arduino-editor/Arduino Pro IDE-v0.0.3-linux/arduino.pro.ide /workspace/arduino-editor/Arduino Pro IDE-v0.0.3-linux/resources/app/node_modules/arduino-ide-extension/lib/node/daemon-watcher.js 4029 4076
   4029    4180    3998    1419 pts/1       3998 Sl+  33333   0:00  |           \_ /workspace/arduino-editor/Arduino Pro IDE-v0.0.3-linux/arduino.pro.ide /workspace/arduino-editor/Arduino Pro IDE-v0.0.3-linux/resources/app/node_modules/@theia/core/lib/node/messaging/ipc-bootstrap.js
   4029    4246    3998    1419 pts/1       3998 Sl+  33333   0:00  |           \_ /workspace/arduino-editor/Arduino Pro IDE-v0.0.3-linux/resources/app/node_modules/arduino-ide-extension/build/arduino-language-server -clangd /usr/bin/clangd -cli /workspace/arduino-editor/Arduino Pro IDE-v0.0.3-linux/resources/app/node_modules/arduino-ide-extension/build/arduino-cli
   4246    4252    3998    1419 pts/1       3998 Sl+  33333   0:00  |               \_ /usr/bin/clangd

[jankeromnes]

Upon further investigation, it seems that at least some of these left-over processes are caused by Theia itself: https://github.com/eclipse-theia/theia/issues/6685

For example, the daemon-watcher.js process is waiting for the Theia process to disappear before killing the daemon and then itself. However, it stays alive indefinitely, because when stopped the Theia process itself becomes a zombie (i.e. it shuts down, but nobody listens to its exit code, so it stays around in "defunct" state "just in case"):

gitpod      1499  0.7  0.0      0     0 pts/0    Z    14:57   0:01 [node] <defunct>

[svenefftinge]

I understood that the original issue was saying there are clangd processes that are still running.
zombie processes are definitely not nice, but leaking actually running processes is much worse IMHO.

[jankeromnes]

@svenefftinge Theia itself becoming a zombie process causes other processes to remain alive and resource-consuming (see the daemon-watcher.js example).

I don't know for sure whether this can cause clangd to keep on running, but I do know that at least daemon-watcher.js keeps on polling ps forever because of this, which is not nice.

[jankeromnes]

Actually, I haven't been able to reproduce an active, resource-consuming clangd process yet. Only dead zombie clangd processes like:

[clangd.main] <defunct>

What remains active & running after IDE shutdown in my trials are these two processes:

arduino-ide-extension/lib/node/daemon-watcher.js
@theia/core/lib/node/messaging/ipc-bootstrap.js

If they're a significant resource drain, we may want to fix these. But if they're negligible & fixing is too complicated, maybe we can ignore them for now.

[cmaglie]

We had the same problem with arduino-cli, the problem arises when the parent process crashes unexpectedly and the subprocess continue to run.

A simple but very effective solution is to continuosly check stdin in the subprocess, and if stdin is closed then terminate the subprocess (because it means that the parent process is dead).

Here how we did it in go: https://github.com/arduino/arduino-cli/pull/488/files#diff-aab5babf75b671a8b8ccb9fcc6b09f46R77-R84

[jankeromnes]

Thanks for the great suggestion with stdin!

Relatedly, I was able to "fix" some left-over zombie processes in Theia: https://github.com/eclipse-theia/theia/pull/6714. I'm not sure whether my approach is good, but calling childProcess.kill() in process.on('exit'), process.on('SIGINT') and process.on('SIGTERM') seems to clean up the zombies. (I didn't try to observe stdin, because inputs and outputs were being piped/redirected in ways I didn't understand in Theia.)

To summarize the situation with Arduino Pro IDE, currently on master I get this process tree (with ps axjf):

 |   \_ node /home/gitpod/.nvm/versions/node/v10.16.3/bin/yarn --cwd ./browser-app start
*|       \_ /bin/sh -c theia start
 |           \_ /home/gitpod/.nvm/versions/node/v10.16.3/bin/node /workspace/arduino-editor/browser-app/node_modules/.bin/theia start
*|               \_ /home/gitpod/.nvm/versions/node/v10.16.3/bin/node /workspace/arduino-editor/browser-app/src-gen/backend/main.js
*|                   \_ /bin/sh -c "/workspace/arduino-editor/arduino-ide-extension/build/arduino-cli" daemon -v --log-level info --format json --log-format json
 |                   |   \_ /workspace/arduino-editor/arduino-ide-extension/build/arduino-cli daemon -v --log-level info --format json --log-format json
!|                   \_ /home/gitpod/.nvm/versions/node/v10.16.3/bin/node /workspace/arduino-editor/arduino-ide-extension/lib/node/daemon-watcher.js 5086 5111

I annotated with a * the processes that become zombie/defunct after the IDE quits, and ! the process that keeps on running (because it's checking whether the Theia process is still in ps, and it still is, although as a zombie).

I couldn't get clangd to run / become defunct this time, because the IDE kept loading forever for some reason (probably fixed in the 0.0.3 branch though).

However, we can see that, if my Theia PR gets merged, the main.js zombie process will get cleaned up, and thus the daemon-watcher.js process should exit normally. (But maybe we'll still want to make daemon-watcher.js observe Theia's stdin instead, as that's more robust, e.g. if Theia crashes unexpectedly.)

[jankeromnes]

Additional suggestions to tackle this problem:

Make the children read on a pipe from the parent, when they read eof, they kill themselves because the parent is gone.

That's more reliable than signals

(source)

---

try just not binding _any_ handlers, and use child_process.fork(..., {stdio: 'inherit'})

https://nodejs.org/api/process.html#process_event_disconnect

If the Node.js process is spawned with an IPC channel (see the Child Process and Cluster documentation), the 'disconnect' event will be emitted when the IPC channel is closed.

so in the child, process.on('disconnect', ...) should trigger when the parent process exits (or closes the ipc channel)
that seems like the cleanest approach, since it doesn't depend on code in the parent running

(source)

[jankeromnes]

Actually, doing this in a child doesn't help:

process.on('disconnect', () => process.exit(0));

The child _already_ exits successfully today, it's just that the parent process doesn't collect the exit code, thus the dead child is left behind as a zombie.

[jankeromnes]

I also tried this technique in the daemon-watcher.js, because that one does continue running after Theia dies (detached subprocess).

However, it didn't work (probably because 'disconnect' doesn't happen if you're already detached):

   PPID     PID    PGID     SID TTY        TPGID STAT   UID   TIME COMMAND
    869    5263    5263     869 pts/0       5263 Sl+  33333   0:00  |   \_ node /home/gitpod/.nvm/versions/node/v10.16.3/bin/yarn --cwd ./browser-app start
   5263    5285    5263     869 pts/0       5263 S+   33333   0:00 *|       \_ /bin/sh -c theia start
   5285    5290    5263     869 pts/0       5263 Sl+  33333   0:00  |           \_ /home/gitpod/.nvm/versions/node/v10.16.3/bin/node /workspace/arduino-editor/browser-app/node_modules/.bin/theia start
   5290    5297    5263     869 pts/0       5263 Sl+  33333   0:03 *|               \_ /home/gitpod/.nvm/versions/node/v10.16.3/bin/node /workspace/arduino-editor/browser-app/src-gen/backend/main.js
   5297    5329    5263     869 pts/0       5263 S+   33333   0:00 *|                   \_ /bin/sh -c "/workspace/arduino-editor/arduino-ide-extension/build/arduino-cli" daemon -v --log-level info --format json --log-format json
   5329    5330    5263     869 pts/0       5263 Sl+  33333   0:46  |                   |   \_ /workspace/arduino-editor/arduino-ide-extension/build/arduino-cli daemon -v --log-level info --format json --log-format json
   5297    5335    5335    5335 ?             -1 Ssl  33333   0:00 !|                   \_ /home/gitpod/.nvm/versions/node/v10.16.3/bin/node /workspace/arduino-editor/arduino-ide-extension/lib/node/daemon-watcher.js 5297 5329
   5297    5407    5263     869 pts/0       5263 Sl+  33333   0:00  |                   \_ /home/gitpod/.nvm/versions/node/v10.16.3/bin/node /workspace/arduino-editor/node_modules/@theia/core/lib/node/messaging/ipc-bootstrap.js
   5297    6066    5263     869 pts/0       5263 Sl+  33333   0:00  |                   \_ /workspace/arduino-editor/arduino-ide-extension/build/arduino-language-server -clangd /usr/bin/clangd -cli /workspace/arduino-editor/arduino-ide-extension/build/arduino-cli -fqbn arduino:samd:arduino_zero_edbg -board-name "Arduino Zero (Programming Port)"
   6066    6071    5263     869 pts/0       5263 Z+   33333   0:00 *|                       \_ [clangd.main] <defunct>

(legend: * ends up as a zombie after I kill the app, and ! continues running)

[cmaglie]

The child already exits successfully today, it's just that the parent process doesn't collect the exit code, thus the dead child is left behind as a zombie.

Why the parent doesn't collect the exit code?

[jankeromnes]

Why the parent doesn't collect the exit code?

I have really no idea. It spawns processes with Node.js' child_process.fork(), and then when you exit the parent process, all child processes also stop.

However, for a strange reason that I don't understand, all the dead child processes are re-parented to PID 1 (i.e. they're orphaned), and they remain listed in ps axjf as "defunct" (i.e. zombie), which I understand is a process that has exited but since nobody asked for its exit code it is kept in the process tree "just in case".

I would like to remove them all automatically, and the only way I've found to do that until now is to have the parent explicitly call childProcess.kill() for every child process just before exiting itself.

EDIT: I know that a potential alternative to prevent zombie child processes is maybe to explicitly ignore SIGCHLD (e.g. by using SIG_IGN or something like that?) but I don't fully understand this solution yet and also I think Node.js doesn't support explicitly ignoring signals.

[cmaglie]

I see that child_process.fork() has a different behaviour compared to child_process.spawn() becuase:

Unlike the fork(2) POSIX system call, child_process.fork() does not clone the current process.

Also for the spawn call the documentation provides an example that collects the return code.

const { spawn } = require('child_process');
const ls = spawn('ls', ['-lh', '/usr']);

ls.stdout.on('data', (data) => {
  console.log(`stdout: ${data}`);
});

ls.stderr.on('data', (data) => {
  console.error(`stderr: ${data}`);
});

ls.on('close', (code) => {
  console.log(`child process exited with code ${code}`);
});

I know it's very empirical, but maybe worth giving a try?

[kittaakos]

@F-K-F-F, does the issue still happen with the latest release? https://github.com/arduino/arduino-pro-ide/releases

I set up an Ubuntu 18.4 locally, started the Pro IDE, LS (clangd) has started and after closing the IDE, the clangd process has terminated. Thanks!

[kittaakos]

Any updates on this @F-K-F-F?

[kittaakos]

@F-K-F-F ping?

[kittaakos]

Please reopen if required.