Libuv: fork() in uv_spawn tries to reserve as much memory as parent

[zbjornson]

• Version: at least 1.23.2 through 1.24.1 (found in Node 8.12.0)
• Platform: Linux 4.15.0-1026 (Ubuntu 16.04)

(Moved/translated from https://github.com/nodejs/node/issues/25382.)

A tiny uv_spawn call from a large process attempts to reserve as much memory as the large process. For example, when Node.js is using at least half of the otherwise-available memory, exec() causes spawn ENOMEM:

// If your system has more than 4 GB of mem, repeat these two lines until >50% of memory is used:
let x = Buffer.allocUnsafe(2e9);
x.fill(2); // virtual -> reserved
// Causes ENOMEM even if there's >1 GB of available memory:
require("child_process").exec("pwd", console.log)

# (strace)
[pid  3017] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7ffae3ea7a10) = -1 ENOMEM (Cannot allocate memory)

Apparently this is a common problem when using fork()/clone(), and using posix_spawn(3) avoids it.

• Good write up of the issue: https://www.oracle.com/technetwork/server-storage/solaris10/subprocess-136439.html
• Related glibc bug discussion from JDK devs working on same issue https://sourceware.org/bugzilla/show_bug.cgi?id=10311
• JDK bug that goes with above https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6850720

This could be the underlying cause of this very popular SO issue for some users: https://stackoverflow.com/questions/26193654/node-js-catch-enomem-error-thrown-after-spawn

[zbjornson]

Using posix_spawn seems unlikely to avoid it on Ubuntu

What makes you say that? Edit I misread part of the Solaris article, I see posix_spawn() still uses fork() on Linux. There's the gnu-specific posix_spawn(3) with POSIX_SPAWN_USEVFORK but that might have the same issues as using vfork directly

Isn't vfork a non-starter for the reasons listed in #1715 and #141? Likewise, clone(2) with CLONE_VM has similar implications (cannot use any of libc from the child). My understanding is posix_spawn(3) is the replacement for vfork() that doesn't have vfork's issues.

[bnoordhuis]

Likewise, clone(2) with CLONE_VM has similar implications (cannot use any of libc from the child).

That's surmountable. Libuv can make direct system calls.

(We currently call glibc's syscall() but if that turns out to be problematic we can switch to syscall/int $80/swi/etc. instead.)

The problem with pthread_atfork() handlers remains - they won't run - so it'll need to be opt-in. I'm undecided on whether it makes sense for Node.js to turn it on, now or in the future, but I'm leaning towards 'no'. That would make it less useful, of course.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[vtjnash]

This is in active use over at JuliaLang, to address problems similar to nodejs/node#25382 and give a significant performance increase. (as I promised above, someday I plan to work on upstreaming that work.)

[saghul]

Is Julia using vfork instead? Don't the darwbacks outlined in #141 apply?

[zbjornson]

FWIW I've started (slowly) working on a Linux kernel patch that would make it so MADV_DONTFORK'ed memory isn't reserved when forking. I have no idea yet if it's viable, but it might be an easier solution since I think all of the other options have major caveats. That would help with v8 pages and Node's arraybuffer allocations, but native addons would have to madvise if they make large allocations also.

[vtjnash]

Yes, we use vfork (on linux only currently). If you read the standard closely, you may notice that most of the drawbacks outlined in #141 are also applicable to fork anyways, so I don't think they aren't really a reason to avoid it:

If a multi-threaded process calls fork(), ..., the child process may only execute async-signal-safe operations until such time as one of the exec functions is called

It sounds like there's the additional restriction on nodejs that pthread_atfork code won't get executed. For us, that's a feature (we avoid running extraneous code just before throwing away the whole process), but I can understand that nodejs may be stuck with it. I figure we'd keep this behind a flag for that reason.

[saghul]

Yeah, having the opt-in flag would be a good way forward. In fact, that way, it could land in v1.x I think. Feel free to send the PR and I think most of the concerns can be fixed with documentation, since this is opt-in.