Node: Debugger doesn't work for processes which fork other processes

[medikoo]

I'm running:

$ node --inspect some-node-script.js

Where some-node-script.js uses plain child_process.fork (run with defaults mostly) calls to initialize few other processes internally. Right after that I receive message _Unable to open devtools socket: address already in use_:

$ node -v
v7.0.0
$ node --inspect some-node-script.js
Debugger listening on port 9229.
Warning: This is an experimental feature and could change at any time.
To start debugging, open the following URL in Chrome:
    chrome-devtools://devtools/remote/serve_file/@60cd6e859b9f557d2312f5bf532f6aec5f284980/inspector.html?experiments=true&v8only=true&ws=localhost:9229/edbce9e9-0a9d-4c24-8f2b-bcaaeb4a5965
Unable to open devtools socket: address already in use

Also forked process crashes so technically application doesn't run (I've skipped that part of a log to avoid not related noise).

Behavior is same in both latest Node.js v7 and v6 (Tested on OSX, both El Captain and Sierra, with latest Chrome on board)

Am I doing something wrong, or there's no support currently for multi-process Node.js apps?

I've found similar issue which states that this probably should just work, but gives no clue why it actually doesn't.

I'll be happy to provide simple test case if needed

[mscdex]

/cc @nodejs/v8-inspector

[bnoordhuis]

I've found similar issue which states that this probably should just work, but gives no clue why it actually doesn't.

child_process.fork() doesn't fix up execArgv, only cluster.fork() does. See https://github.com/nodejs/node/issues/8495#issuecomment-246600965.

[medikoo]

child_process.fork() doesn't fix up execArgv, only cluster.fork() does. See #8495 (comment).

Thanks, that clearer now. Is there a plan to fix it for child_process.fork?

[medikoo]

After reading this comment I see that probably #5025 is meant to fix this issue

[bmeck]

with v8 landing RemoteExecutionContext, it should be possible for a debugger to work for multiple processes. I don't know if we are ready for that and it requires v8 version >=55 which isn't going to land on Node v6

[eugeneo]

V8 inspector protocol has support for the "subtargets" - we are currently
looking into leveraging that for the node processes that fork children.
https://github.com/nodejs/node/pull/9630 updates the HTTP transport for the
inspector, now it should be able to track multiple "sessions" and also list
multiple targets.

Currently we are considering introducing a special "inspector" pipe from
the parent to a child so the child does not have to run the HTTP server.
All communications with the children will happen through the parent, parent
will also be able to list individual children.

Note that this work is in experimental stage and the design will likely
change a lot.

On Thu, Nov 17, 2016 at 1:21 PM Bradley Meck [email protected]
wrote:

with v8 landing RemoteExecutionContext, it should be possible for a
debugger to work for multiple processes. I don't know if we are ready for
that and it requires v8 version >=55 which isn't going to land on Node v6

—
You are receiving this because you are on a team that was mentioned.

Reply to this email directly, view it on GitHub
https://github.com/nodejs/node/issues/9435#issuecomment-261373137, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AARkrb9lYhslDNRD2jh8KnSe5jJ03pspks5q_MVegaJpZM4KoFlU
.

[Trott]

@eugeneo Should this remain open?

[eugeneo]

I believe it can be closed.

[medikoo]

Does it mean, there's no plan to fix it?

Multi-process Node.js configurations are not that uncommon, and currently debugger functionality is not usable having such setup. I see it as quite important issue.

[Trott]

/ping @bnoordhuis Is the reasoning in https://github.com/nodejs/node/issues/8495#issuecomment-246600965 basically an explanation of why the behavior observed here is the way it is, and why it is unlikely to be changed?

@medikoo Does it work in Node.js 8.x if you invoke the script like this?:

$ node --inspect --inspect-port=0 some-node-script.js

[bnoordhuis]

I'm not mordicus against changing child_process.fork() to be more like cluster.fork(), symmetry is good, but I do think there is a non-zero change it will break third-party code.

[medikoo]

Any chance to have that reopened?

[refack]

@medikoo AFAIK GitHub should let you reopen issues you submitted, but in anyway, @eugeneo has set a goal to implement multiplexing multiple inspector "channels" into a single socket connection. This will allow subprocesses to be "inspected" through their parent's socket.

Also as mentioned before, in [email protected] it's possible to use arbitrary port binding for inspector with:
node --inspect=0 some-node-script.js
you only need to watch the output messages for information on which ports were allocated.
You could also pass an explicit port in execArgv for cp.fork(), similar to what cluster.fork() does:
https://github.com/nodejs/node/blob/bb294059040def8e8c0b1719cc17f6537ab5cb39/lib/internal/cluster/master.js#L119-L122

[medikoo]

@medikoo AFAIK GitHub should let you reopen issues you submitted

Only if it's me that closed it. Here it's not the case, I'm not able to reopen it.

[Freundschaft]

Im having this issue now using a library that forks processes, where I have no influence on how the child processes are called. As soon as I use debugging mode, it screws up. Is there an open issue somewhere if we cant reopen this one?

[Trott]

Seems like this ought to be re-opened, at least until theres either a third-party solution we can point to or else there's a solid decision that this feature will never be implemented.

Feel free to close again if you think I'm wrong about that.

[bnoordhuis]

One reason we can't just change this, or at least not without special-casing is that, courtesy of the execPath option, there exists code out there that does this:

const args = [ /* ... */ ];
const conf = { execPath: 'python', execArgv: args, stdio: ['pipe', 'ipc'] };
const proc = child_process.fork('script.py', conf);
proc.on('message', handleMessage);
proc.stdin.write(data);

It doesn't actually spawn a node process and hence we cannot just blindly rewrite the arguments because --debug and --inspect are not unique to node.

When should rewrite rules be applied? When options.execArgv !== process.execArgv? Or when options.execPath !== process.execPath? Both have drawbacks:

1. options.execArgv is often just process.execArgv with one or two additional options.
2. options.execPath can be a shell script that does some housekeeping before starting node.

You want rewriting in such cases.

One solution is to add an opt-in flag but that's ugly and doesn't help when you don't control the call site¹. A command line flag or environment variable is too horrible to contemplate.

¹ Although that's more of an issue with poorly designed third-party code.

[DavidSouther]

This is probably worth its own thread, but I'm also seeing this when Node is a child process of a non-node process, specifically protoc.

[medikoo]

@bnoordhuis I think the real problem here, is not specifically that forked processes are not automatically inspectable (although having such possibility would be nice), but that master process cannot be inspected if it forks some processes as they crash in such scenario with failed: address already in use

Is there any chance to ensure that it doesn't behave that way (so e.g. on inspect. we have mean to inspect just master process, and leave forked children as not inspected?). That would be a good start.

[bnoordhuis]

@medikoo What do you propose? The best I can come up with is to strip debugger options iff a bunch of conditions are met: options.execPath not set, options.execArgv not set or contains exact copy of debug flags in process.execArgv, and so on.

Pull requests welcome, of course.

[refack]

In node>=8 there is the NODE_OPTIONS environment variable that allows enabling inspector (with explicit port number or ephemeral port 0) in all child processes without having to manipulate the command line directly.

[Glennvd]

At the moment there is no workaround for this? Setting NODE_OPTIONS env variables on the fork() doesn't seem to do anything.

[ilijaz]

Have the same issue with --inspect-brk parameter. Set execArgv to an empty array and it forks.

const options = {
  execArgv: []
};
const child = Child.fork(program, options);

[FedericoBiccheddu]

Hi to all, there is a workaround for this?

Working locally with nodemon (and docker just fyi) debugging is not possible, even using NODE_OPTIONS.

Screenshot example

[bmeck]

There is a terrible workaround I use:

> NODE_OPTIONS=`--inspect-port=0`

And then use the terminal output to put into chrome:inspect manually.

[FedericoBiccheddu]

Hi @bmeck, thank you for the response.

An ephemeral port is not portable and configurable so, unfortunately, is not a solution because I should update the port in the IDE's config everytime.

PS: I don't think Chrome is suitable for a server side project debugging session 😞

[vith]

@FedericoBiccheddu in a pinch you should be able to use SSH port forwarding to connect your local chrome to a remote (localhost) debug port. I believe with openssh it's -L <local_port:remote_addr:remote_port> where remote_addr is from the perspective of the ssh server you're connecting to. So 127.0.0.1 as remote_addr can forward the connection to the remote server's localhost ports.

[FedericoBiccheddu]

Hi @vith, thank you for the response.

As I said in the previous comment unfortunately is not a solution. Being more specific:

1. Some team members use only Firefox
2. It's a server side application (I don't think Chrome is the right tool to debug a server side application)
3. The debugger is integrated in the IDE (WebStorm or VSCode)
4. Using docker where the app has multiple services (not docker-machine)
5. Front-end and back-end team should work with the same code base; we need to have a portable and maintanable solution
6. openssh is a third-party program

I think the workaround should be limited on the node or docker side.

[sergiybuch]

"terrible" workaround is not helping me. I am trying to debug Protractor tests, using Webstorm.

[bmeck]

I've opened https://groups.google.com/forum/#!forum/google-chrome-developer-tools to start discussion on things, but all these non-chrome approaches such as IDEs would need to adopt any solution that would be agreed upon if the feature is seen as safe/valuable enough to implement.

[eason-shsf]

this problem still exists on node 8.11.3, is there a plan for fixing this?

[CFKevinRef]

@bmeck this is the link to the actual post, right?

Chrome DevTools › [for Node] Expand chrome:inspect discovery for debuggable targets

[bmeck]

@CFKevinRef yes, but it looks like Google is working on https://github.com/GoogleChromeLabs/ndb (recently made public) which should supercede that

[CADBOT]

Even if it works in ndb, it would be great to have in the command line debugger. I'm a huge fan of it as it involves less context switching.

[trusktr]

I'm sure once it's fleshed out it will find its way in. But prepending ndb to any command is so easy!👍

[mdrichardson]

@ilijaz Thanks!

const options = {
  execArgv: []
};
const child = Child.fork(program, options);

Although not an ideal fix, this was a great workaround.