Rescript-compiler: bsb -make-world -w exits early when backgrounded

[lpil]

I've been running bsb like so ./node_modules/.bin/bsb -make-world -w &.

This worked with the 2.* range, but now with the 3.0.0 release bsb exits after compiling instead of watching.

$ ./node_modules/.bin/bsb -make-world -w &
ninja: no work to do.
ninja: no work to do.
>>>> Start compiling
Rebuilding since just get started
ninja: Entering directory `lib/bs'
ninja: no work to do.
>>>> Finish compiling

Job 3, './node_modules/.bin/bsb -make-w…' has stopped

[bobzhang]

hmm, seems only one commit https://github.com/BuckleScript/bucklescript/commits/master/lib/bsb
I tested on vscode terminal, it works properly

[lpil]

I'm on Ubuntu 16.04 Linux.

I've tested with multiple terminals on bash and fish shells and the same problem occurs.

It doesn't seem to occur with sh shell though

[OvermindDL1]

@lpil and @bobzhang I do see multiple issues with 3.0!

(_/me should probably update from 1.6/7* sometime..._)

I tested in a variety of shells, results:

Both ZSH and Bash in redhat, it looks like something is really funky with the stream handling in bsb, specifically when it is backgrounded and then anything happens over stdin then it gets suspended, like it's not buffering data (wtf?)... When stdin is resumed then it works fine.

Bash in cygwin, it works fine, although it zombified when stdin closed, I now have a zombie bsb watcher running on the windows (what the heck?! BUG!!!).

For comparison I just tested, 1.7 is working fine on bash in cygwin on windows, and 1.7 is working fine on both zsh and bash on redhat, all when backgrounded and all. Something got royally borked on both linux and windows (in different ways) since 1.7... (EDIT: Nope, 1.7 works fine too)

[tcoopman]

From what I can find (https://superuser.com/a/973036 and https://github.com/phoenixframework/phoenix/issues/2215#issuecomment-341900651), detaching a process in background mode detaches stdin. So this would have been the behaviour for some time already.

Proposal: I'll submit a PR that adds a flag/config option to listen to stdin or not, so that you have the choise to run it in background (without the flag) and also use it on windows (because there listening to stdin is not enabled at the moment)

[bobzhang]

curious why do you need run watch mode in background?

[lpil]

Just a simple unix-y way of running all the build tools without faffing with multiple terminals or writing a program that will manage multiple subprocesses.

[OvermindDL1]

@tcoopman It detaches stdin, doesn't close it, and reattaches stdin when foregrounded again.

Still though @lpil, why not tmux?

[lpil]

Still though @lpil, why not tmux?

Because that's a lot more work than putting & at the end of the command and it assumes that tmux will be on everyone's machines.

[OvermindDL1]

Won't it output into your console though, mixing up whatever else you are doing anyway then?

[lpil]

With the exception of bsb everything running in the console only prints minimal information unless there is an error, so it's very readable.

[tcoopman]

Both ZSH and Bash in redhat, it looks like something is really funky with the stream handling in bsb, specifically when it is backgrounded and then anything happens over stdin then it gets suspended, like it's not buffering data (wtf?)... When stdin is resumed then it works fine.

I've just tried this out myself and it's really easy to reproduce (https://nodejs.org/dist/latest-v9.x/docs/api/process.html#process_signal_events)

#!/usr/bin/env node

const fs = require('fs');

process.stdin.resume();
fs.watch('to_watch', ()=> {console.log('changed')});

If you run this in background mode you get the same behaviour. Press enter and watch stops. Put it on the foreground and watch resumes.

Anyone with more node knowledge who can give a hint on how this can be improved?

For now I'm still on the opinion that a flag to enable this behaviour would be the best.

[OvermindDL1]

Flags are good, and who knows about node, maybe newer versions are acting weird, wouldn't surprise me, should probably be an OCaml native file instead. 😄

[tcoopman]

@bobzhang are you ok with a flag? (This is probably a bit more work because I'll need to add the flag and documentation to other scripts as well?)

[bobzhang]

@OvermindDL1 if you have a cross platform solution using OCaml, that would be perfect : )

I am not sure that running bsb -w in background is a common use case, given the added complexity, I am not sure it is worthwhile. The startup time of bsb.exe is very low, if you want to take 100% control, you can have your own watcher and call bsb.exe whenever relevant changes are spot

[lpil]

Is a flag required? This sounds like a bug to me as it is exiting when stdin is not closed.

[tcoopman]

@lpil see my earlier comment. This looks like a bug in node (submitted it here: https://github.com/nodejs/node/issues/20148)

@bobzhang the flag would still allow windows people to run it, now it's hard coded to linux/mac

[OvermindDL1]

I've never made a file watcher in OCaml but inotify is in the opam repository (would just need a cygwin build for windows, which most ocaml stuff needs anyway).

I think a flag override is fine personally, though I would opt to follow the unix standard by default and have the flag disable it (perhaps --daemon to daemonize it actually, that is also the 'unix-way').

I was curious about simple single-file no-op with the latest bsb, I got 0.085s, not bad for node at all I do think, though even notoriously slow MakeFiles for C++ (a project that was already compiled but had a few hundred files) took less than half that time (0.041s) and the same project (it's build with cmake so different generators are easy) already built so just having ninja try to compile an already built project took 0.008s, so node is not native-fast by any stretch, but it is 'fast enough'. ^.^;

[OvermindDL1]

They state:

That's working as expected. When you put the node process in the background and then feed its stdin, it gets sent a SIGTTIN signal that pauses it. When you fg it again, the shell sends it a SIGCONT that unpauses it.

Yeah, but that's not a close signal, so... why is the process pausing...

[bobzhang]

hi @OvermindDL1, I meant bsb.exe clearly not bsb. Do me a favor not to criticize node loudly in public, it is not cool, I can list 10 items why your favorite tech stack sucks which makes no sense. Also inotify is linux only stuff, we should really do at least some research before pointing figures

[OvermindDL1]

There are inotify shims for Windows though that run fine when compiled via cygwin. Not sure where I bemoaned node though? I did show it was pretty blazing fast, within a magnitude of C++, which is impressive (is what I was saying, compared to, oh, java, which takes a while just to setup the VM before it runs any code).

[MoOx]

Maybe related: https://github.com/lerna/lerna/issues/1745

I am trying to use lerna to execute multiple bsb -make-world w in parallel (using node spawn with stdio set to ignore stdin) and I am unable to...
How could I help on this issue?

[TomiS]

Any solutions in sight for this? I've also tried running bs in monorepo by using lerna, oao and yarn workspaces run without success. It always either builds only one module or no modules at all but never does it listen all modules and rebuild based on file changes. I don't want to sound dramatic but this really feels like a big blocker for people like me who consider adopting ReasonML into a large JS project with multiple sub modules/repositories when just having everything in the same folder is not really an option.

[MoOx]

My VERY DIRTY UGLY workaround is that I change lerna code on npm prepare hook... https://github.com/phenomic/phenomic/blob/5ab7337850b5359a707a69aa62acdd256c5f1831/package.json#L59-L62

We should really check if lerna can help here. Not sure who is the real culprit.

[J-Zeitler]

I ran into this as well (bs-platform 7.3.2) trying to use a docker container running bsb -make-world -w and parcel when developing. Managed to get it to work by redirecting /dev/zero into bsb.

working package.json configuration:

  "scripts": {
    "dev": "npm-run-all --parallel serve start",
    "start": "bsb -make-world -w < /dev/zero",
    "serve": "parcel serve -p 7357 --hmr-port 7358 index.html",
    "clean": "bsb -clean-world"
  }

and dockerfile:

CMD npm i && npm run dev