Bazel: Bazel does not terminate subprocesses on ^C on windows

[Artem-B]

ATTENTION! Please read and follow:

• if this is a _question_ about how to build / test / query / deploy using Bazel, or a _discussion starter_, send it to [email protected]
• if this is a _bug_ or _feature request_, fill the form below as best as you can.

Description of the problem / feature request:

Bazel does not terminate subprocesses on ^C on windows.
When interrupted with ^C the already launched commands continue to run.
If subsequent execution of bazel needs to restart the server, it will silently hang until the old commands have finished.
If one of the old commands is manually killed, then all remaining commands appear to be killed/cleaned up right away.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

BUILD:
genrule(
  name = "long",
  outs = ["foo"],
  cmd = "while true; do sleep 5; echo tick >> $@; done;"
)

Reproduction:

b2e80172 /tmpfs/repro > /tmpfs/bazel200.exe build long -s
INFO: Invocation ID: 042e0282-409d-4ff7-92ee-1629cae57a45
Loading: 
Loading: 0 packages loaded
Analyzing: target //:long (1 packages loaded, 0 targets configured)
INFO: Analyzed target //:long (1 packages loaded, 1 target configured).
INFO: Found 1 target...
[0 / 2] [Prepa] Executing genrule //:long
SUBCOMMAND: # //:long [action 'Executing genrule //:long', configuration: 0c76e4e27246f1cc8c360b2c11de5f1aafe2797237101c507bc86c53960ac843]
cd C:/tools/msys64/home/kbuilder/_bazel_kbuilder/iqatem7v/execroot/__main__
  SET PATH=[...]\
    SET RUNFILES_MANIFEST_ONLY=1
  C:/tools/msys64/usr/bin/bash.exe -c source external/bazel_tools/tools/genrule/genrule-setup.sh; while true; do sleep 5; echo tick >> bazel-out/x64_windows-fastbuild/bin/foo; done;
[1 / 2] Executing genrule //:long; 1s local, remote-cache

Bazel is interrupted here and we're back on the console.
The action continues to run in the background and keeps appending lines to foo:

b2e80172 /tmpfs/repro > ls -l bazel-out/x64_windows-fastbuild/bin/foo
-rw-r--r-- 1 kbuilder None 10 Jan 13 20:14 bazel-out/x64_windows-fastbuild/bin/foo
b2e80172 /tmpfs/repro > wc -l bazel-out/x64_windows-fastbuild/bin/foo
3 bazel-out/x64_windows-fastbuild/bin/foo
b2e80172 /tmpfs/repro > wc -l bazel-out/x64_windows-fastbuild/bin/foo
4 bazel-out/x64_windows-fastbuild/bin/foo
b2e80172 /tmpfs/repro > wc -l bazel-out/x64_windows-fastbuild/bin/foo
5 bazel-out/x64_windows-fastbuild/bin/foo

What operating system are you running Bazel on?

Windows Server 1809

What's the output of bazel info release?

release 2.0.0

Have you found anything relevant by searching the web?

No. Discussed w/ @meteorcloudy who suggested filing a bug.

[meteorcloudy]

/cc @laszlocsomor

[laszlocsomor]

I can repro when running Bazel from MSYS, but not when running from cmd.exe
@Artem-B , are you running under MSYS or WSL, or are you running it from cmd.exe?

[Artem-B]

bazel is launched from an msys2 shell, though the shell itself is launched from a nested combination of msys2 bash scripts & .bat files executed by cmd /c.

Are you saying that things should work better if I do cmd /c bazel build long? I've just tried that and things appear to be worse. On ^C cmd itself does terminate, but bazel continues to be running and so does the genrule loop it launched.

[laszlocsomor]

I mean if you run bazel build long from an interactive cmd shell, then ^C works as expected.

Thanks for the clarification, I'll investigate more.

[laszlocsomor]

This seems to be an ancient MSYS2/MingW bug. Apparently they terminate programs on Ctrl+C instead of allowing them to handle the signal: https://stackoverflow.com/a/52657873/7778502

Open ~bugs~ threads from 2016 and 2003 (!):

• https://sourceforge.net/p/msys2/mailman/message/35533019
• https://sourceforge.net/p/mingw/mailman/message/15221148

This is why Ctrl+C will terminate the client and its children, but not the server nor the server's children (which bash and sleep are).

One workaround people suggest is to run Bazel under winpty: https://stackoverflow.com/a/48323689/7778502 . (Never mind that it talks about Git Bash, because that is actually MSYS under the cover.) I never tried this program though.

Another workaround is to use --batch mode, because then the server is a child of the client and terminating the client will also kill the server... but this is abrupt termination, so the processes may leave stale state behind. (Also, --batch mode means the client starts a new server, so incremental builds are slower.)

Or run Bazel from cmd.exe, everything works nicely there, even in --nobatch mode.

[laszlocsomor]

Summary: we can't fix this bug.
I will update the website not to recommend running under MSYS and also link to this bug.

[konste]

Is it known if the bug applies to both MSYS and WSL? If WSL is also affected is it Ok to have it installed as long as the actual build is run from CMD.EXE? Because it is really hard to design Windows build without relying on WSL.

[meteorcloudy]

@konste We never tested Bazel on Windows with WSL. Are you using Bazel with WSL? If so, can you try to reproduce this issue in WSL?

[tdm20]

I have WSL (Ubuntu) and MSYS2 installed.
Using an MSYS2 bash exhibits the behavior in this bug (Ctrl-C does not kill the server, I had to kill java.exe).

Using windows bash works fine (Ctrl-C stops the bazel server correctly)
Using WSL Ubuntu bash works fine (ditto).

I am using Bazel 2.1.0 on Windows 10

[meteorcloudy]

@tdm20 That's great to hear! Thank you!