Bazel: Allow "bazel run" to release the lock after building but before launching

[dfabulich]

I'm filing an issue to capture the discussion in this thread. https://groups.google.com/d/msg/bazel-discuss/mS98nXz9PbY/WLV3zLST3SwJ

There's no built-in way in Bazel to bazel run two long-running processes, e.g. server processes. bazel run locks the Bazel server; the lock isn't released until the launched process is finished.

It's possible to work around this issue by using bazel run --script-path, which generates a shell script to run instead of executing the process. Then you can use it like this:

bazel run --script_path foo //foo && foo

There's a script for this workaround available in Bazel's source, https://github.com/bazelbuild/bazel/blob/master/scripts/bazel-run.sh

In the thread, I wished for something like bazel run --nolock that would execute the process without a lock.

If possible, provide a minimal example to reproduce the problem:

In this git repository https://github.com/dfabulich/bazel-long-running there are two long-running server processes.

bazel run :1 & bazel run :2

Actual: one process will run, and the other bazel process will wait for the other one to finish. "Another command (pid=53060) is running. Waiting for it to complete..."

Environment info

• Operating System:
  Tested on macOS Sierra 10.12.2 and Ubuntu 16.04.1

• Bazel version (output of bazel info release):

release 0.4.3-homebrew

[katre]

The problem is that there isn't a lock per se, it's built into how Bazel operates. It'd be far simpler to update IntelliJ to use --script_path than to have Bazel recognize which situations are safe to allow to commands to run, without a fundamental change to how Bazel is written.

This is effectively a dupe of #532 so I will close it and link you there.

[dfabulich]

I don't think I agree.

532 is a very ambitious bug. It proposes attempting to identify when/how two builds could be run simultaneously, subtly teasing apart which parts of the build can run safely in parallel. (Perhaps just the analysis phase? And if the outputs are guaranteed not to overlap…?)

To fix #2337, I'm imagining that bazel run would always use --script_path; the client process would just ReleaseLock in blaze.cc when the build is done, and then invoke the --script_path script itself. I guess it's a bit hacky, but it's no hackier than running this bash script https://github.com/bazelbuild/bazel/blob/master/scripts/bazel-run.sh and IMO it wouldn't require a significant change in architecture.

Fixing the bug in the IntelliJ plugin might actually pretty tricky. IntelliJ's API says, "OK, it's time to Run! Go execute your process now." Implicitly, it's allowing us to run just one process; there may be a clever way to get it to launch a pre-run build and then launch a second process for debugging, and I'll investigate that, but it's not obviously a slam dunk. (I know almost nothing about Eclipse's API, but it sounds like we'd have a similar problem to fix there.)

So fixing it in the IDE may require some extremely hairy integration code, per IDE. (Will there be other IDEs/runtime environments to support in the future??) And at the end of all that, the bug in the CLI still won't be fixed.

Am I approaching this problem incorrectly? If I put together a PR for the "hacky" fix I'm proposing, is that something that the team would be willing to merge in?

[katre]

I'll be happy to review a PR for your proposal.

I agree that being able to run two targets at once (or even the same target twice) would be a good thing, I'm just worried about how deep the changes will need to get. But please feel free to take a look and try your idea and I'll be happy to take a look at what you come up with.

[ulfjack]

This is a long-standing request that we also have internally. Internally, we currently advise people to use the shell script workaround, but I for one think that's not ideal and Bazel should be doing it natively.

I agree that this needs to involve the client, for several reasons. I think the client-server protocol reply should be extended to let the client know which path to execute. Also, for perfect fidelity, the client should replace itself and make sure the new process is connected to the right stdin / stdout to get proper access to the terminal (note that this would not be possible in the server). If we can get this to work, it should become the default for bazel run.

[dfabulich]

While working on this, I started by hacking up another shell-based workaround, similar to bazel-run.sh, that automatically applies --script_path and execs the script if and only if the user has not supplied a --script_path.

If you install the following shell script in tools/bazel and mark it executable, bazel will run it instead of running bazel directly, which allows you to do useful tricks like this.

#!/bin/bash

has_script_path=false

i=0
for arg in "$@"
do
    i=$((i+1))
    case "$arg" in
        --) : ${split_pos:=$i} ;;
        run) is_run=true ;;
        --script_path) has_script_path=true ;;
    esac
done

if [ -n "${split_pos:-}" ]
then
    extra_args=( "${@:$split_pos:$#}" )
    set -- "${@:1:$split_pos-1}"
fi

if [ "$is_run" = true -a "$has_script_path" = false ]
then
    runcmd="$(mktemp /tmp/bazel-run.XXXXXX)"
    set -- "$@" --script_path="$runcmd"
fi

test -n "${split_pos:-}" && set -- "$@" "${extra_args[@]}"

"$BAZEL_REAL" "$@"

if [ "$is_run" = true -a "$has_script_path" = false ]
then
    exec "$runcmd"
fi

[dfabulich]

Since the Bazel plugin for IntelliJ itself runs the bazel CLI tool, installing a tools/bazel script causes the IntelliJ plugin to run this workaround as well. That means this script is also a workaround for the plugin bug https://github.com/bazelbuild/intellij/issues/34

[stefan-improbable]

Sometimes I'm using this snippet to debug flaky go tests:

for i in $(seq 1 100); do
  for j in $(seq 1 10); do
    go test &
  done
  wait
done

it would be nice if I could do the same with bazel.

[ulfjack]

With Bazel, you can use this snippet:

bazel test --runs_per_test=1000 //my/test

[benjaminp]

I think this is completely fixed now? Actual execution of the target now happens in the client and doesn't hold the server lock.

[kerrick-lyft]

@ulfjack can you comment on why this was closed? Is it fixed?

[ulfjack]

Yes, it's fixed like @benjaminp said above (the second sentence is a description of the current state).

[asad-awadia]

@ulfjack

Is it on a specific bazel version?

▶ bazel version
Build label: 0.24.1
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Apr 2 16:32:47 2019 (1554222767)
Build timestamp: 1554222767
Build timestamp as int: 1554222767

I still can't bazel run multiple targets and bazel-run.sh also does nothing differently

[benjaminp]

You'll have to be more specific about what's not working If you want to pass multiple targets to "bazel run" at a time, that's not what this issue was a about.

On Mon, Oct 21, 2019, at 10:33, Asad Awadia wrote:

@ulfjack https://github.com/ulfjack

Is it on a specific bazel version?

▶ bazel version Build label: 0.24.1 Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar Build time: Tue Apr 2 16:32:47 2019 (1554222767) Build timestamp: 1554222767 Build timestamp as int: 1554222767
I still can't bazel run multiple targets and bazel-run.sh also does
nothing differently

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/2337?email_source=notifications&email_token=AABVSTVUQHYVSRRDFLQP22DQPXRVHA5CNFSM4C3L6MSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEB3ER5A#issuecomment-544622836, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABVSTWEWAZSJLIVUHPARDTQPXRVHANCNFSM4C3L6MSA.