Bazel: py_binary doesn't automatically pick a Python runtime matching the requested version

[emcfarlane]

Description of the problem / feature request:

Running py_test() and py_binary() rules under bazel doesn't select python3 when only srcs_version="PY3", default_python_version="PY3" attributes are set on the rules. How do you specify python2/3 builds?

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I wrote an integration test: https://github.com/afking/bazel/commit/5a6feef7953d2fe664e7a3444a36c9a875416913

bazel.log

What operating system are you running Bazel on?

Mac

What's the output of bazel info release?

release 0.10.0-homebrew

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

[email protected]:afking/bazel.git
d3e102c9fb62f4416099dc9f097a5f207aa7d442
5a6feef7953d2fe664e7a3444a36c9a875416913

Have you found anything relevant by searching the web?

https://github.com/bazelbuild/bazel/issues/3871
https://github.com/bazelbuild/bazel/issues/3517
https://github.com/bazelbuild/rules_python/issues/62

[buchgr]

@lberki can you take this please?

[brandjon]

A proper solution would probably involve allowing py_runtime to be used to specify both py2 and py3 interpreters, and allowing individual py_binary rules to declare their runtimes (probably by exposing the hidden py_interpreter attribute).

But in the meantime, a quicker and simpler solution would probably be to make the default behavior (where py_interpreter / --python_top is not given) simply choose shell commands "python2" or "python3" depending on the python mode.

[brandjon]

The proper fix for this bug is to implement py_toolchain [edit: now py_runtime_pair] and to create a default toolchain that sensibly locates the system's installed Python 2 and Python 3 runtimes.

Until that's implemented, the workaround is to dispatch to the system interpreters in a py_runtime rule using select(), and point to that py_runtime in --python_top. The best way to do this from 0.23 onward is as follows:

py_runtime(
    name = "myruntime",
    interpreter_path = select({
        # Update paths as appropriate for your system.
        "@bazel_tools//tools/python:PY2": "/usr/bin/python2",
        "@bazel_tools//tools/python:PY3": "/usr/bin/python3",
    }),
    files = [],
)

Then run your build with the flag --python_top=//path/to/the:myruntime.

[Toxicable]

I could be totally missing something about how Python works here, so feel free to correct me.
But is there a reason why we're relying on the system having python installed at all?
Couldn't you just have a python binary downloaded similar to how rules_nodejs does it.
https://github.com/bazelbuild/rules_nodejs#installation-with-a-manually-specified-version-of-nodejs-and-yarn
This will make it much easier to use the python rules across platforms or even the same systems
Also allowing you to specific exactly what python version is being used by the repository rather than just going with whatever the system has installed

[brandjon]

At the moment we don't bundle a specific standard Python interpreter with Bazel. We would need an interpreter for each platform that Bazel supports, and we'd have to worry about coordinating upgrades to this version.

We don't tend to bundle tools with Bazel and its rule sets in other languages either, e.g. there's no gcc included in Bazel.

However, as a user you can include a Python interpreter in your repo and reference it via a py_runtime rule. Then you'd pass --python_top=//path/to/that:py_runtime to make the Python rules all use that interpreter instead of the system default. (This mechanism will soon change when toolchains are implemented, so that different runtimes can be used for different platforms.) In this way, you can have your py_binary target executing using a vendored (built-from-source or checked-in) Python interpreter, rather than the system one. However, a system interpreter is still needed to execute the Python stub script that launches your py_binary target.

Having a workspace rule fetch a Python interpreter sounds like a reasonable feature off the top of my head. Created #7467.

[Toxicable]

Ah yes, I didn't mean that bazel should ship an interpreter itself.
But instead as you mentioned; a way to fetch it.

Your plan for fetching it should work perfect for us, we have a lot of systems that don't necessarily have Python on them and would prefer avoiding any transient issues due to version skews.

[depthwise]

This is especially bad when building Python programs (or Subpar *.par's) for e.g. RaspberryPi. You build and copy your program, and then it barfs because it uses Python2 instead of 3 even though everything in the build tells it to use Python3.

[brandjon]

A solution to this is implemented in Bazel 0.25 if --incompatible_use_python_toolchains is enabled. Windows support is tracked in #7844.

[brandjon]

I'm downgrading this to P2 because with #7899 closed, this problem only affects Windows (tracked in #7844).