Bazel: Bazel's 2to3 fails with no messages

[yaroslavvb]

I'm trying to debug failure here.

It seems bazel fails at 2to3 on this file and also on this one. The 2to3 command returns Status 1, but doesn't print anything.

bazel-out/host/bin/external/bazel_tools/tools/python/2to3 --no-diffs --nobackups --write --output-dir bazel-out/local_linux-py3-opt/genfiles/python3/tensorflow/contrib/immediate/python/immediate/extra_tests --write-unchanged-files tensorflow/contrib/immediate/python/immediate/extra_tests/math_ops_test.py

Here are the full steps to reproduce, any suggestion how to debug?

[damienmg]

2to3 is currently a dummy wrapper script. We should connect to python 2 to 3 transpiler. /cc @lberki

[yaroslavvb]

The work-around is to use six to make it cross-compatible, and mark target with srcs_version = "PY2AND3"

[brandjon]

The Bazel Python rules attempt to invoke 2to3 when they encounter a target marked with srcs_version = "PY2" that is to be built in PY3 mode. If you use srcs_version = "PY2ONLY" it'll instead fail-fast, and if you use srcs_version = "PY2AND3" it'll leave the source untransformed and use it directly.

The bug here is that the rules want 2to3 to exist, but 2to3 was never open sourced and is just a stub. The two options are to implement support for 2to3 or remove the attempted conversion altogether.

We'll go with removing 2to3 "support" for three reasons:

1. Our internal Python team advised us that the way of the future is to write source code that lies in the intersection of Python 2 and Python 3, rather than source transformation tools that run at build time.

2. The Bazel community didn't seem opposed to deprioritizing 2to3 support.

3. Given that Python source files are visible in runfiles (so they can be imported), and that a 2to3-transformed file has the same name as the source file, this leads to runfile conflicts for targets that depend on the same file in both PY2 and PY3 modes. (It's not a conflict when 2to3 isn't used because the file contents for both modes are the same.)

[chrismcbride]

Im a little confused as to how I should be setting the srcs_version, the bazel error led me to this issue but Im happy to open a new one if this is not the right spot for this conversation.

Im using bazel 0.28.1 and have a BUILD file like so:

py_library(
    name = "lib",
    srcs = [
        "lib.py",
    ],
    srcs_version = "PY2",
)

py_test(
    name = "test",
    srcs = ["test.py"],
    python_version = "PY2",
    srcs_version = "PY2",
    deps = [":lib"],
)

bazel test :test works and correctly uses python2, however if I do bazel build :lib (or bazel test ...) I get the following error:

ERROR: /home/mcbride/src/py2only/BUILD:1:1: Converting to Python 3: lib.py failed (Exit 1) 2to3.sh failed: error executing command bazel-out/host/bin/external/bazel_tools/tools/python/2to3.sh --no-diffs --nobackups --write --output-dir bazel-out/k8-fastbuild/bin --write-unchanged-files lib.py

Use --sandbox_debug to see verbose messages from the sandbox
ERROR: 2to3 is not implemented.
Please either build this target for PY2 or else set srcs_version to PY2ONLY instead of PY2.
See https://github.com/bazelbuild/bazel/issues/1393#issuecomment-431110617
Target //:lib failed to build

Setting srcs_version of the py_library to PY2ONLY fixes this issue, but according to the python_rules docs (https://docs.bazel.build/versions/master/be/python.html#py_library):

The values "PY2ONLY" and "PY3ONLY" are also allowed for historic reasons, but they are essentially the same as "PY2" and "PY3" and should be avoided.

This seems to not be the case, is there any way to get bazel test ... to work in a non-deprecated way?

EDIT:

Heres a repo to reproduce: https://github.com/chrismcbride/py2-bazel-bug

If I clone that and run bazel test ... I get the above error

[brandjon]

The badness of the PY2 vs PY2ONLY naming aside, you can avoid the error by building the :lib target for PY2. This happens automatically when you build :test because it has a python_version = "PY2" attribute, but py_library has no such attribute, so you have to set it with a command line flag:

bazel build :lib --python_version=PY2

[chrismcbride]

But I have a repo with mixed PY2 and PY3 targets, and we currently run tests by doing bazel test ... at the root. I assume that wouldnt be possible by using the --python_version flag?

[brandjon]

bazel test ... should filter out non-test targets so py_library shouldn't give you any issues there. For bazel build ... you'd want to change to srcs_version = "PY2ONLY" until the stub 2to3 is removed.

[chrismcbride]

bazel test ... is giving me that same error on that sample repo I posted (https://github.com/chrismcbride/py2-bazel-bug), it appears to be building the py_library targets too

[brandjon]

So it appears. I guess the filtering I'm thinking of applies only to running the tests, not to building the targets.

[limdor]

@brandjon I also came across this issue, the biggest problem that I see is that from the documentation it looks like PY2 and PY2ONLY are the same but that is not really the case
https://docs.bazel.build/versions/master/be/python.html#py_binary.srcs_version
If making them to behave exactly the same is not an option, at least updating the docs would be really good

I tried with bazel 2.2.0, if in latest bazel there is no difference then forget what I said

[wolfd]

If I don't force any python version on the command line, and use srcs_version = "PY2", bazel build //... tries to "build" py_librarys with the wrong Python version, erroring out with the 2to3 message.

If I instead use srcs_version = "PY2ONLY", this issue goes away. Am I missing something?

The py_binary calls eventually specify the python interpreter version as PY2, which seems to work fine.

Commenting here because this issue seems to have discussion on the differences between PY2 and PY2ONLY