Bazel: Python 3 support

[abergmeier-dsfishlabs]

I would like for this issue to collect all issues concerning Python 3.

As far as I know there are multiple problems:

1. The supported values for srcs_version are PY2, PY3, PY2AND3, PY2ONLY, PY3ONLY.
   
   
   
   • PY3 is documented wrong. It actually handles similar to PY2, which is fine.
   
   
   • PY3ONLY is not even documented, although it mostly is what a lot of Python 3 users want.
   
   
2. By default Bazel always executes Python 2. Which might be an ok choice for now for PY2, PY3, PY2AND3 and PY2ONLY. It is however totally wrong for PY3ONLY. (#1406)
   It seems like checkSourceIsCompatible is used instead of a hypothetical selectPythonVersionForSource.
   
   
   
   • Question there would be why it is not using Python 3.
   
   
   • How big is Googles interest in Python 3 support on a scale of 0 (want it now) to 5 (who cares)?
   
   
   • Would anyone inside of Google be available for mentorship for extending Python 3 support?
   
   
3. Is there a way for Bazel to use a Python version embedded into a repository?
4. --force_python and --host_force_python are given to users, yet they are not documented.
5. py_binary wrapper uses Python 2 although it may not even be present (#544)
6. py_binary wrapper spawns a new process for executing python scripts. I do know that we are not talking about great performance with Python but that seems a bit excessive.
7. pkg_tar seems to internally use Python2 only. Does not work with 3.
8. Python cannot be used as a label (#2244)
9. pyd cannot be created straight-forward (#2497)
10. Support embeddable Python for Windows (#2509)

[abergmeier-dsfishlabs]

Hope I collected all input from #1406, #544 and #1446.
Updated main post.

[bsilver8192]

Some updates on 6: 3bed4af4f27bbd22b5531454095f0eda30bfac9f changed to exec instead of a whole separate process, but 679e911e948e6965afe1c23bb20ac4efdd65bac4 added a (not default yet) code path which brings it back.

[ahyangyi]

I noticed that many wrapper scripts in Bazel are python 2 only. Would be useful if I can bundle a minimal python implementation so that our Bazel repository will work on various distributions.

[ahyangyi]

As an example, PackageTar in Bazel (tools/build_defs/pkg) is Python 2 only.

[benley]

Is bazel capable of using a python interpreter that comes from a label yet? That was the main blocker the last time I looked into doing something similar, iirc.

[ahyangyi]

@benley A possible workaround: use a sh_binary that depends on your python script and your python interpreter label, and a wrapper shell script that feeds the script into the interpreter.

If that works you can do make it into a general function or rule so you can reuse the code and keep things organized until the devs add the proper supports.

[abergmeier-dsfishlabs]

Integrated info of #2352

[abergmeier-dsfishlabs]

Is bazel capable of using a python interpreter that comes from a label yet? That was the main blocker the last time I looked into doing something similar, iirc.

Nope, still not working. Integrating info of #2244

[abergmeier-dsfishlabs]

Integrated info of #2497

[abergmeier-dsfishlabs]

Integrated info of #2509

[tux21b]

It looks like I have encountered the same problem as described in #2352, which had been merged into this ticket. But I think this ticket is sightly misleading. I am not interested into supporting Python 3 (which might be a long term goal), I am interested in getting bazel to run right now using Python 2. Editing the $PATH so that "python" also points to py2 doesn't work either. Any suggestions?

~ bazel build ttbackup:debian-data --force_python=py2only --host_force_python=py2 --ignore_unsupported_sandboxing --strategy=PackageTar=standalone --verbose_failures --python2_path=/usr/bin/python2 --python3_path=/usr/bin/python3
INFO: Found 1 target...
ERROR: /home/christoph/projects/gopath/src/mgit.at/xxx/ttbackup/BUILD:45:1: null failed: build_tar failed: error executing command
  (cd /home/christoph/.cache/bazel/_bazel_christoph/9f259bee499666daa5e76d94ef621852/execroot/xxx && \
  exec env - \
  bazel-out/host/bin/external/bazel_tools/tools/build_defs/pkg/build_tar '--flagfile=bazel-out/local-fastbuild/bin/ttbackup/debian-data.args'): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
Traceback (most recent call last):
  File "/home/christoph/.cache/bazel/_bazel_christoph/9f259bee499666daa5e76d94ef621852/execroot/xxx/bazel-out/host/bin/external/bazel_tools/tools/build_defs/pkg/build_tar.runfiles/__main__/../bazel_tools/tools/build_defs/pkg/build_tar.py", line 22, in <module>
    from tools.build_defs.pkg import archive
  File "/home/christoph/.cache/bazel/_bazel_christoph/9f259bee499666daa5e76d94ef621852/execroot/xxx/bazel-out/host/bin/external/bazel_tools/tools/build_defs/pkg/build_tar.runfiles/bazel_tools/tools/build_defs/pkg/archive.py", line 17, in <module>
    from StringIO import StringIO
ModuleNotFoundError: No module named 'StringIO'
Target //ttbackup:debian-data failed to build
INFO: Elapsed time: 0.259s, Critical Path: 0.12s
command exited with code 1

[abergmeier-dsfishlabs]

@tux21b I would then perhaps create a new Issue since this here is really intended for Python3 specifically.

[skycoop]

Is there any word yet on plans for this? This is breaking parts of Tensorflow, which explicitly supports Py3.

In some cases, simply switching to using six imports would solve the problem. For example, using from six import StringIO on this line would solve the problem and maintain compatibility with Python 2.

[MarkIennaco]

bazel-0.5.0-rc2 still has this behavior.
Any timeline for making Python 3 a first class target?

[MarkusTeufelberger]

It might be enough in the bazel/tools/build_defs/pkg/archive.py case to simply import io.StringIO (which exists on Python2 as well and is not used much in the rules) or to also catch "ModuleNotFoundError"s for Python3.

[treuherz]

I've made some changes so that everything in bazel/tools/build_defs/pkg/archive.py passes tests in python 2.7 and 3, here treuherz/bazel@d6d0889. Since this is pretty small should I go ahead and open a PR or should I run it by the mailing list first?

[abergmeier]

@treuherz Everything in that direction is great. Be aware, though that it is best to also have tests, that ensure, that it continuous to work on both (Google internally does not seem to test Python 3).
Summoning @damienmg

[treuherz]

Yeah, the need for that occurred to me as I was doing it. Don't know enough about how google manages testing behind the scenes, and also don't know the rest of Bazel's codebase well enough to know how much work it would take to get it to pass if the test suite was run against both interpreters, tox-style.

I'll submit the PR with that caveat so it's at least ready.

[gregestren]

See https://github.com/bazelbuild/bazel/commit/8b5bf1f440ef8d140eb3843861fbce65709438ea for some relevant steps forward (particularly proper honoring of default_python_version).

[brandjon]

I've created a list of PY2/3 issues that I'm prioritizing in #6444, some of which are mentioned here. I'll keep this thread open as a useful index of issues but mark it P3.