Rules_docker: docker_pull fails on syntax error

[vrischmann]

Hi,

I'm using ArchLinux and I get an error when adding that to my workspace:

docker_pull(
    name = "busybox",
    registry = "index.docker.io",
    repository = "library/busybox",
    tag = "1.26.2-musl"
)

the error is:

    Pull command failed: Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/sphax/.cache/bazel/_bazel_sphax/420d00f9ffa546a452fc5bdf768b31b7/external/puller/file/puller.par/__main__.py", line 26, in <module>
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 646, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 616, in _load_backward_compatible
  File "/home/sphax/.cache/bazel/_bazel_sphax/420d00f9ffa546a452fc5bdf768b31b7/external/puller/file/puller.par/containerregistry/client/__init__.py", line 23, in <module>
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 646, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 616, in _load_backward_compatible
  File "/home/sphax/.cache/bazel/_bazel_sphax/420d00f9ffa546a452fc5bdf768b31b7/external/puller/file/puller.par/containerregistry/client/docker_creds_.py", line 26, in <module>
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 946, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 885, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1157, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1131, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1112, in _legacy_get_spec
  File "<frozen importlib._bootstrap>", line 427, in spec_from_loader
  File "<frozen importlib._bootstrap_external>", line 544, in spec_from_file_location
  File "/home/sphax/.cache/bazel/_bazel_sphax/420d00f9ffa546a452fc5bdf768b31b7/external/puller/file/puller.par/httplib2/__init__.py", line 942
    print "connect: (%s, %s) ************" % (self.host, self.port)
                                         ^
SyntaxError: invalid syntax

By default Arch uses Python 3, looks like this is the problem ?

Is there a way to tell bazel to use python 2 ? I tried to run bazel with --python2_path=/usr/bin/python2 when it doesn't work.

[mattmoor]

That is a very good question... @duggelz @damienmg who will certainly know more than me.

[damienmg]

Try bazel build --force_python=py2 --python2_path=/usr/bin/python2. Ultimately we should make all that code works for both py2 and py3.

[duggelz]

This is a known problem with ArchLinux: https://www.python.org/dev/peps/pep-0394/

"... however, end users should be aware that python refers to python3 on at least Arch Linux (that change is what prompted the creation of this PEP), so python should be used in the shebang line only for scripts that are source compatible with both Python 2 and 3."

But making the code also work under Python3 would be a laudable goal.

[vrischmann]

Hi,

thanks for the suggestions. I will try again tonight, with and without a virtualenv to try and make it work. I had another problem related to Python 3 when using pkg_tar and somehow I managed to make it work by using a virtualenv, but when I first tried it it didn't work..not sure why.

Anyway I'll do more experimenting and let you know

[vrischmann]

I did some more testing and I can't make it work. Tried with --force_python=py2 --python2_path=/usr/bin/python2 it doesn't look like it makes a difference.

Running with a virtualenv doesn't work either. Looking at the output of bazel build --verbose_failures:

(cd /home/sphax/.cache/bazel/_bazel_sphax/45d4664ccf59303a1a632af3918c875a/execroot/test-bazel-pull && \
  exec env - \
  bazel-out/host/bin/external/io_bazel_rules_docker/docker/extract_id --tarball external/busybox/image/image.tar --output_id bazel-out/local-fastbuild/bin/cmd/cmd.image.tar.id --output_name bazel-out/local-fastbuild/bin/cmd/cmd.image.tar.name)

it runs with exec env - so I tested with a virtualenv activated and sure enough env - python runs /usr/bin/python not the one in my virtualenv. env python however launches the correct one. I suppose there's a good reason to use env - here though.

Anyway, for now I've been unable to find a good solution/

[damienmg]

Oh I just realized the failure is in docker_pull. @mattmoor: maybe we should make sure that puller is compatible with python 2 or test the python on the path in docker_pull...

[mattmoor]

The puller is certainly python 2 compatible. From the stack trace it seems it isn't python 3 compatible.

Given where it's exploding that makes sense because I specifically import the python 2 version of httplib2, which I did as a simplified form of what we have in third_party. I would really rather depend on a third_party-style definition of httplib2 than continue to roll/improve my own import of it. @steren FYI

@steren @damienmg @duggelz Do we have any reasonable documentation for how to ensure a Bazel project is python 2 and 3 compliant? I know we have some flags internally, but I'm not sure what's publicly available here, or what the level of support is in the toolchain (including subpar).

[duggelz]

It's hard to do something non-hacky when /usr/bin/python is one thing for ArchLinux, and something else for the rest of the world. We can't, for example, use /usr/bin/python2 because that would break MacOS.

Subpar works under Python 2 and 3, which I test via BUILD files like this:

[py_test(
    name = "%s_%s_test" % (src_name, version),
    size = "small",
    srcs = [src_name + "_test.py"],
    default_python_version = version,
    main = src_name + "_test.py",
    srcs_version = "PY2AND3",
    deps = [
        ":compiler_lib",
        ":test_utils",
    ],
) for src_name in [
    "cli",
    "manifest_parser",
    "python_archive",
    "stored_resource",
] for version in [
    "PY2",
    "PY3",
]]

As a stop-gap, you could add if sys.version_info >= (3,0,0,0): sys.exit('Helpful message') to your code, maybe with a link to this thread.

[duggelz]

Does ArchLinux even have Python 2 installed? We could try something hacky in the puller like re-execing itself under /usr/bin/python2.

[vrischmann]

Does ArchLinux even have Python 2 installed

yes, it's under /usr/bin/python2.

[andrew-d]

I just ran into this problem. If you want to go down the hacky re-exec route, you can try something like this, which works for me:

#!/usr/bin/python

import os
import sys
import distutils.spawn


if sys.version_info[0] > 2:
    py2 = distutils.spawn.find_executable('python2')
    if py2 is None:
        sys.exit('Not compatible with Python 3')

    print('Re-executing as python 2')
    os.execl(py2, py2, *sys.argv)


print("Hello world")

[duggelz]

Bazel in version 0.53 broke essentially all Python 3 usage. I'm still not sure why they thought that was ok. So the example I have above for subpar doesn't work at all any more. I don't have a good answer, but "write code that works under Python 2 and 3" seems like the best option at present.

Also, we might look into the 'requests' library as a generally superior alternative to 'httplib2' that works on both Python 2 and 3.

[mattmoor]

@damienmg Are you aware of this?

[duggelz]

See https://github.com/bazelbuild/bazel/issues/3517 for Python 2/3 issues with 0.53

[Mistobaan]

I stumbled on this bug right now. a quick fix is to set the BAZEL_PYTHON ENV variable

export BAZEL_PYTHON=$HOME/miniconda2/bin/python2.7

[mattmoor]

Yeah, this workaround was probably made possible by this recent PR.

You should now be able to alias bazel with BAZEL_PYTHON=... bazel.

I don't think passing --python_path will help this case (I think it only effects py_binary) or you could put that into your .bazelrc :(

[duggelz]

Trying to make this code work under Python 2/3, and ran into the problem that Bazel itself has code that doesn't work under Python 3: https://github.com/bazelbuild/bazel/issues/3816

[tanner-bruce]

Running on bazel@master I am able to get past this, however bazel then fails with

  File "/home/tanner/.cache/bazel/_bazel_tanner/4aec6733e67a1c2f75ff344f0772f1aa/bazel-sandbox/7176817177856009665/execroot/__main__/bazel-out/host/bin/external/io_bazel_rules_docker/container/create_image_config.runfiles/__main__/../io_bazel_rules_docker/container/create_image_config.py", line 115, in main
    for label, value in labels.iteritems():
AttributeError: 'dict' object has no attribute 'iteritems'

Setting BAZEL_PYTHON=/usr/bin/python2 does _not_ resolve this, however running bazel with --python_path=/usr/bin/python2 does succeed.

[mattmoor]

@hwright has been fixing all of our Python3 problems, this looks like one he just fixed, which hasn't merged yet because Bazel broke all our CI with 0.8 :(

[hwright]

@tanner-bruce #255 should resolve that specific issue, but if you find more, please let us know.

[nlopezgi]

is this still an issue? please reopen if so.