Rules_docker: Cannot use numpy in py3_image

[liuyl]

Got this error message when using numpy in a py3_image

Traceback (most recent call last):
  File "/app/binary/main.binary.runfiles/default_deps_pypi__numpy_1_17_4/numpy/core/__init__.py", line 17, in <module>
    from . import multiarray
  File "/app/binary/main.binary.runfiles/default_deps_pypi__numpy_1_17_4/numpy/core/multiarray.py", line 14, in <module>
    from . import overrides
  File "/app/binary/main.binary.runfiles/default_deps_pypi__numpy_1_17_4/numpy/core/overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
ImportError: No module named 'numpy.core._multiarray_umath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/binary/main.binary.runfiles/__main__/binary/main.py", line 1, in <module>
    import numpy as np
  File "/app/binary/main.binary.runfiles/default_deps_pypi__numpy_1_17_4/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/app/binary/main.binary.runfiles/default_deps_pypi__numpy_1_17_4/numpy/core/__init__.py", line 47, in <module>
    raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy c-extensions failed.
- Try uninstalling and reinstalling numpy.
- If you have already done that, then:
  1. Check that you expected to use Python3.5 from "/usr/bin/python3",
     and that you have no directories in your PATH or PYTHONPATH that can
     interfere with the Python and numpy version "1.17.4" you're trying to use.
  2. If (1) looks fine, you can open a new issue at
     https://github.com/numpy/numpy/issues.  Please include details on:
     - how you installed Python
     - how you installed numpy
     - your operating system
     - whether or not you have multiple versions of Python installed
     - if you built from source, your compiler versions and ideally a build log

- If you're working with a numpy git repository, try `git clean -xdf`
  (removes all files not under version control) and rebuild numpy.

Note: this error has many possible causes, so please don't comment on
an existing issue about this - open a new one instead.

Original error was: No module named 'numpy.core._multiarray_umath'

Created a simple repo to reproduce this:
https://github.com/liuyl/bazel-py3-image

run command bazel run binary:main

my bazel version:

➜ test_repo git:(master) bazel version
Build label: 1.2.1
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Nov 26 15:27:31 2019 (1574782051)
Build timestamp: 1574782051
Build timestamp as int: 1574782051

docker version:

➜ test_repo git:(master) ✗ docker -v
Docker version 19.03.5, build 633a0ea

[cdg720]

Is this on OSX? I am getting the same problem and it might be the same issue (https://github.com/bazelbuild/rules_docker/issues/372).

[liuyl]

Yes it's on OSX, thanks for the link! We'll try to build & run the image on linux later

[cdg720]

numpy fails on linux as well :/

to reproduce:
git clone https://github.com/cdg720/bazel_examples.git
cd bazel_examples/numpy
bazel build -c opt foo:numpy_test
./bazel-bin/foo/numpy_test

bazel 1.2.1
macos 10.15.1
Debian GNU/Linux 9
Docker version 19.03.5

[luckyskg]

Any update on this Issue. Did you mange to find a fix for this.

[nlopezgi]

@luckyskg please see #372: afaik the only workarounds is to not run Bazel in macos (run it in a Linux container in Mac or in a linux env)

[cdg720]

It fails on Linux as well. I tested on gcp cloudshell.

bazel 2.0.0
Docker version 19.03.5, build 633a0ea838

[nlopezgi]

@cdg720 can you provide more details on the failure (is it the same as above or different)

[cdg720]

i commented before that it was failing on linux:
https://github.com/bazelbuild/rules_docker/issues/1314#issuecomment-565600442

i just reran it with newer bazel.

[nlopezgi]

@cdg720 can you comment a bit more on the error you are getting with the newer bazel? (is it the same as at the top of this issue?)

[cdg720]

last time the bazel was 1.2.1 and this time 2.0.0 and they failed with the same error.

[nlopezgi]

I'm still not sure if the error you get on linux is the same one as the one at the top of this issue, please clarify.

[cdg720]

yes

Traceback (most recent call last):
  File "/app/foo/numpy_test.binary.runfiles/foo_deps_pypi__numpy_1_17_4/numpy/core/__init__.py", line 17, in <module>
    from . import multiarray
  File "/app/foo/numpy_test.binary.runfiles/foo_deps_pypi__numpy_1_17_4/numpy/core/multiarray.py", line 14, in <module>
    from . import overrides
  File "/app/foo/numpy_test.binary.runfiles/foo_deps_pypi__numpy_1_17_4/numpy/core/overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
ImportError: No module named 'numpy.core._multiarray_umath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/foo/numpy_test.binary.runfiles/__main__/foo/numpy_test.py", line 1, in <module>
    import numpy as np
  File "/app/foo/numpy_test.binary.runfiles/foo_deps_pypi__numpy_1_17_4/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/app/foo/numpy_test.binary.runfiles/foo_deps_pypi__numpy_1_17_4/numpy/core/__init__.py", line 47, in <module>
    raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy c-extensions failed.
- Try uninstalling and reinstalling numpy.
- If you have already done that, then:
  1. Check that you expected to use Python3.5 from "/usr/bin/python3",
     and that you have no directories in your PATH or PYTHONPATH that can
     interfere with the Python and numpy version "1.17.4" you're trying to use.
  2. If (1) looks fine, you can open a new issue at
     https://github.com/numpy/numpy/issues.  Please include details on:
     - how you installed Python
     - how you installed numpy
     - your operating system
     - whether or not you have multiple versions of Python installed
     - if you built from source, your compiler versions and ideally a build log

- If you're working with a numpy git repository, try `git clean -xdf`
  (removes all files not under version control) and rebuild numpy.

Note: this error has many possible causes, so please don't comment on
an existing issue about this - open a new one instead.

Original error was: No module named 'numpy.core._multiarray_umath'

[smamessier]

Any news on this? I have the same issue on Linux.

[smamessier]

In my case, the problem was that I was using an Alpine base which has a different version of libc (musl libc) than the one against which bazel built numpy native stuff (ubuntu's libc). I changed py3_image's base to using slim-buster and it works fine now. @cdg720 could it be that the bare py3_image doesn't have libc shared libraries at all?

[cdg720]

@smamessier could you share your example with me? I tried to use slim-buster as the base but I am getting /usr/bin/docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"/usr/bin/python\": stat /usr/bin/python: no such file or directory": unknown.

Also I don't know the answer to your question.

Thanks,
DK

[smamessier]

@cdg720 Sure, in the WORKSPACE I'm using

container_pull(
        name = "python3_slim_base",
        registry = "index.docker.io",
        repository = "python",
        tag = "3.6.10-slim-buster",
        digest = "sha256:f6383806178accdc1a321058e63c65081f29c5eb27488c5abba1e8698ebdbea9"
    )

Then you need to add some symlinks (or change PATH) - that's your error -

container_image(
    name = "python3_base",
    base = "@python3_slim_base//image",
    symlinks = {
        "/usr/bin/python": "/usr/local/bin/python",
        "/usr/bin/python3": "/usr/local/bin/python3"
    },
    visibility = ["//visibility:public"]
)

Then you can use base = "//:python3_base" in your py3_image.

[cdg720]

thanks!

[alwaysprep]

@smamessier @cdg720 did it work for macOS. I could not make it work.

[tferi]

That doesn't work for me either.

[tferi]

In my case this problem was related to a mismatch between the Python versions installed on the host and execution environments. Bumping both to the same version (3.8) solved the problem in my case after I've symlinked /usr/bin/python3 to the new version.

Update:

To avoid symlinks I've pinned down the python interpreter used by the build system via a python toolchain to /usr/bin/python3.8. However, pip_install doesn't care about toolchains, and it has its own attribute where /usr/bin/python3.8 needs to be provided separately. So that's another landmine to watch out for.

[kaaquist]

I had a similar issue and I setup a python toolchain to be able to build everything using python v3.7.3. @tferi thanks for your comment that helped us.