jaxlib wheels aren't manylinux2010 compliant

[hzxie]

ERROR: An error occurred during the fetch of repository 'com_google_absl':
   java.io.IOException: Error downloading [http://mirror.tensorflow.org/github.com/abseil/abseil-cpp/archive/43ef2148c0936ebf7cb4be6b19927a9d9d145b8f.tar.gz, https://github.com/abseil/abseil-cpp/archive/43ef2148c0936ebf7cb4be6b19927a9d9d145b8f.tar.gz] to /home/hzxie/.cache/bazel/_bazel_hzxie/f706e0b0833259ca7656ed73b2a32787/external/com_google_absl/43ef2148c0936ebf7cb4be6b19927a9d9d145b8f.tar.gz: All mirrors are down: [GET returned 404 Not Found, connect timed out]
INFO: Call stack for the definition of repository 'pybind11' which is a tf_http_archive (rule definition at /home/hzxie/.cache/bazel/_bazel_hzxie/f706e0b0833259ca7656ed73b2a32787/external/org_tensorflow/third_party/repo.bzl:124:19):
 - /home/hzxie/.cache/bazel/_bazel_hzxie/f706e0b0833259ca7656ed73b2a32787/external/org_tensorflow/tensorflow/workspace.bzl:924:5
 - /home/hzxie/.cache/bazel/_bazel_hzxie/f706e0b0833259ca7656ed73b2a32787/external/org_tensorflow/tensorflow/workspace.bzl:58:5
 - /home/hzxie/Development/jax/WORKSPACE:41:1
ERROR: /home/hzxie/Development/jax/jaxlib/BUILD:59:1: //jaxlib:cublas_kernels.so depends on @com_google_absl//absl/strings:strings in repository @com_google_absl which failed to fetch. no such package '@com_google_absl//absl/strings': java.io.IOException: Error downloading [http://mirror.tensorflow.org/github.com/abseil/abseil-cpp/archive/43ef2148c0936ebf7cb4be6b19927a9d9d145b8f.tar.gz, https://github.com/abseil/abseil-cpp/archive/43ef2148c0936ebf7cb4be6b19927a9d9d145b8f.tar.gz] to /home/hzxie/.cache/bazel/_bazel_hzxie/f706e0b0833259ca7656ed73b2a32787/external/com_google_absl/43ef2148c0936ebf7cb4be6b19927a9d9d145b8f.tar.gz: All mirrors are down: [GET returned 404 Not Found, connect timed out]
ERROR: /home/hzxie/Development/jax/jaxlib/BUILD:59:1: //jaxlib:cublas_kernels.so depends on @com_google_absl//absl/strings:str_format in repository @com_google_absl which failed to fetch. no such package '@com_google_absl//absl/strings': java.io.IOException: Error downloading [http://mirror.tensorflow.org/github.com/abseil/abseil-cpp/archive/43ef2148c0936ebf7cb4be6b19927a9d9d145b8f.tar.gz, https://github.com/abseil/abseil-cpp/archive/43ef2148c0936ebf7cb4be6b19927a9d9d145b8f.tar.gz] to /home/hzxie/.cache/bazel/_bazel_hzxie/f706e0b0833259ca7656ed73b2a32787/external/com_google_absl/43ef2148c0936ebf7cb4be6b19927a9d9d145b8f.tar.gz: All mirrors are down: [GET returned 404 Not Found, connect timed out]
ERROR: Analysis of target '//build:install_xla_in_source_tree' failed; build aborted: no such package '@com_google_absl//absl/strings': java.io.IOException: Error downloading [http://mirror.tensorflow.org/github.com/abseil/abseil-cpp/archive/43ef2148c0936ebf7cb4be6b19927a9d9d145b8f.tar.gz, https://github.com/abseil/abseil-cpp/archive/43ef2148c0936ebf7cb4be6b19927a9d9d145b8f.tar.gz] to /home/hzxie/.cache/bazel/_bazel_hzxie/f706e0b0833259ca7656ed73b2a32787/external/com_google_absl/43ef2148c0936ebf7cb4be6b19927a9d9d145b8f.tar.gz: All mirrors are down: [GET returned 404 Not Found, connect timed out]

[hawkinsp]

It sounds to me like this may be a internet connectivity problem on your end, possibly a transient one.

One option would be to use the prebuilt wheels, which I would recommend in general as easier than building from source.

Other than that, perhaps check that you can download those urls (e.g., https://github.com/abseil/abseil-cpp/archive/43ef2148c0936ebf7cb4be6b19927a9d9d145b8f.tar.gz ) directly, e.g., using wget? One guess is that perhaps your network requires an HTTPS proxy (set the HTTPS_PROXY and HTTP_PROXYenvironment variables.)

[hzxie]

I don't have GLIBC >= 2.23 installed, so I have to build it from the source code.
I cannot use a proxy because I cannot connect to bazel server after enabling a proxy.

[hawkinsp]

I'm not sure we can help further with the bazel download problem, although if you don't have working network connectivity you might be able to predownload all the archives that jaxlib needs and pass them to bazel with --distdir.

However, the GLIBC >= 2.23 issue is perhaps something that we should fix. Technically the jaxlib wheels aren't compilant with manylinux2010, which requires a maximum GLIBC version of GLIBC_2.12. We should fix that. Then you would probably be able to use the precompiled wheels.

[hawkinsp]

https://github.com/google/jax/pull/1178 changes our build process to build manylinux2010 compliant wheels (at least on CPU). It's not quite as clear on GPU. You'll need to wait for us to make another jaxlib release.

[hawkinsp]

The Jaxlib 0.1.27 wheels are now manylinux2010 compliant. Hope that helps!

[dionhaefner]

I am currently running into this problem with GLIBC using the GPU releases (e.g. https://storage.googleapis.com/jax-releases/cuda101/jaxlib-0.1.32-cp37-none-linux_x86_64.whl). I am trying to use JAX on a cluster running CentOS with GLIBC 2.17, which is sufficient for TensorFlow, but apparently not for JAX.

Would there be any way to make the GPU wheels compliant with a compatible version of GLIBC, just as the PyPI wheels?

[mattjj]

Thanks for asking! See also #854 and maybe #189.

Our policy here is basically "contributions welcome". Our build process, including Docker images and the kick-off script jax/build/build_jaxlib_wheels.sh, is all open-source. If you can get JAX working on more systems, that'd be a great contribution!

[dionhaefner]

A very relatable position. I understand that supporting outdated software stacks is not exactly a top priority.

For now, I succeeded to build Jax from source. If anything with a similar setup comes across this, this is what worked (should work on CentOS 7 and RHEL):

$ scl enable devtoolset-8 bash  # for a recent compiler toolkit
$ export TF_CUDA_PATHS=/software/nvtoolkit/cuda-10.1  # otherwise TF would not find the right CUDA path
$ python build/build.py --enable_march_native \
      --enable_cuda --cuda_path /software/nvtoolkit/cuda-10.1 \
      --cudnn_path /software/cudnn
$ pip install -e build
$ pip install -e .

(assuming you have SCL and the devtoolset installed)