Ray: Bazel build error for boost when building from source on ARM(aarch64)

[heavyinfo]

I'm trying to build ray from source on Jetson Nano (aarch64).
I already have working PyArrow, bazel in my system, I compiled the requisite files to the ray build folder as required.x

Ubuntu 18.04, Linux Titan 4.9.140-tegra #1 SMP PREEMPT Sat Oct 19 15:54:06 PDT 2019 aarch64 aarch64 aarch64 GNU/Linux
bazel 2.1.0- (@non-git)
Python 3.6.9
ray - latest cloned from here.

But, I get the following error at 116:1 when building ray -

ERROR: Analysis of target '//:ray_pkg' failed; build aborted:
/home/abishek/.cache/bazel/_bazel_root/f9684c56d3bfa17b4fade757272d3420/external/boost/BUILD.bazel:116:1: Configurable attribute "srcs" doesn't match this configuration (would a default condition help?).
Conditions checked:
 @boost//:linux_arm
 @boost//:linux_x86_64
 @boost//:osx_x86_64
 @boost//:windows_x86_64
INFO: Elapsed time: 1203.931s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (5 packages loaded, 2763 targets configur\
ed)
Traceback (most recent call last):
  File "setup.py", line 221, in <module>
    license="Apache 2.0")
  File "/home/abishek/.local/lib/python3.6/site-packages/setuptools/__init__.py", line 144, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/home/abishek/.local/lib/python3.6/site-packages/wheel/bdist_wheel.py", line 223, in run
    self.run_command('build')
  File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/usr/lib/python3.6/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "setup.py", line 102, in run
    subprocess.check_call(command)
  File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['../build.sh', '-p', '/usr/bin/python3']' returned non-zero exit status 1.

If I manually add linux_arm to

boost_library(
    name = "context",
    srcs = BOOST_CTX_ASM_SOURCES + select({
        ":linux_arm": [
            "libs/context/src/posix/stack_traits.cpp",
        ],
        ":linux": [
            "libs/context/src/posix/stack_traits.cpp",
        ],
        ":osx_x86_64": [
            "libs/context/src/posix/stack_traits.cpp",
        ],
        ":windows_x86_64": [
            "libs/context/src/windows/stack_traits.cpp",
        ],
    }),

New error occurs at 1586:1,

+ /usr/local/bin/bazel build //:ray_pkg --verbose_failures
ERROR: /home/abishek/.cache/bazel/_bazel_root/f9684c56d3bfa17b4fade757272d3420/external/boost/BUILD.bazel:1586:1: Configurable attribute "defines" doesn't match this configuration (would a default condition help?).
Conditions checked:
 @boost//:linux_arm
 @boost//:linux_x86_64
 @boost//:osx_x86_64
 @boost//:windows_x86_64
ERROR: Analysis of target '//:ray_pkg' failed; build aborted:

/home/abishek/.cache/bazel/_bazel_root/f9684c56d3bfa17b4fade757272d3420/external/boost/BUILD.bazel:1586:1: Configurable attribute "defines" doesn't match this configuration (would a default condition help?).
Conditions checked:
 @boost//:linux_arm
 @boost//:linux_x86_64
 @boost//:osx_x86_64
 @boost//:windows_x86_64
INFO: Elapsed time: 627.317s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (2 packages loaded, 2546 targets configur\
ed)
    Fetching @boringssl; fetching
Traceback (most recent call last):
  File "setup.py", line 221, in <module>
    license="Apache 2.0")
  File "/home/abishek/.local/lib/python3.6/site-packages/setuptools/__init__.py", line 144, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/home/abishek/.local/lib/python3.6/site-packages/wheel/bdist_wheel.py", line 223, in run
    self.run_command('build')
  File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/usr/lib/python3.6/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "setup.py", line 102, in run
    subprocess.check_call(command)
  File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['../build.sh', '-p', '/usr/bin/python3']' returned non-zero exit status 1.

If I add linux_arm there as well,

boost_library(
    name = "thread",
    srcs = select({
         ":linux_arm": [
            "libs/thread/src/pthread/once.cpp",
            "libs/thread/src/pthread/thread.cpp",
         ],
        ":linux": [
            "libs/thread/src/pthread/once.cpp",
            "libs/thread/src/pthread/thread.cpp",
        ],
        ":osx_x86_64": [
            "libs/thread/src/pthread/once.cpp",
            "libs/thread/src/pthread/thread.cpp",
        ],
        ":windows_x86_64": [
            "libs/thread/src/win32/thread.cpp",
            "libs/thread/src/win32/tss_dll.cpp",
            "libs/thread/src/win32/tss_pe.cpp",
        ],
    }),

Second error (1586:1) disappears, but first error reappears again (116:1).

I have installed boost libraries in system already for other projects, can I some how avoid bazel download of boost and add it manually like I did for PyArrow?

[JasonWayne]

@heavyinfo Hi, heavyinfo. I encountered the same issue, so do you solve it now?

[heavyinfo]

@JasonWayne Yes I did! But, after a long deep dive into the world of Bazel build rules and boost libraries.

Basically, the above errors require "//conditions:default": [], which will then follow with an error at,

external/boost/libs/context/src/asm/make_arm_aapcs_elf_gas.S: Assembler messages:
external/boost/libs/context/src/asm/make_arm_aapcs_elf_gas.S:46: Error: junk at end of line, first unrecognized character is `@'
external/boost/libs/context/src/asm/make_arm_aapcs_elf_gas.S:47: Error: operand 1 must be a SIMD vector register -- `bic a1,a1,#15'
external/boost/libs/context/src/asm/make_arm_aapcs_elf_gas.S:49: Error: junk at end of line, first unrecognized character is `@'
external/boost/libs/context/src/asm/make_arm_aapcs_elf_gas.S:50: Error: operand 1 must be an integer or stack pointer register -- `sub a1,a1,#124'
external/boost/libs/context/src/asm/make_arm_aapcs_elf_gas.S:52: Error: junk at end of line, first unrecognized character is `@'
external/boost/libs/context/src/asm/make_arm_aapcs_elf_gas.S:53: Error: operand 1 must be an integer register -- `str a3,[a1,#104]'
external/boost/libs/context/src/asm/make_arm_aapcs_elf_gas.S:55: Error: junk at end of line, first unrecognized character is `@'
external/boost/libs/context/src/asm/make_arm_aapcs_elf_gas.S:56: Error: operand 1 must be an integer or stack pointer register -- `add a2,a1,#108'
external/boost/libs/context/src/asm/make_arm_aapcs_elf_gas.S:57: Error: operand 1 must be an integer register -- `mov a3,a2'
external/boost/libs/context/src/asm/make_arm_aapcs_elf_gas.S:58: Error: operand 1 must be an integer register -- `str a3,[a1,#64]'
external/boost/libs/context/src/asm/make_arm_aapcs_elf_gas.S:60: Error: junk at end of line, first unrecognized character is `@'
external/boost/libs/context/src/asm/make_arm_aapcs_elf_gas.S:61: Error: operand 1 must be an SVE vector register -- `adr a2,finish'
external/boost/libs/context/src/asm/make_arm_aapcs_elf_gas.S:62: Error: junk at end of line, first unrecognized character is `@'
external/boost/libs/context/src/asm/make_arm_aapcs_elf_gas.S:63: Error: junk at end of line, first unrecognized character is `@'
external/boost/libs/context/src/asm/make_arm_aapcs_elf_gas.S:64: Error: operand 1 must be an integer register -- `str a2,[a1,#100]'
external/boost/libs/context/src/asm/make_arm_aapcs_elf_gas.S:69: Error: unknown mnemonic `bx' -- `bx lr@return pointer to context-data'
external/boost/libs/context/src/asm/make_arm_aapcs_elf_gas.S:72: Error: junk at end of line, first unrecognized character is `@'
external/boost/libs/context/src/asm/make_arm_aapcs_elf_gas.S:73: Error: operand 1 must be an integer register -- `mov a1,#0'
external/boost/libs/context/src/asm/make_arm_aapcs_elf_gas.S:74: Error: junk at end of line, first unrecognized character is `@'
external/boost/libs/context/src/asm/make_arm_aapcs_elf_gas.S:75: Error: unexpected characters following instruction at operand 1 -- `bl _exit@PLT'
external/boost/libs/context/src/asm/make_arm_aapcs_elf_gas.S:78: Error: junk at end of line, first unrecognized character is `@'
Target //:ray_pkg failed to build
INFO: Elapsed time: 1501.207s, Critical Path: 160.72s
INFO: 74 processes: 74 processwrapper-sandbox.
FAILED: Build did NOT complete successfully

That's because in boost library bazel rule context points to assembly files under linux_arm which doesn't compile on ARM64 in-spite of boost having ARM64 version of it.

So, TL,DR;
I've made a patch to address that and written the procedure to build and install ray on ARM64.

[heavyinfo]

The patch has been merged in the upstream, so this may no longer be an issue provided ray continues to use upstream bazel rules for boost libraries.