Envoy: Switch to a build system that works with cross-compilation

[aduskett]

Title: Bazel is terrible and makes it impossible to cross compile.

Envoy is awesome. So awesome that I want to use it with my other projects.
As an embedded Linux engineer, I have to cross-compile everything. Generally
I use Buildroot, although I also use Yocto/Openembedded as well.

Envoy uses Bazel as it's build system, which isn't on either platform.
Bazel has some seriously bad requirements that make it very hostile for any
cross-compiled system.
1) The documentation for cross-compiling is terrible/outdated/non-existent with 0 examples.
2) It requires Java to function

Looking at the Bazel build files, it looks like Meson would probably be the easiest build system to port to. This would allow for simple cross-compiling support and a far wider adoption rate from the embedded community.

Are there any special things that Bazel is doing that makes it absolutely necessary to use?

[mattklein123]

@aduskett I'm sorry that Bazel is not working great for you, but there is a 0% chance we are going to switch build systems. I would much rather focus on what issues you have and getting Bazel fixed to support those use cases. As such, I would recommend closing this issue and opening specific requests either with this project or against Bazel. Thank you.

[aduskett]

I would strongly ask you to reconsider. Bazel seems to be used by just one or two projects, and requiring java (8 non-the-less!) to build a c++ program is super unfriendly.

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions.

[Virtual-felix]

Do someone has been able to cross compile bazel for arm32 ? I found a lot of issues about building on arm but no one is speaking about cross compiling. It seems that the bazel BUILD file doesn't have a target for arm32. Is there a way to do it ?

[staehle]

This GitHub issue is number 5 on the Google search results for "yocto bazel", which is _absolutely hilarious_.

This shows about how mature and appropriate Bazel is for cross-compile environments. My current task is to look at adding Bazel support in our proprietary build system (it works similar to a Buildroot-Yocto middle ground), and it's not looking pretty.

For comparison, we support CMake and GNU Autotools. All we have to do with those is pass in a bunch of variables and arguments that specify the cross-compiler executables, CFLAGS, etc. Then the normal build rules with steps of $ ./configure or $ cmake, then $ make install DESTDIR=/collector and 98% of our FOSS components generally "just work" with our toolchains with no source code modification or patches.

Everything I'm seeing suggests that Bazel requires
1) _configuration files_ to be added to every component/project you want to add a new arch with rigid paths, which is just... what, why is it like this? Say, if I have 4 target boards with different toolchains, you're telling me that if I add 20 new components to our OS that use Bazel, we need to edit the source code for each one of those to add support for a each toolchain? And it doesn't seem like there's an automated/boilerplate way to do that. Bruh.
2) It looks like the build commands for Bazel _have_ to include the project name in them, like $ bazel build //your-project:main. That doesn't make it very nice for generic build rules. At least Make knows to run the "all" target if you just run $ make lmao.

This support request from 2017 to the Bazel devs explains the cross-compile issue pretty well, as well as a relatively decent suggestion at the end that would solve problem 1. But of course it looks like that never happened: https://groups.google.com/forum/#!topic/bazel-dev/H9bHVX-LC6c

My work on this was prompted by the need for the addition of Abseil, another Google project: https://abseil.io/docs/cpp/quickstart.html
They recommend using Bazel, but it just seems... ridiculous. But they also have CMake support, so I think I'm just going to go that route and decline my "add Bazel support" ticket with incompatibility reasoning.

CMake is by far and large the king of make systems in my experience. Meson looks interesting @aduskett. We don't have any components that use it, but I'll have to take a look at that.

This really has nothing to do with the Envoy project, and this issue doesn't actually affect me, so sorry if you got a notification about this. I just wanted to comment this since this happens to be such a high ranking search result for anyone else looking at this issue.

Actually addressing this GitHub issue, @mattklein123 maybe Envoy could add CMake support as a companion to Bazel -- Abseil does it, so it seems like Google themselves aren't confident enough to go 100% Bazel.

[mathewprokos]

This is 100% possible with bazel its just poorly documented.

I made a bazel.bbclass that works for yocto to cross-compile to any target.

1. without modifying configuration files for every component/project
2. Using the built-in :all target "bazel build :all" or having a unique target per recipe at a minimum.

Basically you need to generate the toolchain configuration in yocto. And add the ${STAGING_DIR_TARGET} to the cxx_builtin_include_directories of cc_toolchain_config_info. This way all dependent libs/headers and be found by bazel w/o fighting the bazel sandbox.
There are also a few other CLI options that are needed so you don't have to patch the bazel projects you are building with yocto --override_repository=yocto_toolchain=${WORKDIR}/${PATH_TO_GENERATED_TOOLCHAIN}
--extra_toolchains=@yocto_toolchain//:toolchain''
--crosstool_top=@yocto_toolchain//:yocto_suite