Bazel: Support consuming autoconf/make external dependencies via a Bazel rule

[htuch]

Having recently converted a mid-sized project from cmake to Bazel (https://github.com/lyft/envoy, https://github.com/lyft/envoy/issues/415), one of the very significant challenges was dealing with more than a dozen external dependencies (https://lyft.github.io/envoy/docs/install/requirements.html), almost all of which had no native Bazel support and were mostly autoconf/make based.

We had a requirement to consume these external dependencies in multiple ways. Some organizations (notably Lyft, the project owner) required the dependencies as prebuilts, essentially .a and .h files in a standard --prefix layout. This was also required by our CI system for performance reasons. Other developers wanted a way to have Bazel perform the build of the dependencies. To compose with other projects, allowing Envoy to be consumed as a git_repository, we wanted to avoid having the consuming project figure out its own dependencies, so we needed a self-contained Bazel build of the dependencies as well for this reason.

We settled on building the dependencies under a common recursive make both when creating the prebuilts and also ran the recursive make under a Bazel genrule for the Bazel build. Unfortunately, AFAICT, genrules need to know ahead of time all the headers and libraries in its outs, we can't just glob them. Explicitly declaring the libraries is fine, each dependency only had 1 or 2. The headers would be a nightmare though by hand, there are O(1k) of these.

We settled on performing an offline build outside of Bazel, then analyzing the build outputs to generate a .bzl with the smarts to provide a genrule for the dependencies. This .bzl then gets checked into the repository. See https://github.com/lyft/envoy/pull/716. This seems like a huge amount of engineering effort to allow for easy consumption of external deps that are non-Bazel based (Please let us know if there is a better way!).

I think Bazel should have an option in its git_repository for taking an external repository, performing a native build under autoconf/make (or whatever native build system it has) and discovering its generated files (perhaps with globs), that runs prior to building the dependency graph.

CC: @mattklein123 @lizan @tschroed

[steven-johnson]

+1; this is a nontrivial blocker to C++ projects that use existing non-Bazel libraries (i.e.: most of them). (I had hoped that the "contributed Bazel BUILD files" repository would help solve this, but it appears to be mostly stagnant.)

[htuch]

I think what we're doing in https://github.com/lyft/envoy/pull/716 is actually not the best way to approach it. By defining a repository_rule (https://bazel.build/versions/master/docs/skylark/repository_rules.html) and performing the autoconf/make there rather than in a genrule, we can probably achieve the goal of having Bazel pick up on the make outputs prior to building the dependency graph.

[hlopko]

Can you investigate if repository_rule solves your needs and then ping this thread?

[htuch]

@mhlopko It does, see https://github.com/lyft/envoy/pull/747. However, it's got a horrible UX because of https://github.com/bazelbuild/bazel/issues/1289. I've filed an issue to improve documentation of best practices for consuming C++ autoconf/make dependencies at https://github.com/bazelbuild/bazel/issues/2814. I'll close this issue out, since the machinery is there today.

[htuch]

Re-opening. As discussed in https://github.com/lyft/envoy/pull/747#discussion_r111295953, Bazel could do more to support finer grained builds of autoconf/make dependencies under a repository_rule, by acting as a make job server. Otherwise, all external deps that build under autoconf/make really need to be built as a single monolithic dependency for performance reasons, which is bad for the reasons discussed in that PR. CC @lizan.

[htuch]

I think https://github.com/lyft/envoy/pull/747#discussion_r111400548 provides some useful data on what the performance issues are. There's a foundational issue when it comes to dealing with multiple autoconf/make deps and how the repository_rule(s) should be structured. In particular, since repository_rule is sequential today, we're forced to group all deps under a single rule.

There's also an issue that hasn't been previously raised. If the the autoconf/make dependency (e.g. Lightstep in our case) wants to consume from something that can be cleanly imported as a git_repository (e.g. protobuf for Lightstep) as it has an existing BUILD file, there doesn't seem a clean way to plumb the location of the headers/libraries into the repository_rule for autoconf/make to "get" at the Bazel-side dependency. I.e. there's no way to express dependency relationship between repositories. For this reason, we're forced to fetch/build protobuf as an autoconf/make dependency rather than first-class Bazel.

It seems we need to either make repository_rule capable of handling these scenarios or introduce first-class support for autotools deps.

[dslomov]

Use https://github.com/bazelbuild/rules_foreign_cc