Among the multiple "release" outputs of my project, I'd like to include a single static library, which packages the objects of all dependencies into it. However, from what I can tell, bazel cannot build such targets.
The following target produces a single .so
containing all symbols from //product/foo
and //product/bar
.:
cc_binary(
name = "libproduct.so",
linkshared = 1,
deps = [
"//product/foo",
"//product/bar",
],
)
I believe there is no way to generate a similar static library (libproduct.a
).
Is that something that will be of interest to add?
(#492 seems somewhat related, but not exactly)
+1; my project has similar hard requirement and similar problem.
Is this what you are looking for?
https://www.bazel.io/versions/master/docs/be/c-cpp.html#cc_binary.linkstatic
@hermione521 : I don't think so.
If I try to create the .a
using a cc_binary
rule like so:
cc_binary(
name = "libproduct.a",
linkstatic = 1,
deps = [
"//product/foo",
"//product/bar",
],
)
Then the output isn't an object archive.
And my understanding of linkstatic=1
in cc_library
rules is that it means that the symbols in the library will be statically linked into the binary that depends on it, not that the library itself will include symbols of its transitive dependencies.
Ah, I'm sorry. Let's ask @lberki
No, we don't support that at the moment :( We were considering implementing something like that with @mhlopko , but we have other priorities at the moment.
A workaround would be to use a genrule to package the static library like this:
cc_library(name="a")
cc_library(name="b")
genrule(name="g", srcs=[":a", ":b"], cmd="$(AR) $(locations :a) $(locations :b)")
I took a bit of artistic freedom since $(locations)
will return both a static and a dynamic library and that's not exactly how ar
should be called, but it should work as long as you don't need symbols from transitive dependencies.
Thanks, though I do want to include symbols from transitive dependencies
(for example, building a static library for tensorflow (today, there is a shared library))
This is a pretty major problem for us, too. I'll try the genrule workaround suggested above, but it makes me mighty nervous.
cc_binary with
I'm a bit concerned about the increased likelihood of accidental ODR violations, which requires a globally consistent partition into cc_export rules. That's not a problem per se, except when different people want different partitions. Internally, we have everything in a single repository, and we actually have requests to support multiple different overlapping partitioning schemes. Externally, you usually don't have a choice. Whatever upstream decides is what you get.
@ulfjack: there are a bunch of options. One is a cc_export
rule you suggested, another would be an output group on regular cc_binary
rules that would put all the .o files in its transitive closure into a single .a .
I was thinking that the partitioning should be the responsibility of whoever is writing the cc_binary
rule(s). I'm not particularly worried about coordination problems if there are multiple such rules with multiple authors because people should know what symbols a library they use contains and library authors should know what symbols they want to export. Do you think that's too optimistic?
I think it's very easy to mess up the dependencies. Consider for example a workspace that has two cc_library and two cc_export rules, with each of the cc_export rules depending on exactly one library. Now an owner of one of the lower-level libraries isn't aware of the cc_export going on in some completely different part of the workspace, and adds a dependency on the other library. Everything keeps working just fine, except that the application blows up _at runtime_, because it contains two copies of some symbols. They may not be exported symbols, so neither the static linker nor the dynamic linker would be able to check.
I understand that. But who is in a better situation to sort these things out if not the owner of the top-level rule?
I'm not saying that they aren't. But I think it's a problem that it's a runtime error rather than a compile-time error. If we can make it a compile-time error, then I think that may be an acceptable solution for now. If the burden on the top-level owners is too large, or if it causes users to create duplicate (multiplicate) rule hierarchies, then we can still see if we can come up with a better solution. We certainly need to document it carefully because it's easy to shoot yourself in the foot.
I can think of two ways of making it a compile-time error. 1. Use constraints to annotate every rule that goes into a specific cc_export rule. 2. Look for all cc_export rules in the transitive closure of a cc_binary or cc_test and check that their corresponding transitive closures are non-overlapping.
2 seems like it could be expensive; 1 would be optional.
I'd like to keep the option open of statically linking everything even if there are intermediate cc_export rules, so maybe cc_export isn't a good name for that. Maybe we should call it cc_package or cc_transitive_library. Then at the cc_binary level we can decide whether we cut the binary into individual .so's at the cc_transitive_library level or whether to link everything statically after all.
Consider that many debian packages come with both .so and .a which allows developers to decide after the fact whether to link a specific library statically or dynamically.
Related to my previous comment: I have put together a utility script for myself that does the necessary hackery with with the .a files to produce a statically-linked output for my purposes; unfortunately, this introduced me to https://github.com/bazelbuild/bazel/issues/1740, in which a cc_library() has different outputs depending on compilation mode (in -c opt it also outputs a .pic.a library).
@steven-johnson
I am the creator of issue #1740.
Yes, I also have some homebaked rule that builds a fully static shared library (and with extra functionalities such as linking with a version script).
Currently I use a hack: if a cc_library outputs at least some .pic.a library, I use them, otherwise I use all .a library. It somehow worked in all combinations of settings I encounter, so I do recommend you to do something else if you need it to work immediately.
However if we get better support from Official Bazel I can simplify things. You know, I am the sole maintainer of our Bazel workspace, since it contains too much cryptic workarounds that work around cryptic workarounds ...
if a cc_library outputs at least some .pic.a library, I use them, otherwise I use all .a library
Thanks, I'll give that a try. On the whole, Bazel is great, but the number of workarounds necessary for various outside-of-Google necessities of life is still disappointing.
@lberki @ulfjack : Just wanted to check in on this. Would you guys have the bandwidth to address this soon?
@mhlopko is OOO for a long while, so I don't think we can get to this soon-ish :(
So I'm back :) We have no design ready for this (and other linking related requests), but I'll try to push this forward. We really have to polish our linking story. Sadly, I expect it will take a few weeks to fully implement considering my current load.
@mhlopko how can we help? where should we begin?
I don't think there is anything you could do right now. I'm discussing the design internally and soon I'll publish a design doc to bazel-discuss. Then I'll start implementing.
How about publishing the doc now, and having the discussion on bazel-discuss?
Finally! :) Please take a look and comment on the transitive libraries design doc: https://docs.google.com/document/d/1-tv0_79zGyBoDmaP_pYWaBVUwHUteLpAs90_rUl-VY8/edit?usp=sharing
@mhlopko
Thank you for sharing the doc!
What are the next steps?
We still discuss this a lot. I'll add one more approach to the design doc this week that would make the possibility of introducing diamond "by accident", and therefore possibility of changes to distant dependencies breaking the top level build, smaller. More points still need investigation:
I'm flying to New York next week and will discuss this with the rest of the team. I really hope to have a final design ready by the end of the next week.
Eager to hear how this turns out -- it's one of the last sticking points for my project.
Good news, work on transitive libraries has started :)
Any update on the rule for building static libraries with Bazel? I would like to use tensorflow as a static library in my project (which is not built with Bazel) as it is the rule for all its dependencies.
Hi @annemenini
https://github.com/bazelbuild/bazel/issues/4078#issuecomment-345620850
Let me repeat it here: transitive libraries have been deprioritized in favor of C++ sandwich. We will post an update for transtive libraries later in Q1/2018.
I don't understand how the "C++ sandwich" approach is in any way a substitute for being able to build real static libraries. This missing feature is a deal-breaker for my project in terms of adopting Bazel.
With C++ sandwich you (and we) will be able to quickly create new rules that can integrate with native C++ rules.
@ventrescadeatun
I'm with Steven in that I don't think this is at all a replacement for static library compilation; I hope this will still be addressed at some point.
It's not a replacement, it's a building block that will allow us to write cc_static_library rule mentioned in the https://docs.google.com/document/d/1d4SPgVX-OTCiEK_l24DNWiFlT14XS5ZxD7XhttFbvrI/edit in Skylark.
This will also be useful for Envoy and other projects that want to use https://github.com/google/oss-fuzz, since it's nice to get a .a
that we can late link to.
Is there any progress on cc_static_library
availability?
Nope :/ I don't expect any development in this area before #4570 is implemented.
I am interested in a static library also.
I also have such a requirement with bazel to build a fully static library for tensorflow lite.
I too have been attempting to build tensorflow statically, so far without much success.
Hi all,
Please see our roadmap:
https://docs.google.com/document/d/1K3Dq1JDDWjGtvTyFcIwy_-Crm7ZuD9FzSxjDawDhJRA/edit
Current estimation is that you'll be able to build transitive static
library by the end of 2018.
Thanks!
On Wed, Apr 18, 2018 at 5:02 PM Robert Grosse notifications@github.com
wrote:
I too have been attempting to build tensorflow statically, so far without
much success.—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/1920#issuecomment-382418567,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAxt5hVizd4mgh_kOLOnsmJUsHG_VfOwks5tp1WFgaJpZM4KSz2R
.
--
Marcel Hlopko | Software Engineer | [email protected] |
Google Germany GmbH | Erika-Mann-Str. 33 | 80636 München | Germany |
Geschäftsführer: Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
| Registergericht
und -nummer: Hamburg, HRB 86891
Hi there,
It's getting close to the end of 2018. Is there any update on this?
Thanks.
We have a tracking issue to which you can subscribe to: https://github.com/bazelbuild/bazel/issues/5200. #1920 will be fixed as part of #5200.
Very excited for this feature, to link a single static lib built with bazel
with node-gyp
native addons.
Looking forward to this feature being implemented. I too have a requirement for bazel to build a fully static library for tensorflow lite.
Work on #5200 has started, we plan to share designs in the Q3 and have it finished in Q4.
Note that it is possible to implement naive rule that will unarchive all transitive static libraries, and rearchive them into one big .a file today. See the example in rules_cc that can be extended to do this https://github.com/bazelbuild/rules_cc/blob/master/examples/my_c_archive/my_c_archive.bzl. So you don't have to wait for #5200 to be implemented if you don't need to be able to filter out which deps will make it into the final archive in the same way as cc_shared_library
will.
@oquenchil can you please provide an update here?
We would need to add a Starlark rule to rules_cc that leverages the toolchain API (get_tool_for_action
) to create the static library. A design doc would be needed first.
We don't have plans to work on this right. Any help is welcome.
Most helpful comment
Work on #5200 has started, we plan to share designs in the Q3 and have it finished in Q4.
Note that it is possible to implement naive rule that will unarchive all transitive static libraries, and rearchive them into one big .a file today. See the example in rules_cc that can be extended to do this https://github.com/bazelbuild/rules_cc/blob/master/examples/my_c_archive/my_c_archive.bzl. So you don't have to wait for #5200 to be implemented if you don't need to be able to filter out which deps will make it into the final archive in the same way as
cc_shared_library
will.