Bazel: Example iOS apple_crosstool_top / any documentation?

[bkase]

I'd like to make a custom apple_crosstool_top for a few reasons:

1. It seems like a better place to store all our enabled warnings/errors than passing them in as args to bazel or into to specific objc_library rules
2. I've wanted to override the builtin clang a few times, and I'm usually able to hack something in, but this seems like that _right_ way to use a custom clang.
3. As far as I can tell, it is encouraged as a place to centralize compiler information

Unfortunately, I'm having trouble understand what I exactly need to do in order to get this done.

What files do I need to make and what should the contents be?
Is there a way to look at the default apple crosstool to modify it? I imagine I'd just need to adjust a few parts.
Could you point me at documentation if it exists? If not, any advice would be appreciated.

Environment info

• Operating System: macOS Sierra 10.12.3

• Bazel version (output of bazel info release): 0.5.0rc8

Have you found anything relevant by searching the web?

https://github.com/bazelbuild/bazel/issues/1279 talks about creating an iOS crosstool and points to a broken iOS crosstool: https://gist.github.com/variac/58c124429615c98019de65f789638d3b and this commit https://github.com/bazelbuild/bazel/commit/cc21998c299b4d1f97df37b961552ff8168da17f . This doesn't include an example though.

Anything else, information or logs or outputs that would be helpful?

I'm attempting to workaround #3036 . I think an apple_crosstool_top with clang stubbed out will work (and is the "right" way to set up our build environment). But, is there a way to override the xcrunwrapper.sh that bazel uses? That would also let me investigate a workaround for #3036

Thanks!

[c-parsons]

I'll write some documentation for this, because writing your own crosstool is not straightforward. (And, as you've noted, the crosstool implementation and location has recently changed).

In the interim, though, I can give a brief outline of what you can do.

1. Start with the existing crosstool (and its dependencies and BUILD target) built by bazel as a reference point. You can generate and view this crosstool by invoking:

bazel build @local_config_cc//...
ls $(bazel info execution_root)/external/local_config_cc

Feel free to copy the contents to a directory in your workspace to get started.

1. Make modifications to crosstool as appropriate. (There is much you can do here and I won't go into detail yet unless you have something specific you are having difficulty with).

2. Point to your workspace crosstool endpoint as an override flag. For example,
   --crosstool_top=//my/workspace:crosstool. Feel free to add this to your bazelrc.

[bkase]

@c-parsons The existing crosstool inside @local_config_cc// looks like it doesn't control ios builds (am I wrong about this?). The only iOS cc configuration I see is ios_x86_64 but it sets /bin/false to all the tools.

Specifically, I'd like to either to change the objc_compile's clang invocation to point to a custom clang inside a checkout of LLVM (that I've specified in WORKSPACE).

If that can't be changed directly, overriding the xcrunwrapper.sh script itself would work to (inside that script I think I can figure out how to point to the LLVM that I've specified in WORKSPACE)

Are either of these things something that can be controlled with a crosstool? If so, would you mind explaining how I can do this?

Thanks!

[c-parsons]

The changes to @local_config_cc// to handle apple builds are recent. (And, whoops, look like are causing some issues. But that's a separate matter.)

On Mac machines which have Xcode installed, it generates a specific apple-capable crosstool based off of a template CROSSTOOL.tpl, as well as including various tools which that crosstool requires. (xcrunwrapper, libtool).

If you're not seeing this, either your machine does not have xcode installed, or you may be using an older version of bazel.

Once you have found this generated crosstool and can start to base your changes off of it, you need only to change the tool paths (for example, where you've noted that we're setting /bin/false for some toolchains) to whatever tool you would like to be using.

[bkase]

Great! Thank you -- you're right I was on 0.5.0rc8 (which didn't have it), but 0.5.0 does. I see everything I need now.

There is one more thing that I would love advice on:

The big reason we want to use a custom clang is to use a clang compiler plugin. This is another input to the build, but doesn't fit into any of the fields of objc_library.
The best way I can think of to achieve this is to stick our compiler plugin.dylib into the hdrs field of objc_library (via some file like __plugin_dylib.h) and then lifting it out into the proper flag and renaming the file inside xcrunwrapper.sh, replacing -I __plugin_dylib.h with something like -Xclang -load -Xclang -plugin plugin.dylib. However, this is admittedly pretty hacky -- do you have a better idea? Is there a way just by manipulating the crosstool to get in these extra inputs somehow?

It's important that we stay with the objc_library infrastructure so all the other bazel things will work properly without re-inventing the wheel (tulsi, bazel's extra_actions, aspects, etc).

I'm also going to include a VFS template (which I'm planning for now on doing in a similar way as the clang plugin -- with a __vfs_template.h)

[bkase]

cc @c-parsons

[c-parsons]

Apologies but I'm a bit out of my element regarding common uses of clang plugins -- do you anticipate deriving the plugin dylib differently between builds, or is this a consistent part of your toolchain?
If it's the latter, then you should be able to simply augment xcrunwrapper.sh (or you could do it by adding always-on flags to compile actions via crosstool, if you prefer) so that you include flags -Xclang -load -Xclang -plugin plugin.dylib for appropriate actions.

You need only put plugin.dylib adjacent to xcrunwrapper.sh in your crosstool directory -- it doesn't need to be a part of your library BUILD files if it's already prebuilt and just used by the toolchain.

I hope it's that simple -- let me know if I'm oversimplyfing this.

[bkase]

@c-parsons no problem -- we actually do anticipate deriving the plugin dylib differently between builds. We're using this plugin to run custom lint-rules on the Clang AST to throw compile errors when people use certain APIs improperly. Whenever a new rule is added (arbitrary C++ code essentially), we rebuild the dylib. In order to keep our build reproducible, this dylib should be treated as a build input since adding a new rule for example could cause source files that weren't touched to incorrectly compile.

[bkase]

@c-parsons

I'm still having trouble using the crosstool -- I'm unable to get an objc-compile action to point to my overrided wrapped_clang or xcrunwrapper.sh.

I'm testing with this command (not sure whether to use apple_crosstool_top or crosstool_top so I'm using both):

bazel build -s '//objc/library/target' --apple_crosstool_top=//local_config_cc:toolchain --crosstool_top=//local_config_cc:toolchain --ios_multi_cpus arm64

I'm certain that this build command is picking up my copy of the toolchain because if I change //local_config_cc:toolchain to //local_config_cc:toolchain2 the build fails.

I'm certain that it's not invoking my wrapped_clang or xcrunwrapper.sh because I changed the contents of those scripts to fail and that did not fail a build. Additionally, I'm able to comment out the contents of osx_tools_{arch} filegroup:

    filegroup(
        name = "osx_tools_" + arch,
        srcs = [
#          ":cc_wrapper",
#          ":libtool",
#          ":make_hashed_objlist.py",
#          ":wrapped_clang",
#          ":xcrunwrapper.sh",
        ],
    )
    for arch in OSX_TOOLS_ARCHS
]

and my build still succeeds.

Do you see anything wrong in what I've done?

Thanks!

[c-parsons]

Regarding using a custom dylib -- as long as the prebuilt dylib is a build input, my advice should essentially be the same. (In reality, you might need to use a genrule dependency to generate your dylib and move it under the crosstool directory)

It's hard to help debug without being able to see the entire contents of your crosstool directory and CROSSTOOL file. Could you upload to a github repo so I can take a look?

[bkase]

@c-parsons re: custom dylib -- right! That makes sense, it's just a build-input to the rules defining the crosstool, makes sense.

re: being unable to override the crosstool: Here is an example GitHub project showing my problem (make sure you're on the crosstool-override-tests branch):

https://github.com/bkase/debugging-tulsi-broken/tree/crosstool-override-tests

Thanks for your help!

[rahul-malik]

@c-parsons - Have you had a chance to look at @bkase project? We're trying to see if we can continue the development of our custom crosstool. Thanks!

[c-parsons]

Thanks for the ping, and sorry about not following up.
So it looks like you are commenting out a few items in your custom crosstool in an attempt to "break" it, to use as proof that we are indeed using your custom crosstool? :)

(In the future, you can also build with the -s, and it will output logs for what the actual action invocations are. You can thus verify any of your crosstool settings are being applied. Just an option!)

Can you try using the following additional flags in your invocation? I believe they may be necessary to ensure the crosstool is being used for objc_ rules:

--apple_crosstool_transition --experimental_objc_crosstool=all

[bkase]

Great that works! Thanks for your help!

Docs explaining this (especially these hidden args) would definitely be really helpful for future people wondering about this.

[c-parsons]

We're in the process of fixing these flags to have the correct default values (and thus we won't have to document them!), but we're running into a couple hitches migrating some key users.

You're an "early adopter" ;)
Sorry for the growing pains, and glad this is resolved now.

I'll leave this open to address documenting making a custom crosstool

[rahul-malik]

@c-parsons - Thanks for all the help, we're iterating on this now!

Just from an integration perspective: Is it more common to check in a crosstool into our repository or to have a rule that creates a custom toolchain and reference that from the CLI?

[c-parsons]

No problem, and glad to help!

To answer your question, it depends a lot on your requirements.
Err on the side of using a consistent crosstool across those developing your repository. However, if there are sufficient machine-to-machine differences (for example, developers may be using different compiler versions, or different versions of xcode...), you will need to use a rule that creates the toolchain locally (after determining whatever system state it needs to be informed of)

[susinmotion]

From a quick glance, it looks like this should have been closed with cparsons' work! Feel free to reopen if there's more to be done.