Rust: Support Apple app store bitcode

[brson]

Bitcode is the future of Apple's app distribution and we don't support it now. Doing so is tricky because Apple choses whatever rev of LLVM they want and upgrade it per their whims. We can't couple our LLVM to theirs because we have our own needs and can't be forced to upgrade whenever Apple decides.

Here are some options:

• We ship an entirely separate toolchain for targetting Apple bitcode. This is quite ugly.
• We change rustc to dynamically load LLVM, create a rust-llvm package and an optional rust-llvm-apple package, and have rustc load the apple LLVM to deal with their bitcode.
• We create a rustc_llvm_apple crate and just link two entire LLVMs into rustc at all times. This probably would not fly
• We create a build of rustc called rustc-apple and ship it in an optional rustc-apple package. It is exactly like rustc except instead of linking to rustc_llvm it links to rustc_llvm_apple. When rustc gets a request to emit bitcode it just defers completely to the rustc-apple binary. I actually fancy this solution quite a bit.
• We could investigate a bitcode sanitizer to translate our bitcode to theirs. I think the likelihood of success here is low because maintenance. I believe RenderScript does this.

I think the LLVM dynamic loading and defer-to-alternate-rustc solutions are most promising.

cc https://users.rust-lang.org/t/ios-rust-integration/6928/4

cc @bluejekyll

[nagisa]

We change rustc to dynamically load LLVM, create a rust-llvm package and an optional rust-llvm-apple package, and have rustc load the apple LLVM to deal with their bitcode.

Dynamically loading LLVM is certainly not an option. The LLVM API is less stable than its bitcode and, if we build against LLVM x.y, then having LLVM x.z instead of x.y will almost certainly break rustc. Just link to the system LLVM statically (like --llvm-root already does).

We create a rustc_llvm_apple crate and just link two entire LLVMs into rustc at all times. This probably would not fly

Just link the system LLVM (statically) for apple distributions. In my experience it already works kinda well.

We could investigate a bitcode sanitizer to translate our bitcode to theirs. I think the likelihood of success here is low because maintenance. I believe RenderScript does this.

Okay… so, the thing is that LLVM bitcode format is kinda stable between versions of LLVM. Does apple do something unusual to make that not true, or is the bitcode format used not LLVM altogether (i.e. has apple specific stuff)?

---

I feel like I heard all the way back in 2013-ish that using LLVM bitcode as distribution format is stupid. I certainly agree; does distributing native binary libraries not work anymore for iOS?

[nagisa]

I’m also very interested in what would happen when Apple upgrades to LLVM version with incompatible bitcode format internally and people still compile stuff with old compilers.

[bluejekyll]

I experimented briefly with this, i.e. output bitcode during cargo builds, then tried to create a static library with the bitcode included. When I tried to llvm-link against the bitcode library, I got an incompatible LLVM version from the bitcode.

I don't have an easy test to reproduce this, but my guess is there is just a dumb version check that denies linking between different LLVM versions? Or, I just did something entirely wrong. I will try to come up with a test case when I have some time to look at this again.

[sanxiyn]

It would be useful to have exact error message from llvm-link.

[nagisa]

@bluejekyll LLVM has a number of bitcode files in its test directory. Tests against these bitcode files are run continuously (llvm-dis-3.8/opt-3.8 understands a 3 year old bitcode file from 3.2 just fine, for example), so it must be some apple stuff.

[ricky26]

Just link the system LLVM (statically) for apple distributions. In my experience it already works kinda well.

This is somewhat easier said than done - as far as I'm aware, the only blessed apple-llvm version for App Store uploads is the one shipped with the current Xcode. This also potentially means maintaining the LLVM bindings for two LLVM versions (not necessarily two neighbouring minor versions, either). I don't think it's valid to just use an old version of apple-llvm.

I’m also very interested in what would happen when Apple upgrades to LLVM version with incompatible bitcode format internally and people still compile stuff with old compilers.

I think they avoid this by only allowing the latest Xcode to submit apps (and I'm pretty sure it's burned into the output image which LLVM version you used).

IIRC, the bitcode is packaged for each architecture, since cross-architecture bitcode isn't generated by clang (I think that's an anti-goal of clang and bitcode in general). It's stored in a section for each object file so at the very least it's duplicated. That could be part of why someone mentioned that bitcode might not be the best way of going about this.

I feel a bit like all of the recommended solutions are somewhat icky. The least messy way I can think of is to allow different targets to overload the codegen behaviour and have the Apple codegen path be in a dynamic crate. (Which would just be the regular codegen path compiled against apple-llvm.)

Since this bug mentions the App Store, is it worth talking about the exception handling story here? (i.e. panic=abort is strictly required at the moment.)

[bluejekyll]

LLVM has a number of bitcode files in its test directory. Tests against these bitcode files are run continuously (llvm-dis-3.8/opt-3.8 understands a 3 year old bitcode file from 3.2 just fine, for example), so it must be some apple stuff.

@nagisa thanks for letting me know this. It gives me hope that there might still be a solution here, and that I was probably doing something wrong.

[nagisa]

@ricky26 good points.

as far as I'm aware, the only blessed apple-llvm version for App Store uploads is the one shipped with the current Xcode.

Isn’t Xcode LLVM on apple the same thing as system LLVM? I meant the Xcode LLVM then. We would have to make sure xcode is always the most recent version when shipping rustc trains.

Of course the way apple does things rules out us producing a valid apple bitcode with an old version of rustc and essentially forces us throw all the benefits our stability story provides out the window, and I don’t see any way this could be fixed.

This also potentially means maintaining the LLVM bindings for two LLVM versions

We already maintain¹ support for LLVM versions 3.7 through 3.9 (and potentially trunk). As long as Xcode’s LLVM is not some ancient version, I think we’re good in that regard. If Xcode LLVM is really some ancient/custom/etc version, then I don’t see us being able to support this feature at all, then. Especially since we have no option of sending patches to _that_ LLVM in order to add the features we need. I also wouldn’t want to lock rustc into supporting 3.7 forever in case Apple decided to not update Xcode LLVM until 2038.

¹: however, if rustc was built against LLVM x.y, it must be linked to LLVM x.y exactly.

[brson]

Dynamically loading LLVM is certainly not an option. The LLVM API is less stable than its bitcode and, if we build against LLVM x.y, then having LLVM x.z instead of x.y will almost certainly break rustc. Just link to the system LLVM statically (like --llvm-root already does).

@nagisa The C++ API is unstable, but we use the C API and have had a lot of success supporting multiple versions of LLVM at once. I don't see the difference in terms of API support.

Just link the system LLVM (statically) for apple distributions. In my experience it already works kinda well.

We could just blanket ship Apple's LLVM for all apple platforms, but this means coupling our LLVM to Apples for even desktop machine code generation, and precludes the option of supporting iOS bitcode on non-apple hosts.

Okay… so, the thing is that LLVM bitcode format is kinda stable between versions of LLVM. Does apple do something unusual to make that not true, or is the bitcode format used not LLVM altogether (i.e. has apple specific stuff)?

The bitcode format is not stable between versions.

does distributing native binary libraries not work anymore for iOS?

It does work today. It is not the preferred method and it's not obvious it will continue to be supported.

This also potentially means maintaining the LLVM bindings for two LLVM versions (not necessarily two neighbouring minor versions, either). I don't think it's valid to just use an old version of apple-llvm.

@ricky26 We successfully maintain compatibility between several versions of LLVM. As long as Apple's and ours don't drift too far apart it should be doable, but there's always the risk of such great breakage that the divide can't be crossed, and I know there are major API changes coming.

[bluejekyll]

As long as Xcode’s LLVM is not some ancient version, I think we’re good in that regard.

From this page https://gist.github.com/yamaya/2924292:

clang-700.0.72 => LLVM 3.7.0
clang-700.1.76 => LLVM 3.7.0
clang-700.1.81 => LLVM 3.7.0
clang-703.0.29 => LLVM 3.8.0
clang-703.0.31 => LLVM 3.8.0

[nagisa]

The C++ API is unstable, but we use the C API and have had a lot of success supporting multiple versions of LLVM at once. I don't see the difference in terms of API support.

That is not true. We have (quite large!) a number of bindings to C++ APIs in form of rustllvm. There’s a number of cases where we compile that wrapper depending on the version of LLVM compiled against. In case version of LLVM used and compiled against does not match, you’ll get dynamic linker errors, or worse, run into issues at runtime.

precludes the option of supporting iOS bitcode on non-apple hosts.

If Apple does not want to take bitcode generated by anything else than their fork of LLVM, then I don’t see how we could do anything here short of maintaining a similar fork and reverse-engineering their internal patches.

The bitcode format is not stable between versions.

Sure, but it is quite fair to assume¹ that bitcode between various revisions of LLVM known as 3.7.0, for example, is sufficiently compatible for purposes of generating bitcode for consumption by another build of LLVM from 3.7.0 series. Its certainly better than linking to libLLVM dynamically.

¹: especially given that bitcode from 3.2 series is still compatible with 3.8 LLVM, even if its a very small specimen.

[comex]

Some notes:

• Xcode doesn't even ship with a linkable libLLVM (static or dynamic), only a libclang.dylib which statically links to LLVM.
• Apple does ship the source to their 'clang' (including LLVM) under 'Developer Tools' on https://opensource.apple.com, but that site tends to take forever to be updated.
• The story may be better with Swift, which has its own fork of LLVM with release tags - but they may not correspond perfectly to what's shipped with Xcode. (It's possible to use open-source toolchain snapshots with Xcode, but projects built with such toolchains can't be submitted to the App Store.)

[aturon]

cc @rust-lang/compiler

[mitsuhiko]

Would be curious to hear how other programming languages plan on dealing with this. In particular mono and go.

[ricky26]

Unity's answer to this problem is il2cpp - building all of their IL assemblies into C++ code.

Relevant golang bug: https://github.com/golang/go/issues/12682; the suggestion there seems to be that they could use the LLVM go toolchain (which isn't as featured as the standard go toolchain).

All in all, the story for bitcode support outside of Apple is poor.

[mitsuhiko]

Mono proper goes via apple LLVM it would appear: http://tirania.org/blog/archive/2015/Sep-02.html

[mitsuhiko]

One stumbling block will be that you cannot carry inline assembly in bitcode :(

For the mono story I had a quick exchange with Miguel de Icaza about what Mono does for the curious: https://twitter.com/mitsuhiko/status/769458873237434368

[comex]

@mitsuhiko You _can_ have inline assembly in bitcode on iOS and tvOS, but not watchOS, for some reason.

[russellmcc]

Any movement on this? I don't feel at all good about using Rust on iOS without a plan for supporting bitcode. Apple has a history of making optional things like this non-optional fairly suddenly, and indeed bitcode is already required on watchOS and tvOS.

I feel a bit like all of the recommended solutions are somewhat icky. The least messy way I can think of is to allow different targets to overload the codegen behaviour and have the Apple codegen path be in a dynamic crate. (Which would just be the regular codegen path compiled against apple-llvm.)

This approach (by @ricky26) seems to be the most natural to me as a rustc user.

[alexcrichton]

I don't believe anything has changed on this recently to my knowledge at least. With LLVM's recent announcement about versioning they indicated that bitcode should (I think) always be loadable by future versions of LLVM. That may mean that this issue is "solved" on a fundamental level, but it'd still require a more ergonomic interface to extract all the bitcode.

[adrianbrink]

Are there any updates on this?

[samkhawase]

This commenter on HackerNews has succeeded in using Bitcode generated from Rust on macOS and iOS. The thread has some detailed information on how to enable bitcode for rust binaries, which sounds like a great news!

https://news.ycombinator.com/item?id=14305084

[comex]

As the commenter in question, quick notes:

• I used -C lto --emit llvm-bc to get rustc to emit one .bc file containing the current crate and all dependencies. This works, but is essentially a hack; in particular, it doesn't work for C dependencies, including jemalloc (though that isn't used on iOS anyway). It would be better if rustc properly supported emitting Mach-Os with "embedded bitcode"; you can look at the clang source to see how it's done.

• If you want to try it and don't mind the hack, it does seem to 'just work', except for the version issue:

• The main practical obstacle is that Rust syncs with LLVM trunk more frequently than Xcode, which seems to only do so yearly. Newer LLVM versions can load old bitcode, but not the other way around, so you have to either use an old version of rustc or build the latest against an old LLVM. (Actually, rustc 1.17 seems to still work with Xcode 8.x, but not nightly since the LLVM 4.0 upgrade; that's just coincidence though.)

• Ideally Rust would ship official binaries built against a suitable LLVM version. In practice, it seems to be fine to use the correct version of stock LLVM, but if you do want to use Apple's fork:

  • As I said in a previous comment, Xcode doesn't ship LLVM binaries in any linkable form, even dynamically; only clang and libclang.dylib (which exports the Clang API but not the LLVM one).
  • But, as I also said, the source code usually goes up on opensource.apple.com (under Developer Tools -> clang; the archive includes all of LLVM), just somewhat inconsistently/after a delay. However, especially given the expanded backwards compatibility guarantees noted by @alexcrichton, it may not be the end of the world to not always be up to date. Currently, the most recent source release is for Xcode 8.2.1, whereas the latest is 8.3.2.

[comex]

(As a test, I just tried compiling Rust against the Xcode 8.2.1 Clang. rustllvm fails to build, because its nest of #if LLVM_VERSION_GE(..) conditionals, which are meant to allow it to compile against both older and newer LLVM, gets confused by this branch - which claims to be LLVM 3.9svn, but API-wise is somewhere in between 3.8 and 3.9. Probably not worth fixing, since Xcode is almost due for its next yearly refresh anyway.)

[regexident]

Has any progress been made on this front? Alternatively would you (core team) consider this ordeal an issue with a solid permanent fix in sight or do you expect this version hopping thing to remain for the foreseeable future?

I'm considering implementing parts of an iOS app in Rust for its superior type system and low-level semantics and would rather not be stuck with non-bitcode or regular hacks/delays whenever there's an update for either Xcode or rustc.

[arielb1]

@regexident

I think we are going to be implementing swappable LLVM versions for other reasons. If that is implemented, we shouldn't have a problem shipping separate trans for Apple, wasm and everything else.

cc #45684

[burdges]

Isn't Apple's bitcode still really dangerous ala https://medium.com/@FredericJacobs/why-i-m-not-enabling-bitcode-f35cd8fbfcc5 Impossible to verify builds. Issues for cryptography. etc.

[regexident]

@burdges For certain apps this absolutely is a problem.

But for watchOS and tvOS Apple does in fact require Bitcode for App Store submissions. You don't have a choice. Other than "don't do crypto or critical stuff on those platforms". One also has to fear Apple enforcing Bitcode for iOS at some point in the future. I'd rather not have a bricked product at that point.

[michaeleiselsc]

I tried building with Rust 1.24.0 for iOS using the -C lto --emit=llvm-bc trick, but the linker gave the following errors:

Undefined symbols for architecture x86_64:
  "_backtrace_create_state", referenced from:
      std::sys_common::gnu::libbacktrace::init_state::h686c3e443c712b0f in Logger(x86_64.o)
  "_backtrace_syminfo", referenced from:
      std::panicking::default_hook::_$u7b$$u7b$closure$u7d$$u7d$::h598a932d5bb0d80b in Logger(x86_64.o)
      core::iter::iterator::Iterator::position::_$u7b$$u7b$closure$u7d$$u7d$::hbf03153d55553502 in Logger(x86_64.o)
  "_backtrace_pcinfo", referenced from:
      std::panicking::default_hook::_$u7b$$u7b$closure$u7d$$u7d$::h598a932d5bb0d80b in Logger(x86_64.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Any ideas what might be the issue?

Edit: it appears that I can at least suppress that issue by wrapping all the code in each public extern fn with AssertUnwindSafe

[comex]

libbacktrace is a C library which is included in the Rust source tree:

https://github.com/rust-lang/rust/tree/master/src/libbacktrace

You'll have to compile that as bitcode somehow.

[michaeleiselsc]

I'm confused, because I can only find a libbacktrace built for my laptop, but not for the phone OS's. How does rust insert those symbols into the stdlib? Is there an easier way to do this, by somehow disabling backtrace functionality? I removed the RUST_BACKTRACE env var but that didn't help. I've read that you can compile rustc without backtraces, but I was hoping for something easier, e.g. in the cargo.toml file of my project

[comex]

Fundamentally you want to build the rustc source tree while telling it to pass -fembed-bitcode to the C compiler. Ideally you'd just be able to add it to CFLAGS; unfortunately, I just tried that, and it didn't work properly due to two issues with gcc-rs (used by rustc's build system to build C dependencies). However, I got it to work with a bit of a hacky procedure:

cd /tmp/build
cat >ccwrap.py <<END

#!/usr/bin/env python
import os, sys
os.execv('/usr/bin/xcrun', ['clang', '-fembed-bitcode'] + [arg for arg in sys.argv[1:] if arg not in {'-ffunction-sections', '-fdata-sections'}])

END

chmod 755 ccwrap.py
ln -s /usr/bin/ar ar
export CC_aarch64_apple_ios=`pwd`/ccwrap.py
/usr/src/rust/configure --target=aarch64-apple-ios
make

This should get you a libbacktrace.a (in ./build/aarch64-apple-ios/native/libbacktrace/.libs/) with bitcode embedded. You will still have to manually link it. (It does, but this also ends up getting statically linked into libstd.rlib, which is what you really should be linking.) As I mentioned upthread, the 'proper' solution is for rustc to embed bitcode in the __LLVM,__bitcode section of what are otherwise native libraries, the same way clang does: then there would be no need to manually compile and link .bc files.

(Actually, since the error you quoted looks like a normal linker error, I think you could get past it just by linking a regular, no-bitcode-embedded libbacktrace.a. But at best that would result in an output file with broken embedded bitcode.)

The aforementioned issues:

• gcc-rs passes -ffunction-sections and -fdata-sections with no way to turn them off; clang complains about these when combined with -fembed-bitcode (even if -fno-function-sections is passed later).

I filed an issue report, and as a workaround, the above procedure uses a wrapper script that strips out those arguments. But that caused a second problem:

• gcc-rs tries to find ar in the same directory as CC_aarch64_apple_ios (!?); manually setting AR_aarch64_apple_ios had no effect

I couldn't reproduce this with the latest cc-rs (renamed from gcc-rs), so I didn't file an issue. As a workaround, the above procedure symlinks ar into the same directory.

[michaeleiselsc]

Thanks for the suggestions! I actually just needed bitcode for my own code, not the backtrace stuff, so I just used the compiled libbacktrace without any bitcode embedding

[michaeleiselsc]

With the hack I tried, I'm now hitting another issue: dsymutil segfaults when I try to create a dSYM for it. Normal execution without a dSYM works fine, but dSYM creation is broken for some reason. I removed my libbacktrace.a, replaced it with empty placeholder functions, and things worked fine, so it seems that there's an issue with my rust lib.

[comex]

That sounds like a bug in dsymutil, which should be reported upstream to LLVM. Can you try running dsymutil under LLDB and posting a backtrace of the crash? Or to get better debug info, you could try building LLVM in debug mode and reproducing it with that.

(Note that the binary in a stock LLVM installation is called llvm-dsymutil, but these days Xcode's dsymutil is just a symlink to llvm-dsymutil. dsymutil used to be a separate, closed source utility, but for a few years that version has been shipped as dsymutil-classic and not used by default.)

[alexcrichton]

@comex out of curiosity, should the cc crate compile everything with -fembed-bitcode? Or if we're using the "official" clang is it ok to avoid embedding bitcode?

(sorry I'm not super familiar with ios best practices!)

For Rust code itself, would it be best if we just emit bytecode instead of object files? Is that something that Clang/the linker can understand?

[comex]

@alexcrichton

should the cc crate compile everything with -fembed-bitcode? Or if we're using the "official" clang is it ok to avoid embedding bitcode?

Using the official clang doesn't obviate the need for embedded bitcode. Rather, the bitcode gets included - in addition to native code - in the app archive uploaded to Apple, and then Apple can recompile it on the server side, e.g. to optimize for different microarchitectures.

It would probably make sense to pass -fembed-bitcode by default on iOS/watchOS/tvOS. If so, when building in debug mode, -fembed-bitcode-marker should be passed (instead or in addition, doesn't matter); this tells clang to only include a dummy bitcode section rather than any actual bitcode, since embedded bitcode is only needed for final app archives and it slightly speeds up compilation to omit it. Actually, Xcode passes -fembed-bitcode-marker for everything other than building the final app archive, even development builds that happen to have optimizations on - but since cc doesn't have a separate concept of "really final release mode", it should probably just pass it in debug mode. Note that it's safe to embed bitcode even if the final build doesn't require it.

For Rust code itself, would it be best if we just emit bytecode instead of object files? Is that something that Clang/the linker can understand?

Both clang and ld do support raw bitcode files passed as inputs; this is normally used when LTO is enabled (-flto), in which case the .o files clang generates are actually raw bitcode, not Mach-O. (These can still be formed into static libraries using the normal lipo command.) However, this is separate from the "embedded bitcode" format, which consists of a Mach-O with bitcode stuffed into the __LLVM,__bitcode section, in addition to native code in the usual sections. This format is used in object files when LTO is off, as well as in final linked executables or dynamic libraries regardless of the LTO setting. (If LTO is on, the linker is responsible for creating this section; if LTO is off, the linker simply concatenates the __LLVM,__bitcode sections from each object file like it would for any other section, rather than generating a single combined bitcode module.) EDIT: actually, it does something a bit more complicated, generating an xar archive of the bitcode sections and putting them in a section named __LLVM,__bundle; anyway, not something we have to care about.

I think ideally rustc should produce "embedded bitcode" by default on iOS, not raw bitcode. It's not too hard to do, and it avoids breaking incremental compilation - especially in debug mode where you can just produce a dummy bitcode section, ala -fembed-bitcode-marker, but potentially in release mode too, since the linker just stuffs the objects' bitcode sections together rather than doing anything expensive at link time.

Here's how clang generates embedded bitcode:

https://github.com/llvm-mirror/clang/blob/master/lib/CodeGen/BackendUtil.cpp#L1242

Relatively simple - before invoking the backend, it dumps the current LLVM module as bitcode, then
stuffs that as binary data into a new global variable (in that same module!), placed in the __LLVM,__bitcode section. It seems like kind of a hacky design (why does the frontend have to do this manually?), but it shouldn't be too hard to reproduce in rustc.

[michaeleiselsc]

@comex that approach makes sense, although I think there are reasons (see #48833 that I just made) for using Apple's clang as the backend, while we're thinking about changes like this. Also, how easy is it to compile ourselves from the embedded bitcode?

[comex]

@michaeleiselsc Sorry, I don't understand what you mean by "compile ourselves from the embedded bitcode".

[michaeleiselsc]

As in, instead of just giving a binary to the App Store with bitcode embedded, there are also benefits to taking the bitcode and building it from that ourselves using Apple clang. Two benefits that I'm interested in are the downstream fixes that Apple clang has and the ability to use clang sanitizers, e.g. the coverage sanitizer.

[alexcrichton]

@comex ok so digging in a bit, it looks like the implementation in rustc is easy enough (relatively), but I've got a few questions:

• Why is -fembed-bitcode-marker a thing? Using clang locally it just emits an empty static into a section. Does that mean the mere presence of the section means something? Does some tool along the road in iOS break if the section isn't there? (but it doesn't actually look inside the section because it's empty?
• Do you know if the __cmdline section is necessary? It looks like -fembed-bitcode also embeds some form of a commandline into the binary. I'm not really sure what we would put here as it's probably not rustc's command line, but does any tool look at this? Does compilation break if -fembed-bitcode=bitcode is used with iOS?
• And finally, is there a reason to ever want to disable this? Or should we just unconditionally enable this for the iOS targets and move on?

[mitsuhiko]

Why is -fembed-bitcode-marker a thing?

I think its used by xcode to indicate that something is supposed to use bitcode but bitcode was not actually compiled. If you try to archive such a project later ld will fail with an error.

[michaeleiselsc]

To the question of whether we want to always enable this, there are definitely some cases we'd want to be careful about. For example, if someone submits it to a beta app distribution platform like HockeyApp, will HockeyApp strip that bitcode section, or will beta testers have to download a much larger binary than they need to?

[comex]

@mitsuhiko Yep, the idea is that the linker can verify that the build is properly set up for bitcode, even if you're currently doing testing builds that don't actually need bitcode.

For example, if you compile a C file without mentioning bitcode, then try to link it with -fembed-bitcode-marker, you get:

$ clang -c -o test.o test.c
$ clang -dynamiclib -o test.dylib test.o -fembed-bitcode-marker
ld: warning: all bitcode will be dropped because 'test.o' was built
without bitcode. You must rebuild it with bitcode enabled (Xcode
setting ENABLE_BITCODE), obtain an updated library from the vendor,
or disable bitcode for this target.

The warning goes away if you pass -fembed-bitcode-marker on the first line as well.

@alexcrichton As for __cmdline… hmm… it seems like the linker requires it to be present (search for "Create bitcode"):

https://opensource.apple.com/source/ld64/ld64-274.2/src/ld/parsers/macho_relocatable_file.cpp.auto.html

But I don't see anything that cares about the actual value. Probably best to put a dummy value there.

[chrisballinger]

Was this issue resolved by merging #48896?

[chrisballinger]

I've been playing around with this on the nightly builds, and it works great until you try to Archive with Xcode. It looks like it's adding fembed-bitcode-marker but not fembed-bitcode to the Rust static library output. It's compiled with cargo lipo --release with:

$ cargo --version
cargo 1.27.0-nightly (af3f1cd29 2018-05-03)
$ cargo lipo --version
cargo-lipo 2.0.0-beta-2
$ rustc --version
rustc 1.27.0-nightly (565235ee7 2018-05-07)

ld: '/Users/chrisbal/Documents/Beach/rust-universal-template/target/universal/release/libexample.a(example_generic-be72fb1769c1779b.example_generic6-152d14edfb6970f54250733c74e59b7.rs.rcgu.o)' does not contain bitcode. You must rebuild it with bitcode enabled (Xcode setting ENABLE_BITCODE), obtain an updated library from the vendor, or disable bitcode for this target. for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

When I run otool -arch arm64 -l /Users/chrisbal/Documents/Beach/rust-universal-template/target/universal/release/libexample.a | grep bitcode I get a lot of sectname __bitcode, but I don't know how to check if this is actual bitcode and not just markers.

I'm using nested crates so maybe the inner "generic" crate isn't getting the bitcode flags?

edit: here is our repo demonstrating the issue https://github.com/Raizlabs/rust-universal-template/tree/879e7412d729e8963586c5b083d51b09733aec32

[volodg]

@chrisballinger, it works with additional flag, see
RUSTFLAGS="-Z embed-bitcode" cargo lipo --release

$ cargo --version
cargo 1.28.0-nightly (f352115d5 2018-05-15)
$ cargo lipo --version
cargo-lipo 2.0.0-beta-2
$ rustc --version
rustc 1.28.0-nightly (952f344cd 2018-05-18)

But I have another compile error only for arm7 arch:

.rs.rcgu.o)' does not contain bitcode. You must rebuild it with bitcode enabled (Xcode setting ENABLE_BITCODE), obtain an updated library from the vendor, or disable bitcode for this target. for architecture armv7

Does anybody know how to fix arm7 arch?

[4lex1v]

Sadly can't get it working as well, getting the same error as @chrisballinger:

ld: '/sandbox/target/universal/release/librgame.a(std-da6dba40351cda22.std3-d36cd881bae00a8b5fc36289b5737f78.rs.rcgu.o)' does not contain bitcode. You must rebuild it with bitcode enabled (Xcode setting ENABLE_BITCODE), obtain an updated library from the vendor, or disable bitcode for this target. for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

I've tried to compile it with RUSTFLAGS="-Z embed-bitcode" cargo lipo --release and setting it in .cargo/config. It adds the flag though:

` Executing: "cargo" "build" "--target" "x86_64-apple-ios" "--lib" "--features" "" "--color" "auto" "--release" "--verbose" Compiling libc v0.2.42 Compiling rand_core v0.2.1 Running `rustc --crate-name libc /Users/aleksandrivanov/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.42/src/lib.rs --crate-type lib --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="use_std"' -C metadata=dce309634355ac97 -C extra-filename=-dce309634355ac97 --out-dir /Users/aleksandrivanov/Sandbox/Projects/tetris/rgame/target/x86_64-apple-ios/release/deps --target x86_64-apple-ios -L dependency=/Users/aleksandrivanov/Sandbox/Projects/tetris/rgame/target/x86_64-apple-ios/release/deps -L dependency=/Users/aleksandrivanov/Sandbox/Projects/tetris/rgame/target/release/deps --cap-lints allow -Zembed-bitcode` ...

[rex-remind101]

Any updates here? Heavily considering a shared core library for our app and without this fixed c++ is the likely option.

[volodg]

@chrisballinger, Can it be related to https://github.com/rust-lang/rust/issues/52686?

[chrisballinger]

@volodg Nope, this issue is from a while ago.

[yageek]

With the merge of #48896, is it supposed to work out of the box?

[appaquet]

I'm on 1.29.1 and objects still don't have embedded bitcode by default. As mentioned earlier by @volodg, you can get rust to embed them using RUSTFLAGS="-Z embed-bitcode". But, from what I experienced, you then get into issue where Rust's own libs (compiler_builtins, std) are not compiled with embed-bitcode. Perhaps rebuilding iOS targets using xargo with embed-bitcode would work, but I didn't try.

[yageek]

I tried to use xargo as you proposed but the problem seems to still be here :(

Using this Xargo.toml

[dependencies]
std = {}
[target]
features = ["jemalloc"]

Also adding this to Cargo.toml:

[profile.release]
panic = "abort"

I compiled using xargo build --release --target $TARGET for all those targets:

• aarch64-apple-darwin
• armv7-apple-darwin
• armv7s-apple-darwin
• i386-apple-darwin
• x86_84-apple-darwin

Then I used lipo to create one static library.

I still have a linker error:

ld: '../../cargo/target/universal/libgreetings.a(greetings-ceeec73d35f7dbe0.greetings.9kcaav8v-cgu.2.rcgu.o)' does not contain bitcode. You must rebuild it with bitcode enabled (Xcode setting ENABLE_BITCODE), obtain an updated library from the vendor, or disable bitcode for this target. for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

[appaquet]

Got a bit further with the xargo path.

My Xargo.toml

[dependencies]
std = {}

My Cargo.toml

[profile.release]
panic = "abort"

I don't kwow if it's required, but from xargo doc, you need to explicitly use extern crate compiler_builtins, which I added to my lib.rs.

I then compiled with: RUSTFLAGS="-Z embed-bitcode" xargo build --target $TARGET --release to compile. Make sure it builds core/std/compiler_bultins. xargo clean doesn't cleanup previous builds correctly from what I experienced, so a rm -rf ~/.xargo was necessary to recompile std when I tried different Xargo.toml tweaks.

But then, archiving in Xcode 10, I get (at armv7 linking):

Intrinsic has incorrect argument type!
void (i8*, i8, i32, i1)* @llvm.memset.p0i8.i32
Intrinsic has incorrect argument type!
void (i8*, i8*, i32, i1)* @llvm.memcpy.p0i8.p0i8.i32
Intrinsic has incorrect argument type!
...
(lots of it)
...
LLVM ERROR: Broken module found, compilation aborted!
clang: error: linker command failed with exit code 1 (use -v to see invocation)

[yageek]

Does it means the bug relies on the LLVM's side?

EDIT: How could we help to advance in this issue?

[mpiannucci]

Has anyone tried this recently? It looks like armv7 is the biggest issue?

[jlouazel]

Hi everyone. I have been following this thread for a while now and I am very excited to see the Apple bitcode being supported with Rust. Has anyone been able so far to make this work? What is currently missing and how could we help?
Thanks!

[appaquet]

After a few hours of hunting down this issue, I found the reason Rust's bitcode embedding is not working in Xcode. The short answer is that Xcode's LLVM version is expecting LLVM 6.0 bitcode, while Rust has bumped to LLVM 7.0 in July 2018.

Now, the long answer...

As we can see in LLVM 7.0's changelog, they changed signatures for @llvm.memcpy, @llvm.memmove, and @llvm.memset, and Xcode build error was clear about the wrong argument type as seen in my previous comment.

By the way, Kotlin Native added support for embed-bitcode in October 2018 and is apparently working fine in Xcode. The reason is that Kotlin still uses LLVM 6.0.

Seeing that, I tried to compile using an older version of Rust before the bump to 7.0. I tried with nightly-2018-06-01-x86_64-apple-darwin, and success, it compiles !

I also tried to compile without using Xargo with no success. All dependencies need to be compiled with bitcode embedding.

So bottom line, unless Apple bumps their LLVM version, I don't think we'll see support for embedded bitcode anytime soon... Unless someone finds a clever way to convert embedded bitcode to 6.0's bitcode...

[chrisballinger]

@appaquet Thank you for getting to the bottom of it! It was quite the mystery for me and I don't think I ever was going to dive as deep as you did. Good to know that it should likely start working once Apple bumps the LLVM version.

[jlouazel]

Thanks @appaquet for your clear answer. Now I guess we just have to rely on Apple to do the right thing

[oleksandr-yefremov]

Sideview from somebody who has interest in multiplatform for mobile and compared most of the promising solutions (React Native, Flutter, Kotlin/Native, Rust)

React Native. Supports bitcode from inception, because essentially bridge is written in C++ and is just interpreting JS code in runtime.

Flutter. Does not support bitcode yet - https://github.com/flutter/flutter/issues/15288

Kotlin/Native. JetBrains is considering mobile platforms as a priority and although it took them a while and it's only bitcode markers (not full bitcode yet), it's good enough to start working with - see https://github.com/JetBrains/kotlin-native/pull/1564 and https://github.com/JetBrains/kotlin-native/issues/1202#issuecomment-444022513

Rust. Supports bitcode for LLVM 7, whereas Apple uses version 6.

If I want to have both business logic and UI crossplatform, then I would go with React Native or Flutter. But for many, many serious apps this path is too risky. React Native is not performant enough and API and dependency stability is an issue (is v1.0 ever gonna come?). Flutter is a bit immature but seems to gain more and more traction.

Now if I want to share only business logic in some kind of modules (like earlier big apps did with C++ and JNI/Obj-C++) and then build truly native UI on top, in the beginning I have a choice between all four. Then I cross out React Native, because taking full blown JavaScriptCore and bridge to run business logic between JS<->Native seems a bit over the top (initializing this bridge is quite expensive, too). Flutter potentially can be used but is not intended that way, so again I'd end up using UI framework for running business logic. Plus it does not support bitcode.
Rust and Kotlin/Native are both aiming for this niche, have decent tooling, produce native binary (performance and footprint!). Bitcode is a big problem for those who want to adopt Rust as a language for building multiplatform modules on mobile platforms. And now Kotlin/Native has an advantage.

To sum up why bitcode is very important for iOS devs. Can't write anything on tvOS or watchOS. Many 3rd party frameworks which are distributed as binaries have bitcode enabled (dozens of examples, from the top of my head Google IMA - most popular video advertisement). Some security audits require it. And finally, whenever Apple says "starting next year we don't accept apps without fully embedded bitcode anymore", everybody gets a ticking timer.

Now I guess we just have to rely on Apple to do the right thing

Yeah, right, this is what Apple is notorious for ;)
I don't know what would be the best solution (temporarily downgrading to LLVM 6 is not option, right?), but Rust is losing iOS and mobile crossplatform developers completely.

[jlouazel]

Thanks you so much for this detailed explanation @oleksandr-yefremov. I've been reading a bit on the subject and I found this interesting comment: https://gist.github.com/yamaya/2924292#gistcomment-2738480

The version of LLVM is tied to the version of Swift used in XCode. So far the following support was applied:

Xcode 8.3  --> swift 3.1 --> llvm 4.0.0
Xcode 9.0  --> swift 4.0 --> llvm 4.0.0
Xcode 9.3  --> swift 4.1 --> llvm 5.0.2
Xcode 10.0 --> swift 4.2 --> llvm 6.0.1

While looking at the swift-5.0-branch, I noticed the declared version of LLVM in it is 7.0.0: https://github.com/apple/swift-llvm/blob/swift-5.0-branch/CMakeLists.txt#L25-L33

I don't know if there are further blockers here, but it seems to me we might be able to use Rust to output Bitcode binaries 🎉

[vincentisambart]

It seems Xcode 10.2 will include Swift 5.0 and should be indeed be released pretty soon. But on the other hand LLVM 8.0 is scheduled to be released next week. Rust will pretty likely upgrade before Apple starts using it.

For Rust to properly support bitcode, we would need the Apple ARM architectures builds of Rust to use a pinned version of LLVM (maybe from the https://github.com/apple/swift-llvm repo – Apple doesn't seem use the released LLVM package but their own branches). That version of LLVM would only be updated when Apple releases a new final version of Xcode with a different LLVM version.

Btw, it seems the latest Rust is buildable with LLVM 6 so that definitely seem doable: https://github.com/rust-lang/rust/blob/master/src/bootstrap/native.rs#L282
You would definitely need people stepping in when an issue is created to update the required version of LLVM (the one for LLVM 6: https://github.com/rust-lang/rust/issues/55842).

[mati865]

@vincentisambart Rust has been already using LLVM 8 for nightly builds for 5 months IIRC.
LLVM 6 is minimal supported version if you want to build it yourself and it's tested on CI: https://github.com/rust-lang/rust/blob/706e67b0a0143d651eb03f2fa2c30645899e81ff/src/ci/docker/x86_64-gnu-llvm-6.0/Dockerfile

[alper]

Now I guess we just have to rely on Apple to do the right thing

I've been chasing something along these lines for a while (and have now exited) and I'm guessing platform-wise it does not make sense for Apple to do this, because:

• Swift wants to be Rust in a bunch of ways (see the ownership manifesto for one) but is lagging behind a lot
• Apple needs Swift to be and remain their main platform language so it does not make sense for them to facilitate another big language on their platform nor for them to push Swift into other environments

[vincentisambart]

Rust has been already using LLVM 8 for nightly builds for 5 months IIRC.

Then even if Apple starts supporting LLVM 7 bitcode, the bitcode generated by recent official builds of Rust might not work. Even if it does, it might break the next time Rust moves to a version of LLVM with an incompatible change in the generated bitcode.

LLVM 6 is minimal supported version if you want to build it yourself and it's tested on CI

Then having the official *-apple-ios builds of Rust be built with LLVM 6 (or better something like https://github.com/apple/swift-llvm/releases/tag/swift-4.2.2-RELEASE – if that works) might fix the bitcode problems.

[sercand]

I think Rust currently ships with two LLVM backends (Emscripten and standard) after merging of https://github.com/rust-lang/rust/issues/46819.
Although #46819 mentions ios LLVM backend for bitcode it did not implemented.

[mitsuhiko]

Not to derail this entirely but wouldn't it overall make more sense to try to enable transpiling to C for such cases? It's unlikely that Rust will be able to stay up to date with the bitcode requirements from Apple's side.

[jlouazel]

Even if there are indeed potential sources of incompatibilities, it opens the door for tests where we could try to compile with a compatible older version of Rust, one that embeds the right LLVM version for Apple.

In a complete cross-platform approach, this is still quite blocking obviously as we would need to be able to compile the same source code both from the latest version of Rust to benefit from the latest improvements and also to be able to compile from an older version of Rust, specifically to output bitcode compatible binaries for the Apple ecosystem. That's doable, but not suitable.

I'm still pretty excited to see this happening as it will at least open the doors for more people to test this and push us all forward in the right direction.

[iragsdale]

Speaking as someone who is right now trying to decide on an approach for writing cross-platform mobile code for a fairly large team, we literally cannot use Rust if we can't generate bitcode. One of our apps relies on bitcode to be able to generate a small enough binary to download over cell networks, and we have some large customers that insist on that.

Based on everything I see about Rust I think it has the most long-term upside of any of the possibilities I see out there (Kotlin Multiplatform, cross-compiling Swift, Mono, Dart, C++, React Native), but would definitely need bitcode to be fully supported, even if that meant not being able to keep up with the latest Rust releases.

[mpiannucci]

Even if bitcode worked, Rust still can't target watchOS or tvOS anyways can it?

[arielb1]

It's unlikely that Rust will be able to stay up to date with the bitcode requirements from Apple's side.

The problem here is that Rust's LLVM is too new, not that it is too old. I agree that we should have a better compatibility story here.

[Pacoup]

How feasible is it for rustc to support multiple LLVMs?

Related: Support tvOS and watchOS (and simulator) targets

[CryZe]

It already uses multiple LLVMs. For the emscripten targets it uses emscripten's LLVM fork that is on LLVM 6 atm.

[mitsuhiko]

The problem here is that Rust's LLVM is _too new_, not that it is _too old_. I agree that we should have a better compatibility story here.

The issue is not that it's too old or too new but that even if rust matches the LLVM version, at any iOS release this can break again. Apple (as far as I know) does not in any way guarantee any stability with regards to bitcode.

[iragsdale]

The issue is not that it's too old or too new but that even if rust matches the LLVM version, at any iOS release this can break again. Apple (as far as I know) does not in any way guarantee any stability with regards to bitcode.

In practice we see major Xcode updates once a year when new iOS versions come out. When this stuff happens developers get a pretty clear heads up that they have to start submitting apps with a newer version of Xcode, so it is rarely surprising. I wouldn't say this is particularly hard to plan for.

[mitsuhiko]

I wouldn't say this is particularly hard to plan for.

Since there is absolutely no guarantee around bitcode at all that could turn into something very hard to plan for. Imagine they would decide from one release to another to use their own bitcode format that does not end up in an open source release of LLVM.

[iragsdale]

Since there is absolutely no guarantee around bitcode at all that could turn into something very hard to plan for. Imagine they would decide from one release to another to use their own bitcode format that does not end up in an open source release of LLVM.

If we're just going to assume Apple is going to make breaking changes without regard to anyone else, sure. Having some more clear documentation around their plans for Bitcode doesn't improve that much if we assume that they are malicious or just plain don't care.

But they have clearly made Swift an important part of their platform and many thousands of developers depend on it, including their own teams and some of their most important partners. Swift is based on the fully open source LLVM and is itself being developed fully in the open, including the toolchain that Xcode uses to generate the bitcode apple accepts.

So, for them to just suddenly change that format without anybody noticing or caring would require them to take both their Swift and LLVM repositories private, right? It is certainly possible, but it doesn't seem super likely to me.

If the Rust team decides that the risk of that is a good reason not to support bitcode, that's certainly their prerogative. I'd be a bit sad about that, but it's not my place to tell others where to spend their time.

[sercand]

Since there is absolutely no guarantee around bitcode at all that could turn into something very hard to plan for. Imagine they would decide from one release to another to use their own bitcode format that does not end up in an open source release of LLVM.

Apple releases Xcode Beta around the beginning of June and the final version in September. After 6-9 months later it stops accepting apps build with older Xcode versions to Appstore.

I think there is enough time to prepare bitcode changes during that timeline.

[appaquet]

Just tested with Xcode 10.2 beta 3 (which includes Swift 5 w/ LLVM 7) and I could link with a Rust recent nightly with embedded bitcode. Obviously, this only works with iOS targets, since Rust doesn't have watchOS/tvOS targets.

I also agree that Rust needs to be able to use Apple LLVM fork for all Apple related targets in order to maintain bitcode compatibility.

[Pacoup]

Yeah, the beta release window sounds reasonable. I think the idea that Apple is gonna close source LLVM or their bitcode format is very unlikely, but Apple requiring bitcode in future iOS releases definitely sounds like it will happen. And who knows, maybe they'll even start requiring bitcode for Mac App Store submissions with project Marzipan.

The kind of non-guaranteed way of dealing with compatibility being proposed is admittedly icky, but I think mobile platforms should eventually become Tier 1 platforms, especially for cases where you would have used C++ otherwise, like for video game development. Plus, you could also develop games with Rust for the Apple TV.

I'm hoping an officially sanctioned approach by the Compiler team would encourage someone to work on this, but I think an earlier comment by @brson sums up the hesitation:

We successfully maintain compatibility between several versions of LLVM. As long as Apple's and ours don't drift too far apart it should be doable, but there's always the risk of such great breakage that the divide can't be crossed

Or maybe bitcode will stabilize in the future and we'll all forget this debate ever happened 🙏

[Pacoup]

I will eat my hat now 😞

In which I said:

the idea that Apple is gonna close source LLVM or their bitcode format is very unlikely

Which was mostly in response to the distrust expressed by @mitsuhiko:

Imagine they would decide from one release to another to use their own bitcode format that does not end up in an open source release of LLVM.

But if you look at issue #48833, there is definitely a precedent. As @comex previously wrote:

there have been cases where features show up in Xcode's LLVM before LLVM trunk - such as the entire arm64 port, when it was originally developed, because Apple wanted to keep it secret that they were planning to ship arm64 devices

And @michaeleiselsc's story from the same issue:

an app I was working on got hit hard by random restarts in December 2016, and it's caused by a specific issue that open-source LLVM only fixed December 2017. When we moved from open-source LLVM to Apple's LLVM in December 2016, the problem was fixed

Given that the idea is using Apple's LLVM with Rust, it doesn't sound that bad, but there is definitely a risk of having the occasional nasty bug because of diverging implementations 😢. It's not a risk all projects can shoulder. Transpiling does sound like a better option at this point, but so far, it doesn't sound all that possible.

I still think using Apple's LLVM should be supported, but there should be clear warnings in the documentation explaining there are no guarantees things will keep working (A Tier 1.5 or something).

[tmm1]

Kotlin/Native. JetBrains is considering mobile platforms as a priority and although it took them a while and it's only bitcode markers (not full bitcode yet), it's good enough to start working with - see JetBrains/kotlin-native#1564 and JetBrains/kotlin-native#1202 (comment)

Thanks for pointing this out @oleksandr-yefremov! I dug in a bit further and was able to replicate the kotlin/native approach in golang. I believe you folks should be able to do the same, which would allow for using rust in bitcode-enabled iOS/tvOS/watchOS apps without necessarily emitting bitcode.

[volodg]

Does bitcode work now? Did anybody try?

[amrox]

@volodg I'm a rust novice. However, I ran through this tutorial with the latest rust nightly (at time of writing rustc 1.37.0-nightly (088b98730 2019-07-03)) and it did NOT work.

It seems like the markers are there..

$ otool -arch arm64 -l librust.a  | grep bitcode
  sectname __bitcode
...

But I get the following error when building for iOS device (simulator works):

ld: '/Users/amrox/Documents/Projects/rust-ios-example/hello-rust/libs/librust.a(rust-e6011ffb55678675.rust.8yq9vjk7-cgu.3.rcgu.o)' does not contain bitcode. You must rebuild it with bitcode enabled (Xcode setting ENABLE_BITCODE), obtain an updated library from the vendor, or disable bitcode for this target. for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

That's as far as I got.

[saturday06]

I succeeded to link rustc nightly-2019-09-05 aarch64-apple-ios and rustflags = "-C lto -Z embed-bitcode" staticlib with clang-1100.0.33.5 (Xcode 11 beta 7) and -fembed-bitcode app. Source code is https://github.com/saturday06/rust-ios-bitcode-test .

[thombles]

• Do you know if the __cmdline section is necessary?

Two years later I can answer this - for deploying to physical devices or creating Xcode archives an empty cmdline is fine, however for App Store submissions Apple performs validation of the clang command line. I’ve put a longer description in this PR, which includes a hacky patch to make it work: https://github.com/getditto/rust-bitcode/pull/7

I’d like to upstream this somehow but the options are not appealing. Do we make up some clang options when targeting iOS? Do we provide an envvar to choose exactly how to lie in case Apple changes their validation rules? I’d be happy to make the change if there’s a reasonable choice here.

[Narniab]

-Cembed-bitcode=no does not work for ios target. there are still bitcode section in .a file
how to build without bitcode?