Conan: iOS/OSX universal binaries handling

[SSE4]

universal (fat) binaries are pretty common for Apple platforms - that's just binaries for several architectures combined together.

sorry for the long text - but I am trying to describe my use case as detailed as possible :)
e.g. for OSX it's common to have x86 + x64 universal binaries, while for iOS it's usually armv7 + armv7s + arm64/armv8 combined together, plus optionally x86 and x64, if iOS simulator is requied (so from 3 to 5 architectues). same story for other Apple platforms, such as watchOS and tvOS.
usually, universal binaries are produced by running multiple builds and then running lipo utility on resulting shared/static libraries. there is alternate approach to run single universal build, but it tends to be complicated and error-prone with configure (autotools) style projects - e.g. such configure scripts need to detect size of pointer, which is ambiguous in case of universal binaries (sizeof(void*) - 4 or 8?).

I need some advice on how to proceed with universal binaries in conan. there are several approaches how it could be done:

1. conan receipt invokes N builds for N architectures and runs lipo to combine as postprocessing.
   this is the way I am currently using. typical conan file may look like:

if self.settings.os == "Windows":
   self.build_with_msvc()
else:
   self.build_with_configure()

and then for OSX/iOS it will become much more complicated, like:

if self.settings.os == "Windows":
   self.build_with_msvc()
elif self.settings.os == "iOS":
   arches = ["armv7", "armv7s", "arm64", "i386", "x86_64"]
   for arch in arches :
        self.build_with_configure(arch, "iphoneos")
    self.lipo(arches)
elif self.settings.os == "Macos":
   arches = ["i386", "x86_64"]
   for arch in arches :
        self.build_with_configure(arch, "macos")
    self.lipo(arches)
else:
   self.build_with_configure()

there are few disadvantages of such approach:

• conanfile has very special path for iOS and Macos, although build process is same as for other NIX* platforms
• lots of code copy-pasted from package to package for universal binaries handling
• package "arch" setting is no longer showing the truth, moreover, it's confusing field in case of universal binaries

1. build N conan packages for each architecture and then combine them somehow
   the idea is to build conan packages for each architecture (armv7, arm64, etc) as usual and then somehow combine them into aggregated multi-architecture conan package. I am not sure if it's even possible now to have some approach for conan package to support multiple settings at the same time, but probably there are more use cases for such combined packages (e.g. Windows tools which may run on both x86 and x64, but not on ARM might have combined x86+x64 indication). probably this approach ends up in some new conan command line, like "conan combine" or "conan lipo" which will run lipo for all libraries from packages to be combined.

some unclear points about this approach:

• while binaries are in general easy to handle, headers might be tricky part if they are different for different arches (in my practice I just copy headers from sources, but some headers might be generated from the build process and contain definitions like SIZE_OF_VOIDP)
• anyway, conan somehow needs to know which binaries to process and where are they located (but in worst case, conan still can wildcard *so *a *dylib)
• some convention on how to identify and describe combined packages shall be developed
• new conan command will be too OS-specific which might be not good (or useless for other arches?)
• probably issues with conflict resolution may appear - e.g. if asked for x86 arch, what to install, x86 or x86+x64 package?

[memsharded]

cc/ @tru

[memsharded]

package "arch" setting is no longer showing the truth, moreover, it's confusing field in case of universal binaries

Yes, I think in that case (approach 1), the arch setting should be removed from the settings. Exactly the same, as if packaging a multi-config (debug/release) package, you should remove the build_type from settings.

Logic for having packages that matches different settings can be done in the package_id() method, for example:

def package_id(self):
      if "arm" in self.settings.arch:
             self.info.settings.arch = "AnyARM"

This will build one package for any arm architecture, and one package for x86, and another for x86_64 and so on.

I like the approach (1), it is explicit, reads well. You say it is much more complex, but the intention is so evident and explicit, you can perfectly know what the recipe is doing. And it is just a few lines. So IMHO, it will be more robust and easy to maintain that other approaches. If sharing code, maybe reusing such code via shared python packages could be an option.

You can see that the explicit build of different configurations is what I am proposing for multi-config (debug/release) packages in https://github.com/conan-io/docs/pull/161/files#diff-e4c22b2beb97bec45bf31f8a032893afR120 (PR to the docs, with some new 0.20.0 features)

[SSE4]

@memsharded sounds good, I am going to try approach with package_id()
for the code sharing, I am not sure if such python package can be useful outside of conan, what do you think if I contribute to conan.tools adding several helper functions for dealing with Apple-specific stuff, just like tools.vcvars_command?

[memsharded]

Yes, those utility functions can be added to tools, are the same concept as vcvars_command helper, so they will be easy to use and useful for more people there. So, sounds just perfect, go ahead.

[tru]

I am very interested in this considering I was pestering @memsharded about this the other day. I think the tricky parts are the following:

1) how to save away the binaries between each build phase to have them easily available for the lipo step
2) how to clean out the source directory so that nothing is left to break the next build. Ideally we would be able to have one build dir per arch - that would solve that.

Just spitballing here, but the "perfect" setup would actually be if conan could do the following:

• run source once
• for arch in archs:
  
  
  
  • copy source to build-arch
  
  
  • run build
  
  
  • run package function (to copy all lib headers etc)
  
  
• run combination method that runs lipo on all the artifacts collected in package function.

This could be done by adding a new property variants at a toplevel of the conanfile and then set self.current_variant or something like that between each run. And then add a new post_package function to allow for lipo combination or other post-processing before it's packaged up.

This would make the conanfile much much nicer since it can reuse the same methods without a lot of extra code for one specific platform.

If self.current_variant is not good enough I think we could allow build and package to take a new variable if variants are defined. There would have to be some runtime checking of code to make sure it's backwards compatible.

[tru]

Here is a mock example of how it could look:

class MultipleVariant(ConanFile):
  name = "testmultiple"
  version = "1.0"
  settings = "os", "compiler", "build_type", "arch"

  def configure(self):
    if self.settings.os == "iOS":
      self.variants = ["arm7", "arm64", "x86_64"]
    elif self.settings.os == "MacOS":
      self.variants = ["x86", "x86_64"]

  def source(self):
    tools.download("http://foo.com/bar.bz", "bar.bz")
    tools.unzip("bar.bz")

  def build(self, variant):
    self.run("./configure --arch={0}".format(variant))
    self.run("make")

  def package(self, variant):
    self.copy("*.dylib", src="lib", dst="tmp/" + variant)

  def combine_package(self):
    if not self.variants:
      return

    libs = [l for l in os.listdir("tmp/x86_64") if l.endswith(".dylib")]
    for lib in libs:
      self.run("lipo -o lib/{0} {1}".format(lib, [os.path.join("tmp", var, lib) for var in self.variants]))

    shutil.rmtree("tmp")

[SSE4]

@tru that sounds like a good addition as well. currently in my scripts I am building into the $TMPDIR/{arch}, which is obviously outside of conan source directory. adding variants and post-processing method to the conanfile sounds like a great idea.

[tru]

This can be used for much more than just iOS/MacOS binaries as well. I have at least one other package where I need to build it twice right now that could benefit.

[tru]

For extra points I guess variants could be set on a global level in the profile or something and the packages can choose to use it by adding the variant argument to the build function. Since most likely it's usually controlled on a global level.

[lasote]

This can be used for much more than just iOS/MacOS binaries as well. I have at least one other package where I need to build it twice right now that could benefit.

Why do you need to build twice that package? Remember that we support in 0.20.0 the build_id method. http://docs.conan.io/en/latest/reference/conanfile.html#build-id

For extra points I guess variants could be set on a global level in the profile or something and the packages can choose to use it by adding the variant argument to the build function. Since most likely it's usually controlled on a global level.

I'm not sure about a parallel implementation to support variant overriding. It could be done with envionment variable, remember that now you can use profiles to assign package level variables:

Profile:

[env]
testmultiple:TESTMULTIPLE_IOS_VARIANTS = "armv8,x86_64"
testmultiple:TESTMULTIPLE_MACOS_VARIANTS = "x86"

Conanfile:

def configure(self):
    if self.settings.os == "iOS":
      self.variants = (os.environ.get("TESTMULTIPLE_IOS_VARIANTS", None) or "arm7,arm64, x86_64").split(",")
    elif self.settings.os == "MacOS":
      self.variants = (os.environ.get("TESTMULTIPLE_MACOS_VARIANTS", None) or  "x86,x86_64").split(",")

I like the idea, but I need to think about possible problems implementing this variants feature.

[memsharded]

This seems like the opposite behavior of the one implemented with the new build_id(), which basically allows to build once, package many. The request would be, build many, package once. Will think about it too.

[lasote]

HI, I've been talking with @memsharded and we are not sure about the variants. Please, look at this example code and tell me the problems that you see:

class MultipleVariant(ConanFile):
    name = "testmultiple"
    version = "1.0"
    settings = "os", "compiler", "build_type", "arch"

    def package_id(self):
        if self.settings.arch in self.variants:
            self.info.settings.arch = ",".join(self.variants)

    def source(self):
        tools.download("http://foo.com/bar.bz", "bar.bz")
        tools.unzip("bar.bz")

    def build(self):
        for variant in self.variants:
            variant_dir = self.variant_dir(variant)
            mkdir(variant_dir)
            copy_build_files_to(variant_dir, exclude="tmp_*")  # <= Pseudocode, could provide some tool
            with tools.chdir(variant_dir):
                self.run("./configure --arch={0}".format(variant))
                self.run("make")

    def package(self, variant):
        for variant in self.variants:
            variant_dir = self.variant_dir(variant)
            self.copy("*.dylib", src="%s/lib" % variant_dir, dst="tmp/" + variant)
        self.combine_package()

    @property
    def variants(self):
        if self.settings.os == "iOS":
            return ["arm7", "arm64", "x86_64"]
        elif self.settings.os == "MacOS":
            return ["x86", "x86_64"]
        return [self.settings.arch]

    def variant_dir(self, variant):
        return "tmp_%s" % variant

    def combine_package(self):
        if not self.variants:
            return

        libs = [l for l in os.listdir("tmp/x86_64") if l.endswith(".dylib")]
        for lib in libs:
            self.run("lipo -o lib/{0} {1}".format(lib, [os.path.join("tmp", var, lib) for var in self.variants]))

        shutil.rmtree("tmp")

We think that it's more explicit and do not introduce more complexity in the conan model/processes.
I didn't touch your combine_package method, the variants property is almost the same code, the package_id method is needed also in your example, and the build method is not too much complex, we could provide some tool to ease the sources copy. WDYT?

[tru]

Yeah I don't see a huge problem with what you layed out there. The only thing that I need to dig into is how easy it is to copy the sources without doing it a great many times. I could centralize some of those functions into my own conantools package as well so I think it will be OK.

[lasote]

Ok, feel free to share with us your tool to copy the sources, we can put it in the conans.tools package.

[SSE4]

by the way, usually it even is't required to copy sources - e.g. with most configure-style projects "--prefix" can be used to ensure "make install" copies headers, libraries and other stuff into the required directory, and "make clean"/"make distclean" ensures that working copy has no leftover after the previous build(s).

[tru]

It's a great theory - but I have seen many problems with distclean not cleaning out extra files. I probably want to copy sources anyway to be 100% sure.

[SSE4]

yeah - that's okay and I think is up to package author whether to copy sources or just use --prefix. sometimes it's just waste of time that should be avoided (e.g. in case of boost, which also provides b2 clean).

[lasote]

That's another reason to keep it explicit, we could provide some tool to copy sources, but if any package only needs to change the prefix (or any other mechanism) won't be forced to copy again the sources.

[tru]

Related question to this: I am setting my CFLAGS from my profile, but now when we build all ARCH's in one "go" I wonder what the best way to have variables in the profiles could be, consider this from my profile:

CFLAGS=-arch arm64

changed to:

CFLAGS=-arch {IOS_ARCH}

now I need a way to expand that variable, one idea is to do:

oldenv = os.environ
for env in os.environ:
  newenv[env] = os.environ[env].format(IOS_ARCH=variant)
os.environ=newenv
build(...)
os.environ=oldenv

anyone can think of anything better?

[SSE4]

I would leave CFLAGS in profile without arch, and append arch in conanfile, e.g.:

cflags = os.environ["CFLAGS"] + " -arch=%s" % variant
build(cflags)

actually, there are more flags to be set depends on variant - e.g. -isysroot is usually required

[tru]

Yeah that might be better.

[MathPlayer]

It passed quite some time since last comment and yet there are no changes related to this issue in the tools/apple.py file. Maybe I'm missing the place with related changes? Is anybody still interested/working on adding some helpers for handling universal/fat binaries? I can contribute for such helpers if no one else is working on this.

[memsharded]

Recently, @SSE4 was doing some great work related to apple tools in: https://github.com/conan-io/conan/pull/1815.

I think there is still interest, but the problem might be more challenging than seems at first sight, as the packaging many-to-one flow is not defined in conan so far. So depending on what would be the approach, it might be very difficult. Having some helpers to build all the required variants in the same package, that seems doable.

[MathPlayer]

I analyzed this issue for a bit and yes, it looks quite hard to find a generic solution.
Since I need this kind of functionality only when building for Apple platforms, I think the best thing to do is to add it in a package like https://github.com/theodelrieu/conan-darwin-toolchain

[anton-matosov]

Looks like CMake has direct support for fat binaries now https://cmake.org/cmake/help/latest/variable/CMAKE_OSX_ARCHITECTURES.html
I have not tried it yet, but looks promising.
If Conan would allow array of target architectures instead of single value it would match cmake (and Xcode) approach and make it easy to integrate

[canatella]

This kind of many to one flow is still needed on Android when building with the ndk provided cmake toolchain (which is required if you want to build for a recent NDK).

Another approach would be to allow requirements to specify settings. So I could create a parent package that would require a child with android abi armeabi-v7a and the same child with arm64-v8a. Then assemble them in an aar file for Android distribution.

so I could do a

self.build_requires('mypackage/1.0@repo/stable', settings={ 'os.abi': ['armeabi-v7a', 'arm64-v8a']})