Conan: [bug] cross compilation with build requirements is total broken

[a4z]

Environment Details (include every applicable attribute)

• Operating System+version: OSX latest
• Compiler+version: apple-clang 11
• Conan version: 1.28.1
• Python version: 3.8.5

Steps to reproduce (Include if Applicable)

conan create --profile androidArmV8 --test-folder None /Users/a4z/conan/conan-center-index/recipes/openssl/1.x.x openssl/1.1.1g@
Exporting package recipe
openssl/1.1.1g exports: File 'conandata.yml' found. Exporting it...
openssl/1.1.1g exports: Copied 1 '.yml' file: conandata.yml
openssl/1.1.1g exports_sources: Copied 1 '.patch' file: 1.1.1-tvos.patch
openssl/1.1.1g: The stored package has not changed
openssl/1.1.1g: Using the exported files summary hash as the recipe revision: ea053bb21933acd1944168b54f524bbb 
openssl/1.1.1g: Exported revision: ea053bb21933acd1944168b54f524bbb
Configuration:
[settings]
arch=armv8
arch_build=x86_64
build_type=Release
compiler=clang
compiler.cppstd=17
compiler.libcxx=libc++
compiler.version=9
os=Android
os.api_level=21
os_build=Macos
[options]
libcurl:with_openssl=True
lua:compile_as_cpp=True
paho-mqtt-c:asynchronous=True
paho-mqtt-c:ssl=True
paho-mqtt-cpp:ssl=True
[build_requires]
*: cmake/3.18.1, android_ndk_installer/r21@bincrafters/stable
[env]
openssl/1.1.1g: Forced build from source
ERROR: Loop detected in context host: 'cmake/3.18.1' requires 'openssl/1.1.1g' which is an ancestor too

This is wrong, cmake is a build requirement and on my system or os_build / arch_build and so is openssl

What does conan want to build, cmake for Android?
It shall use the existing cmake I have on my box

Note that I had this working, this break came recently, when updated conan and the recipes from cci

Yesterday I could build all my packages with conan with exactly the same profiles, today I cant, why?

[jgsogo]

Cross-building using only one profile is very problematic. It required recipes intended to be build-requires to have settings = "os_build", "arch_build" which is only valid for those that will be used only as build-requires (that's the reason why, in the past there were _duplicated_ recipes with the _installer suffix like cmake_installer/3.xxx@conan/stable), otherwise you want different binaries for different compilers or build types (protoc provided by protobuf package, the library need all the settings).

Recipes in ConanCenter are moving away from settings os_build and arch_build and, in consequence, only the approach using two profiles will work to build a graph with _host_ and _build_ contexts.

[a4z]

I agree that the 2 profiles approach looks better, but here https://github.com/conan-io/conan-center-index/issues/2566
you said:

Conan v1.x: keep using the current approach with the single profile (this is important, we need to know that the default works). When Conan v2.0 arrives, we will use two profiles.

I am still on conan 1, so I thought, I ill continue using the 1 profile approach, also because of this
https://github.com/bincrafters/community/issues/1255

(did not have any problems building for iOS, as far as I remember, but I stopped experiments with the 2 profiles because as said above)

but, if you tell me that I am better with the 2 profile approach, I will go this way, no problem
and update the android_ndk_installer so it makes sense in this context,

in this case, shall I set the CONAN_V2_MODE variable to ON ?
There is a pretty concrete warning about using this in production, and I am expected to delivers something useful :-)
(but I can cleanup and re upload our packages anytime, so this is not a huge problem)

alternative, I stick with the 1 profile approach, remove the build requirement to cmake and use that one from the system, or set it to ignore openssl, or create a cmake_installer (or something better anmed) with os_buid / os_arch and use that
, what is also not a problem ...

I just do not like it when I have so many different solution possibilities and all seem to be temporary ....

what is that one with the best future proof chances ?
the 2 profile one (with CONAN_V2_MODE or without?)

[a4z]

Hi @jgsogo , I try to re-write https://github.com/bincrafters/community/issues/1255 i a way so it fits the 2 profile approach, but there is something that blows my mind.

the Android NDK makes a lot of use from os_build and os_arch, and I can adopt that to access settings_target
but
how does this harmonize with the conan create command?
at this time , there is no settings_target,
thus, changing the existing settings

    settings = {"os_build": ["Windows", "Linux", "Macos"],
                "arch_build": ["x86_64"],
                "compiler": ["clang"],
                "os": ["Android"],
                "arch": ["x86", "x86_64", "armv7", "armv8"]}

to something like

    settings = {"os": ["Windows", "Linux", "Macos"],
                "arch": ["x86_64"],}

    settings_target = {
                "compiler": ["clang"],
                "os": ["Android"],
                "arch": ["x86", "x86_64", "armv7", "armv8"]}

seems not to be possible.

or do I have to ignore the settings_target and validate them at execution , and move what is not in configure into the package_info function, and hope that this is not used when conan create --profile default .... is called

in package_id , do I need to

        del self.info.settings_target.arch
        del self.info.settings_target.os.api_level

or are those not part of the package id anyway
The android NDK is super huge , and I seriously want it just 1 time ;-)

and / or is there anything else I need to know and understand.
Can you help fixing my mental model for a use case like that please?

[jgsogo]

Hi @a4z

I found some time to play with the Android NDK recipe some time ago and I used it to compile several packages. I did some modifications to the @bincrafters one, you can find mine here: https://github.com/jgsogo/conan-center-index/tree/android_ndk/recipes/android-ndk/all I used the two-profiles approach, any default profile for my _build_ machine and this one for the _host_ profile:

[settings]
os=Android
os.api_level=21
arch=armv7
compiler=clang
compiler.libcxx=libc++
compiler.version=9
build_type=Release

[build_requires]
android-ndk/r21d

If you have a look at the package_info() method it tries to cover three different ways of integrating the NDK. Most important thing is that it uses the settings_target attribute to populate different things. This attribute exists only when the android-ndk recipe is used as a build-requires to compile something in _host_ context, and the settings_target contains the settings from that _host_ context: I use them to compute the triplets.

@theodelrieu motivated me to try this recipe, the idea behind the modifications was to have only one package that can be used for any api-level.

[a4z]

That looks like a promising begin, @jgsogo ! Thanks a lot, and this recipe seems way simpler to work with than the existing one.

Actually, I just started to adopt the recipe my own and moved into the same direction, but it makes no fun to work with a package of this size especially if it is uncertain for me if the outcome will ever be usable.

But I will invest some more time, based on you work

May I ask, the from jinja2 import Template in the recipe, that is not really needed, or is it?

[theodelrieu]

@jgsogo Looks great! I don't exactly remember why I failed to use settings_target appropriately a few months ago, but your code makes a lot of sense!

I'm looking forward to having these changes merged in bincrafters.

[a4z]

Hi @jgsogo , your NDK build is great, but it does not work for all archs, for i686 it builds arm32 binaries, and for x86_64 android builds it fails,

but I made a hybrid, out of yours and the old one, and this seem to work (have running it in the background while writing this, not all done but so far not problems)

works of course only with the 2 profile build, what is OK, but I wonder, what and when will the transition to 2 profiles work for cci?
anyhow,
where could I commit this recipe to?
, so others can also use it, and maybe even improve it?

Edit, instead of having the recipe here, link to there
https://github.com/conan-io/conan-center-index/pull/3004

[a4z]

I think I can close that, the 2 profile approach solves, if not everything, at least a lot
so that's fine for me