Conan: [question] How to specify the "mapping" of settings to pre-built binaries in `conan export-pkg`?

[dragondive]

• [x] I've read the CONTRIBUTING guide.

I have a library with pre-built binaries (and some other files) as shown below:

├── Linux_GNU
│   ├── libmypkg_32.so
│   ├── libmypkg_32_static.a
│   ├── libmypkg_64.so
│   └── libmypkg_64_static.a
├── Windows_GNU
│   ├── mypkg_32.dll
│   ├── mypkg_32_static.a
│   ├── mypkg_64.dll
│   ├── mypkg_64_static.a
│   ├── libmypkg_32.dll.a
│   └── libmypkg_64.dll.a
└── Windows_MSVC
    ├── mypkg_32.dll
    ├── mypkg_32.lib
    ├── mypkg_32_static.lib
    ├── mypkg_64.dll
    ├── mypkg_64.lib
    └── mypkg_64_static.lib

I want to make this into a conan package, so I referred this document: Packaging Pre-built Binaries

As a first step, I decided to start by packaging only the Windows_GNU binaries. Once that's ready, I would do the same for Windows_MSVC and Linux. (The step-by-step approach is to help me learn how this works as I also want to teach conan to others at work.)

However, I ended up with all the binaries packaged into a single conan package, and not based on the os/compiler/arch settings. My (mis?)understanding from the above-linked document is the collect_libs() method will automatically find the binaries compatible to the settings. Even if this doesn't happen "automatically", I'm fine with it but I could not figure out how to tell conan the "mapping" from these settings to the respective binaries.

Below is the export-pkg command I started with and its console output:

$ conan export-pkg conanfile.py mypkg/1.2.0@aravind/testing -s os=Windows -s compiler=gcc -s arch=x86_64 -f
Exporting package recipe
mypkg/1.2.0@aravind/testing: The stored package has not changed
mypkg/1.2.0@aravind/testing: Exported revision: efe93cb3feef5e6521d5c88cb00ee536
mypkg/1.2.0@aravind/testing: Forced build from source
Packaging to 7bcbf9c8d331220d34f23436e89d530ede2c74b0
mypkg/1.2.0@aravind/testing: Generating the package
mypkg/1.2.0@aravind/testing: Package folder ~/.conan/data/mypkg/1.2.0/aravind/testing/package/7bcbf9c8d331220d34f23436e89d530ede2c74b0
mypkg/1.2.0@aravind/testing: Calling package()
mypkg/1.2.0@aravind/testing package(): Packaged 2 '.py' files: conanfile.py, conanfile.py
mypkg/1.2.0@aravind/testing package(): Packaged 3 '.jar' files: mypkg-1.2.0.jar, mypkg-1.2.0-sources.jar, mypkg-1.2.0-test-sources.jar
mypkg/1.2.0@aravind/testing package(): Packaged 1 '.fmu' file: mypkg.bridge-1.2.0.fmu
mypkg/1.2.0@aravind/testing package(): Packaged 4 '.lib' files: mypkg_32.lib, mypkg_64.lib, mypkg_64_static.lib, mypkg_32_static.lib
mypkg/1.2.0@aravind/testing package(): Packaged 4 '.dll' files: mypkg_32.dll, mypkg_64.dll, mypkg_32.dll, mypkg_64.dll
mypkg/1.2.0@aravind/testing package(): Packaged 6 '.a' files
mypkg/1.2.0@aravind/testing package(): Packaged 2 '.so' files: libmypkg_32.so, libmypkg_64.so
mypkg/1.2.0@aravind/testing package(): Packaged 1 '.cpp' file: example.cpp
mypkg/1.2.0@aravind/testing package(): Packaged 2 '.txt' files: CMakeLists.txt, mypkg_1.2.0-Release.Notes.txt
mypkg/1.2.0@aravind/testing package(): Packaged 3 '.zip' files: fmu-1.2.0-csources.zip, mypkg-Wiki-1.2.0.zip, mypkg-1.2.0-csources.zip
mypkg/1.2.0@aravind/testing package(): Packaged 1 '.pdf' file: mypkg__2.0-Doc.pdf
mypkg/1.2.0@aravind/testing: Package '7bcbf9c8d331220d34f23436e89d530ede2c74b0' created
mypkg/1.2.0@aravind/testing: Created package revision 37f66ea0b5c24b4b4223d6aa1176a46c

What am I missing here? Any hints or advice on what I should try next to get it working?

[memsharded]

Hi @dragondive

I think there was some missunderstanding

However, I ended up with all the binaries packaged into a single conan package, and not based on the os/compiler/arch settings. My (mis?)understanding from the above-linked document is the collect_libs() method will automatically find the binaries compatible to the settings.

collect_libs() only gathers the existing libraries, but cannot classify. It is designed to be used in the package_info() method, to avoid explicitly listing libraries in self.cpp_info.libs = ["lib1", "lib2",..] (I still prefer to list libs explicitly).

For your case, you have a couple of alternatives:

• Write your package() function so it has a conditional to resolve to the different folders:

def package(self):
      if self.settings.os == "Windows":
             self.copy(.... , src= "WindowsMSVC", ...)

• Specify the folder where the binaries are:

$ conan export-pkg .... --build-folder=WindowsMSVC

Please let me know if this clarifies the issue. Thanks!

[dragondive]

Thanks @memsharded. Now my misunderstanding is clarified, and this explanation makes complete sense. Now I could separate out those binaries based on various settings and options, and also put other settings-independent files (source code, documents, etc.) into the conan package in a proper way.

[dragondive]

Hello @memsharded, sorry to reopen the issue. I just realized something is not clear to me: Is there a way to assign the settings directly in the package() method without specifying them on the command line?

Here's what I'm trying to do:

def package_info(self):
    // copy common files, such as source code, documents, etc.
    self.copy("*.cpp", dst="src")
    self.copy("*.h", dst="include")
    self.copy("*.pdf", dst="documents")

    // Copy the pre-built binaries for every configuration.
    // For this example, assume only two binaries, one for Windows, one for Linux

    if self.settings.os == "Windows":
        self.copy("mylib.a", src="Windows", dst="lib")
    elif self.settings.os == "Linux":
        self.copy("mylib.so", src="Linux", dst="lib")
    else:
        # not supported, throw exception

I know that I will always have exactly N pre-built binaries, one for each configuration. (N = 2 in above example) Hence, I would need to call export-pkg N times from the command line to make the conan package, which practically means I have to create _another_ script that calls conan export-pkg in a loop.

Is it possible that this looping can be done directly inside conanfile.py?

In other words, I'm sure that each branch of the if condition will be executed during one of the export-pkg calls, hence instead all those self.copy() calls can be run one after the other in one call. I tried to assign values to the settings directly inside the package() method before calling self.copy() but those values seem to be ignored and it instead takes the settings values from the machine where the script is running.

[memsharded]

I know that I will always have exactly N pre-built binaries, one for each configuration. (N = 2 in above example) Hence, I would need to call export-pkg N times from the command line to make the conan package, which practically means I have to create another script that calls conan export-pkg in a loop.

Yes, this is necessary. In exactly the same way the conan create command creates exactly 1 binary for 1 configuration (defined in the profile), the conan export-pkg is nothing but a shortcut to the conan create command, in which the "build" step is skipped and things are packaged directly from the user space. But still only 1 configuration is possible.

If you want to package more than one configuration, then yes, looping over the different profiles/configurations is necessary. The reality is that this is done rarely in developer machines, but in CI machines, which are the responsibles of automating the package creation task and make the whole process reproducible there, so it is not big issue to automate it there. In any case, the typical CI needs to build anyway the different configurations/profiles, with conan create for other packages as well, so the profile becomes an argument to CI pipelines, which allows to add new configurations/architectures/platforms in a scalable way.

For your case, if it is simply for you, I guess a local shell script or bat file will be very simple and should do the work. Conan 2.0 will have custom user commands, which probably would be the best place to automate this, but it will still take some time.

[dragondive]

@memsharded Thanks for the detailed explanation. Considering continuous integration, the current approach seems reasonable. My idea was to package all the pre-built binaries on one machine and then let it be consumed on various machines with different configurations.

The custom commands in Conan 2.0 sounds like a useful feature. However, for now, handling this case with a shell script seems good.