Conan: Improve packaging external binaries howto guide

[shreyasbharath]

I have been going through Conan's documentation for the last couple of days.

What I am trying to do is package an already built project, by copying the header files and static libraries within it and upload it to an artifactory remote for consumption by other projects.

The documentation provided on this page I feel is very lacking. I feel a concrete example for this scenario would really help.

Also, how do users and channels fit into this scenario? There was no explanation of users and channels in the getting started guide. I can improve the documentation myself if someone can please explain how I go about doing this.

I really want to give Conan a chance, I was previously using Maven to package C/C++ libraries and I found it to be a major pain in the backside. Improving the documentation will really help newcomers like me :)

[memsharded]

Yes, totally agree. This is one of my things that is on my TODO list for a while, and have been so busy, couldn't find time to properly do and document it.

Users and channel, there is nothing special about them for this specific scenario, just the normal usage for user and channel fields (you can read about them here: http://docs.conan.io/en/latest/manage_deps/conanfile_txt.html#requires). Is there any specific question you have about them?

The section that is relevant for your use case is this one: http://docs.conan.io/en/latest/howtos/existing_binaries.html#packaging-local-binaries.

The last example would show the complete flow for your use case, most likely it deserves more detail. In summary:

• A package recipe "conanfile.py" is needed for the package definition and meta-info. So it is necessary to create it and export it to the local conan cache, to define the package, before adding the binaries
• Adding the binaries with conan package_files specifying the configuration of that binaries.

Please, do not hesitate to ask further questions or for details while we improve the docs.

[memsharded]

BTW, documentation issues can go to: https://github.com/conan-io/docs :)

[shreyasbharath]

Yup I will update the docs and submit a PR when I figure out how to accomplish what I am trying to do. Feel free to move this issue to the docs project (if that is possible?).

Here's what I am trying to do in conanfile.py, please feel free to correct me if I am doing something wrong.

• build() - copy header files and static libs to "include" and "lib" directories respectively (I am using Python's glob and file walk APIs to achieve this)
• package() - copy "include" and "lib" directories using self.copy("include") and self.copy("lib")

Assuming those function definitions are right, what do I need to do next? I run _conan build_, which copies all the files to the appropriate directories. However if I then run _conan package test/0.1@myuser/testing_ I get an error saying 'package recipe has not been built locally'.

Note if I call _conan package_files_ instead, it seems to copy everything from the root folder where I executed the command, which is not what I want.

[memsharded]

The build() method will typically build things, not copy files. If you are going with the conan package_files approach, you don't need the build() method at all, neither the package() method or the source() method.

For most cases, the simple "bare" recipe generated with conan new --bare will be enough:

from conans import ConanFile

class {package_name}Conan(ConanFile):
    name = "{name}"
    version = "{version}"
    settings = "os", "compiler", "build_type", "arch"
    description = "Package for {package_name}"
    url = "None"
    license = "None"

    def package_info(self):
        self.cpp_info.libs = self.collect_libs()

The conan package_files is to copy existing artifacts. If you don't have such artifacts in a "clean" folder, and you want to extract them, then yes, you might try using the package() method, and then use the command:

$ mkdir mypkg && cd mypkg
$ conan package .. --build_folder=../build
$ conan package_files Pkg/version@user/channel ...

In the conan package the first argument points to the folder containing the conanfile, the --build_folder the root where you want to extract the artifacts from. Please tell me if this makes sense. Thanks!

[shreyasbharath]

Thanks for the reply again!

I tried both the approaches you suggested and unfortunately neither had the desired effect.

The "bare" approach did not work as it packaged all the files/directories in the project folder, even though I had standard "include" and "lib" directories.

With the build and package approach, here's my conanfile.py -

from conans import ConanFile
import glob, os, shutil

class TestConan(ConanFile):
    name = "Test"
    version = "0.1"
    settings = "os", "compiler", "build_type", "arch"
    description = "Package for Test"
    url = "None"
    license = "None"

    def build(self):
        if not os.path.exists("build/include"):
            os.makedirs("build/include")

        if not os.path.exists("build/lib"):
            os.makedirs("build/lib")

        header_files = glob.glob("code/**/*.h", recursive=True)
        for file in header_files:
            if os.path.isfile(file):
                shutil.copy2(file, "build/include/" + os.path.basename(file))

        lib_files = glob.glob("obj/**/*.a", recursive=True)
        for file in lib_files:
            if os.path.isfile(file):
                shutil.copy2(file, "build/lib/" + os.path.basename(file))

    def package(self):
        self.copy("build")

    def package_info(self):
        self.cpp_info.libs = ["Test"]

I ran the following commands per your suggestion.

$ mkdir mypkg && cd mypkg

$ conan package .. --build_folder=../build

    PROJECT: WARN: conanbuildinfo.txt file not found in C:\Work\Test\build
It is recommended for this command
You can generate it using 'conan install -g txt'
PROJECT: Generating the package
PROJECT: Package folder C:\Work\Test\mypkg
PROJECT package(): WARN: No files copied!
PROJECT package(): WARN: No files copied!
ERROR: conaninfo.txt does not exist inside of your C:\Work\Test\build folder. Try to re-build it again to solve it.

I am guessing I am doing something fundamentally wrong here, your help would be appreciated.

[memsharded]

Thanks for the code! I am still missing something, though, like where the code/artifacts are coming from, and where the glob aims to get the files. I suspect that it is something related to the paths, but not sure.

Why don't you set up a github repo with such conanfile in it and some example artifacts (could be text file) with the same layout you have locally, and also a "build.py" python scripts with the command line commands to run. That would really help, I could fork it and propose an automated process and collaborate. Thanks!

[shreyasbharath]

Great idea, I will set it up and report back.

[shreyasbharath]

Okay, here we go -
https://github.com/shreyasbharath/conan-test-package

Note - Assume that the project is pre-built, I have submitted example build artifacts that represent the output of that build process.

All I want is for the public header files and static libs to be packaged and uploaded to a conan remote, which can then be consumed by other clients.

[shreyasbharath]

@memsharded What would you like me to do to close this off? Any ideas on how I can improve the documentation?

I will read the documentation again and see how I can make it better.

[memsharded]

So I think a good approach would be to link to the repo, and briefly explain its layout, aim and steps (also include this in the repo README). Having a fully reproducible example is really, really useful when having doubts.

Renaming the repo to something more meaningful, like "conan package binaries example" could be nice.

Probably a subsection at the end of http://docs.conan.io/en/latest/howtos/existing_binaries.html#packaging-local-binaries could be the best.

We will be doing some improvements over this flow in the next releases, this is a very recent command. Thanks again for your help.

[cleeland]

@shreyasbharath I share your struggles with packaging external binaries. I, too, find the documentation insufficient for understanding and will be looking to contribute improvements via PR.

Best that I can tell from questions I've asked and from experimentation, package_files just sucks up everything that it finds in the present working directory when you execute it. I ended up modifying my external build system to create a staging directory (similar to what you might get from a make install), then do something like:

$ cd path/to/my/staging/directory $ conan package_files mypkgname/1.2.3@ccleeland/testing

Right now, I am struggling with the fact that if I mimic the example and run conan test_package immediately after package_files, conan re-exports my recipe and deletes everything that just got packaged via package_files.

[memsharded]

If you don't want conan test_package to re-build the binaries, thus overwriting the binaries you exported into the local cache, remember to add --build=missing, try:

$ conan test_package --build=missing

We are listening to you, and working on several core improvements towards this, as possibly a conan create command that will run the export + build + (optionally if exists) + test_package. Also considering renaming conan package_files => conan export_binary which is more symmetrical with the export concept. Also preparing the next week webinar with CLion that will show the package_files and many other core tasks. So we are also struggling trying to improve this forward. But we are on it, and you feedback is being extremely valuable :), thanks very much for it.

[cleeland]

Why would --build=missing prevent? From the output of conan test_package -h, --build=missing does the following:

Build from code if a binary package is not found.

On the other hand, the default, --build=never purports to do the following:

Never build, use binary packages or fail if a binary package is not found.

The way I read this, it seems that the default should do exactly what I want, but instead it erases what I packaged.

[cleeland]

We are listening to you, and working on several core improvements towards this, as possibly a conan create command that will run the export + build + (optionally if exists) + test_package. Also considering renaming conan package_files => conan export_binary which is more symmetrical with the export concept. Also preparing the next week webinar with CLion that will show the package_filesand many other core tasks. So we are also struggling trying to improve this forward. But we are on it, and you feedback is being extremely valuable :), thanks very much for it.

Appreciate the attentiveness. I have not digested what you describe above, but my suggestion would be to try to work the concept of package_files in as a subset of the typical workflow. In the model that I've built up in my head, the common workflow packages up stuff that gets staged from the build directory. If there were a way to have source() declare that the source directory was the current directory (or relative to the current directory), then the rest of the workflow for packaging external binaries would be the same as for the "normal" workflow.

[cleeland]

Just to follow up, conan test_package -ne --build=missing did not help. While it did not erase my packaged stuff, it also didn't seem to find the binaries that are packaged in there.

[shreyasbharath]

We are listening to you, and working on several core improvements towards this, as possibly a conan create command that will run the export + build + (optionally if exists) + test_package. Also considering renaming conan package_files => conan export_binary which is more symmetrical with the export concept. Also preparing the next week webinar with CLion that will show the package_files and many other core tasks. So we are also struggling trying to improve this forward. But we are on it, and you feedback is being extremely valuable :), thanks very much for it.

I am glad you are 👍

C/C++ has been crying out for a really good cross platform package manager and I really hope Conan can get there!

[shreyasbharath]

So I think a good approach would be to link to the repo, and briefly explain its layout, aim and steps (also include this in the repo README). Having a fully reproducible example is really, really useful when having doubts.

Renaming the repo to something more meaningful, like "conan package binaries example" could be nice.
Probably a subsection at the end of http://docs.conan.io/en/latest/howtos/existing_binaries.html#packaging-local-binaries could be the best.
We will be doing some improvements over this flow in the next releases, this is a very recent command. Thanks again for your help.

Okay onto it today, will create a PR on the docs when ready.

[shreyasbharath]

A sort of unrelated question, how do we automatically bump up the versions using Conan?

Say if I have released a package v0.1 and I want to release v0.2, do I have to manually edit the Conanfile and then export the package?

[drodri]

@shreyasbharath

Say if I have released a package v0.1 and I want to release v0.2, do I have to manually edit the Conanfile and then export the package?

This is one of the things that we are considering. If you are packaging external sources with the source() method, you will definitely need to bump the version because most likely it is being used in the source() method to retrieve the branch or tag corresponding to that version.
But if you dont need it, because you have the code with exports_sources we are considering:

$ conan export Pkg/version@user/channel

which will not require the version in the recipe, but it will get it from the command line.

[drodri]

@cleeland

Why would --build=missing prevent? From the output of conan test_package -h, --build=missing does the following

Yes, that is a bad explanation from the help. That is the usual for the conan install, but the conan test_package logic is:

• Automatically add --build=PkgName for the PkgName that is being tested
• If --build is specified, it will override that automatic addition

The rationale is that you are creating the package and you want to build from sources. This is exactly what we want to change with the conan create, which is more explicit of what is doing. We also want to make conan test_package estrictly only run the test, but that will take much more time, because many people rely on this and it can break a lot of things.

If there were a way to have source() declare that the source directory

conan package_files has a --source_folder argument that you can use to point to the local source folder, is that what you need? Also the command conan source can be executed locally to execute the source() method, which can conveniently clone the sources repo and put it in the right branch for you

[shreyasbharath]

So I think a good approach would be to link to the repo, and briefly explain its layout, aim and steps (also include this in the repo README). Having a fully reproducible example is really, really useful when having doubts.

Renaming the repo to something more meaningful, like "conan package binaries example" could be nice.

Probably a subsection at the end of http://docs.conan.io/en/latest/howtos/existing_binaries.html#packaging-local-binaries could be the best.

We will be doing some improvements over this flow in the next releases, this is a very recent command. Thanks again for your help.

PR is up for doc update linking to my example repo. I have also added a README to my two repos, Package Binary Example and Package Binary Consume Example as per your suggestion.

Feel free to submit a PR or suggest any improvements and I will action them.

Thanks.

[cleeland]

First, I really like the idea of splitting the creation of packages from the testing and look forward to using that. I'd be happy to give early feedback on it, too.

conan package_files has a --source_folder argument that you can use to point to the local source folder, is that what you need? Also the command conan source can be executed locally to execute the source() method, which can conveniently clone the sources repo and put it in the right branch for you

The option to package_files() isn't quite what I had in mind. In my understanding of the standard conan workflow, source() gathers and prepares all the source into some directory (typically in the local cache), build() then takes over and turns that source directory into package products, package() takes the package products and "fake-installs" or stages them into some directory (typically in the local cache).

What I was suggesting was that there be a way for a recipe to declare the directory in which it's prepared (or will prepare) the source, with the default as something like $LOCAL_CACHE/source. That way, for a legacy build (which is what package_files is trying to address), I could declare that the recipe simply works in the current directory (the one where I decide to run conan create or conan test_package). Then, package_files() doesn't have to exist, because I can make build() empty and package() can either stage the files just as it normally does or it could also declare that the staging directory is some other directory and it already prepared, so that packaging could just slurp that up.

[memsharded]

@shreyasbharath the docs have been merged, thanks!

@cleeland. The command conan create has already been released in 0.25, you can use it.
Regarding your other comments, I think it might be an interesting idea to use conan package with the package() method, to do the test of package_files. It could fix some current minor issues, I will create a new issue for this. So I think this one can be closed now.

[memsharded]

PR https://github.com/conan-io/conan/pull/1549 has introduced usage of package() method in conan package_files command, so the local conan package step in the local flow can be saved.

Will be available in 0.26

[shreyasbharath]

@memsharded Can you please demonstrate how to use this in the 'package binary example' repo?