Conan: [question] How to annotate packages with visible free-form meta data

[unzap]

When packaging and distributing pre-built binaries for multiple configurations
I'd like to annotate the packages with free-form information the users or other tools
can query (conan search , conan get or conan inspect )
something like:

description  # ok, this exists already
topics: # ok, this exists already

free_form_info: {"build_date": "10102019-1234",
                 "module_git_sha1": "1234567890",
                 "features": ["openssl", "zlib", ...]
                }

In our use case the settings (customized) contain the os, arch, compiler etc.
mandatory settings. The other meta information related to the binaries are "good to know"
for the users so when users are installing the packages they will not do it like:

conan install -s os=Linux -s arch=x86_64 -s feature_openssl=True -s feature_zlib=False

but like this:

conan install -s os=Linux -s arch=x86_64

Is there any way to add custom meta information besides the "description" and "topics"?

[memsharded]

Hi @unzap

I think you are interested in the metadata properties field. Have a look to: https://github.com/conan-io/conan/blob/59e983f6d542dba5cfa48cef952b23ff1143a5f7/conans/model/package_metadata.py#L40

This is very new, we are considering that as a place to put those things. We don't want to put them in the recipe (as you cannot put for example the build_date, as that might change and be different for every binary package that you build, and the recipe is common to all of them). So you want for example to embed all the metadata of the build server that built a certain package binary, so something like build_server=MybuildserverWindows14, build_date=xxxxx, jenkins-job:1234, etc, and that cannot be added to the recipe for every binary.

That is only the model, what we need to think is what would be needed to make this feature operational:

• Add a user interface to define properties per recipe and per package binary.
• Check that the metadata is correctly being uploaded to the servers, and retrieved when installing the packages.
• Define user interface to display properties for recipes and package binaries.

As I see it, and you suggested, that is metadata, so it shouldn't affect to other conan processes. Mostly defining it, storing it, displaying it, and making it available for other tools and processes.

My proposal would be to keep the interface decoupled (ref=recipe reference, pref=package reference, including the binaryID):

$ conan properties set <ref> prop=value, prop2=value2...
$ conan properties get <ref>
$ conan properties set <pref>prop=value...

These would operate in the conan local cache. With the -r=remote argument, they could operate in the remote too.

Will annotate it as look-into, because first we need to investigate if this is possible and possible implications.

[memsharded]

Calling @lasote, to discuss if there could be any implication regarding the ongoing work of the api v2.

[unzap]

$ conan properties set <ref> prop=value, prop2=value2...
$ conan properties get <ref>
$ conan properties set <pref>prop=value...

I like this approach.
I think it would be useful also to allow setting properties via wildcards e.g. like this:

$ conan properties set */1.0@user/channel prop=value

One motivation for us is to script/tools on top of conan command line tools to e.g. search binaries matching some criteria. If you build a project containing (git) submodules and you create a package/reference of each submodule and the supermodule/top project recipe defines the (submodule) dependencies it would be useful to annotate the main project "this is the actual product".

[unzap]

Hi @memsharded,
is there a functional version available, in master? I'd like to give it a try?

[memsharded]

Hi @unzap

I am sorry, no, it is not available. We have been checking different possibilities and it will be very complicated at the moment, because the APIs are not prepared for that, we can't introduce them without breaking or entering a nightmare of maintaining different versions incompatible with each other... The API v2 has just been released and is already implemented by the different servers, further changes will most likely have to wait at least after Conan 2.0.

Also, this feature won't cover the use case you are describing, if I understood correctly. This metadata can't be used for installing, as it doesn't affect the binary model. If you are not creating different binaries, you should always use conan install -s os=Linux -s arch=x86_64, and not -s feature_openssl=True -s feature_zlib=False. And if you are creating a different binary depending on when it is linking with openssl, zlib, or not, then you definitely need to model it, it cannot be metadata, it must be part of the conan model (settings, options). Please clarify this if I misunderstood, thanks!

[unzap]

Hi @memsharded @lasote

for the time being I've experimented a bit in the following way:

/metadata.ini
/conanfile.py

class ModuleA(ConanFile):
exports = "metadata.ini"

Then for every build configuration of ModuleA (settings and options) I can embed metadata.ini specific to that build configuration. Then:

conan info <reference> -s option_1=A -s option_2=B ... --paths
/home/user/.conan/data/ModuleA/1.2.3/user/channel/package/cf6bb7d8b0a7b88b5beb9f694ba435abeada38e9/metadata.ini

conan info <reference> -s option_1=C -s option_2=D ... --paths
/home/user/.conan/data/ModuleA/1.2.3/user/channel/package/dde3e986f6b130b4b86d4f8d24da465d5dd59fd5/metadata.ini

As I'm obtaining the path to metadata.ini via official Conan API (conan infor <> --paths) it should be safe to access the file in read only mode?

I did also notice that I can add custom variables to my conan class and those are exposed by:

conan inspect <reference> -a my_custom_variable

But that is per recipe, not per build configuration (settings & options) so the first approach works better so far.

If you have any other (unofficial) way of doing this I'm all ears :)

[lasote]

Yes, looks like a nice way to store metadata.