Conan: [feature] Auto generation of cpp info from cmake

[grifcj]

• [X] I've read the CONTRIBUTING guide.

Motivation

It would be convenient to be ably to cheaply populate cpp info from knowledge that is
already contained in cmake build scripts. Better package data with less effort, who
doesn't like that?

def package_info()
    cmake = self._configure_cmake()
    self.cpp_info = cmake.get_cpp_info()

For this to work best, probably needs the sub-components capability that is currently
in dev.

All this is just some initial prototyping. Would welcome some outside direction and
feedback to help move it along (or discard if there are dead-ends or better ways).

Approach

Rather than trying to go down the rabbit hole of parsing cmake (i.e. package config
files). This approach tries to generate information conan needs in easier to consume
format.

1. CMake Configure/Generation creates a json file with desired information. This requires
   some kind of cmake helper function and a template file. Information filled in via
   configure file to populate variables and file(GENERATE) to convert generator
   expressions.

2. JSON file is included in package or with recipe

3. Build helper reads JSON and fills in cpp info for you in package_info

Code

Template & Generate Function

{
    "name" : "$<TARGET_PROPERTY:@target@,NAME>",
    "filename" : "$<TARGET_FILE:@target@>",
    "includedirs" :
    {
        "build" : "@BUILD_INCLUDE_DIRS@",
        "install" : "@INSTALL_INCLUDE_DIRS@"
    },
    "cflags" : "$<REMOVE_DUPLICATES:$<TARGET_PROPERTY:@target@,INTERFACE_COMPILE_OPTIONS>;$<TARGET_PROPERTY:@target@,INTERFACE_COMPILE_FLAGS>>",
    "cxxflags" : "$<REMOVE_DUPLICATES:$<TARGET_PROPERTY:@target@,INTERFACE_COMPILE_OPTIONS>;$<TARGET_PROPERTY:@target@,INTERFACE_COMPILE_FLAGS>>",
    "defines" : "$<REMOVE_DUPLICATES:$<TARGET_PROPERTY:@target@,INTERFACE_COMPILE_DEFINITIONS>>",
    "requires" : "$<TARGET_PROPERTY:@target@,INTERFACE_LINK_LIBRARIES>"
}

function(generate_cpp_info)
    foreach(target ${ARGV})
        message(STATUS "Evaluate target: ${target}")
        if (NOT TARGET ${target})
            message(WARNING "Argument \"${target}\" is not a target. Cannot export cpp_info.")
            continue()
        endif()

        get_target_property(INCLUDE_DIRS ${target} INTERFACE_INCLUDE_DIRECTORIES)
        string(REPLACE "$<BUILD_INTERFACE:" "$<1:" BUILD_INCLUDE_DIRS "${INCLUDE_DIRS}")
        string(REPLACE "$<INSTALL_INTERFACE:" "$<0:" BUILD_INCLUDE_DIRS "${BUILD_INCLUDE_DIRS}")
        string(REPLACE "$<BUILD_INTERFACE:" "$<0:" INSTALL_INCLUDE_DIRS "${INCLUDE_DIRS}")
        string(REPLACE "$<INSTALL_INTERFACE:" "$<1:" INSTALL_INCLUDE_DIRS "${INSTALL_INCLUDE_DIRS}")

        find_file(template_file "target-cpp-info-template.json.in")
        if (template_file)
            configure_file(
                ${template_file}
                ${CMAKE_BINARY_DIR}/target-${target}-cpp-info-template.json
                )
            file(GENERATE
                OUTPUT ${CMAKE_BINARY_DIR}/target-${target}-cpp-info.json
                INPUT ${CMAKE_BINARY_DIR}/target-${target}-cpp-info-template.json)

            # Erase cache variable set by find
            unset(template_file CACHE)
        else()
            message(FATAL_ERROR "Template file not found. Have CMAKE_INCLUDE_PATH "
                "or CMAKE_MODULE_PATH been set to include package directory?")
        endif()
    endforeach()
endfunction()

Package JSON and Populate Cpp Info

The intent would be that most of the logic goes into cmake build helper and it gains a
method get_cpp_info, but I made due just adding some code to my recipe.

      def package(self):
            cmake = self._configure_cmake()
            cmake.install()
            for t in Path().glob('target-*-cpp-info.json'):
                 self.copy(str(t), dst='package-info')

      def _cmake_targets(self):
            def to_list(cmake_list):
                 return cmake_list.strip(';').split(';')

            for t in Path().glob('package-info/target-*-cpp-info.json'):
                 with open(str(t), 'r') as f:
                      t = json.load(f, object_hook=lambda d: recordtype('target', d.keys())(*d.values()))
                      t.includedirs.build = to_list(t.includedirs.build)
                      t.includedirs.install = to_list(t.includedirs.install)
                      t.cflags = to_list(t.cflags)
                      t.cxxflags = to_list(t.cxxflags)
                      t.defines = to_list(t.defines)
                      t.requires = to_list(t.requires)
                      yield t

      def package_info(self):
            for t in self._cmake_targets():
                 self.cpp_info.libs.extend([t.name])
                 self.cpp_info.defines.extend(t.defines)
                 self.cpp_info.cflags.extend(t.cflags)
                 self.cpp_info.cxxflags.extend(t.cxxflags)
                 if self.in_local_cache:
                      self.cpp_info.includedirs.extend(t.includedirs.install)
                 else:
                      self.cpp_info.includedirs.extend(t.includedirs.build)

Use in consumer

Consumer has to add call to generate helper in CMakeLists.txt and pass the targets they
wish to export. The helper function and file template can be held in a utility package
so they don't need to be added/deployed to consumer. Consumer will generate the json files
in the build directory.

add_library(mylibrary ...)
target_compile_option(mylibrary ...)
target_include_directories(mylibrary ...)
target_link_libraries(mylibrary ...)

# helper function must be injected some how, e.g. defined in
# included build script or defined in some module that gets
# included
generate_cpp_info(mylibrary [...])

Application to Poco

I tried applying this to Poco (another cmake project in conan center), to get some
feedback with something a bit more real. Would be interested in any other good test
projects; the package_info function isn't really all that big for Poco.

Added my cmake extensions module as a requirement, the parsing code to the recipe, and
then the generate call to the CMakeList.txt. I'm leaving out some details of getting the
build to work locally just for brevity.

# CMakeListsOriginal.cmake

include(CMakeExtensions)
generate_cpp_info("${Poco_COMPONENTS}")

It generated some target files for the components.

target-Crypto-cpp-info.json
target-Data-cpp-info.json
target-Encodings-cpp-info.json
target-Foundation-cpp-info.json
target-JSON-cpp-info.json
target-MongoDB-cpp-info.json
target-Net-cpp-info.json
target-PocoFoundation-cpp-info.json
target-PocoJSON-cpp-info.json
target-PocoXML-cpp-info.json
target-Redis-cpp-info.json
target-Util-cpp-info.json
target-XML-cpp-info.json
target-Zip-cpp-info.json

Here's an example of one.

{
    "name" : "Crypto",
    "filename" : "/Users/marianinos/src/conan-center/recipes/poco/all/build_subfolder/lib/libPocoCrypto.a",
    "includedirs" :
    {
        "build" : "/Users/marianinos/src/conan-center/recipes/poco/all/source_subfolder/Crypto/include;",
        "install" : ";include"
    },
    "cflags" : "",
    "cxxflags" : "",
    "defines" : "POCO_STATIC;POCO_NO_AUTOMATIC_LIBS",
    "requires" : "Foundation;/Users/marianinos/.conan/data/openssl/1.0.2s/_/_/package/9c2bc6bb652b363bce80b3b2118be56a4b0fd392/lib/libssl.a;/Users/marianinos/.conan/data/openssl/1.0.2s/_/_/package/9c2bc6bb652b363bce80b3b2118be56a4b0fd392/lib/libcrypto.a"
}

Potential issues

1. For this to work, the information needed by cpp info must exist somewhere in queryable
   properties that variable expansion or generator expressions can populate. The project
   must populate target properties correctly, which generally means setting INTERFACE_*
   properties with the target_* commands.

2. Naming conditioning probably required. I don't know how the exact requirements of
   'libs' property, whether it need logical target names or actual filenames. It seems
   flexible given the Poco recipe manages debug suffixes, but doesn't necessarily need
   prefixes or extensions. But some conditioning seems warranted.

3. Haven't tried multi-config. The file only need to contain information needed for the
   package though, so it won't contain Windows settings in a Linux package for example.

TODOs

1. Need to learn more about cpp info model, especially as it might relate to shared
   libraries and differences between operating systems.

2. Need to track changes coming to cpp info; it looks like it is under heavy dev with
   sub components, and not sure full impact there.

3. Will need to try out in several environments; I have access to Mac/Linux easy enough.
   May need to get windows virtual machine or get more familiar with CI if it offers
   windows test environment.

Other investigated alternatives

1. Python script that parses cmake files. Didn't seem promising and too much effort. Also
   information wouldn't be available until install step.

2. Looked at file api that replaces server mode and is meant to serve IDE integration.
   There is similar content for targets in there, but I saw at least one issue for
   cmake/meson integration where Brad King mentioned that it was not a design goal of file
   api to expose build tree information sufficient to be driven from another build system.

   The template file approach appears to be simpler and can handle things like difference
   between build/install interfaces, which didn't seem to exist in file api target
   content.

Update: Removed links to proto project as they didn't add much context, and I expect to keep using those repos for other purposes, so links would go stale.

[memsharded]

Hi @grifcj

Thanks very much for your detailed proposal.
As much as I would like to see this further automation, I think there are several reasons that would make this unlikely to happen:

• First, the relative complexity for the provided value is high. It needs a bunch of new moving pieces, more CMake code (more difficult to debug). In our experience with thousands of users, it is not perfect to have it defined in the package_info(), but it is not a big pain, these methods are easy to write and maintain. It is a very pragmatic approach.
• It is not only the cost of implementing the feature in the first place. But these things really compound exponentially over time. Support tickets, documentation, breaking things because something in the pipeline (e.g. a new CMake version) changes, supporting corner cases. For example, we have just added new fields (name, system_libs, frameworks) to cpp_info to improve the available information to generators. If this was integrated, we would suddenly need to extract that information from CMake (is it even possible for these things? Do cmake have explicit model for system libraries?). We have learned the hard way that even small features mean a lot of work down the road, and this is no small feature.
• But the most important reason is that the majority of the Conan community is aiming for an even less intrusive way of integrating with Conan. And this is exactly the direction we are moving: last generators like cmake_find_package ones, go in that direction, enabling a more native CMake syntax, and less Conan things in the build scripts. We are right now going further in that direction, proposing the generation of better find_packages and also generating toolchains with the configurations. What you are proposing goes in the other direction, to add even more things related to Conan in the CMake build scripts, so I think this will not get enough support from the community.

Please let me know what you think, and thanks again for your feedback!

[grifcj]

In our experience with thousands of users, it is not perfect ... but it is not a big pain

I think this is probably the most important point. I took another quick tour through a random assortment of recipes in conan-center and most package_info descriptions are under 20 lines, 10 even. If they were pushing 100 to 200 lines in common case, then I think need for more automatic translation could be a hotter topic. I have seen it mentioned a few times, but perhaps it's not so hot.

name, system_libs, frameworks ... is it even possible for these things

• system libs: I have not determined how cmake manages this (the poco recipe didn't seem to include any system libs for Mac where I experimented)
• frameworks - There are various properties related to frameworks in cmake, whether they contain the info cpp_info needs, have to report back.

There definitely seems to be some kind of internal model; whether it's feasible to extract the information easily enough, that's another thing.

Conan community is aiming for an even less intrusive way of integrating with Conan

I think finding resources is one thing, but providing the information so that they can be found is another. In cmake there is the 'find' part and then there is the 'export/install' part. As far as generating cpp info, you could chose to write a package_info method yourself, or use some more automated way to provide it. If that automated way entails adding a single function call to your cmake script rather than writing a larger package_info description in a conan recipe, people might take that trade-off.

The drive for more seamless integration seems to come in part from a preference to use cmake config modules solely, rather than conan generated ones, and in that use case people could be omitting a package_info method altogether. Which, this might work fine in an isolated and homogenous build environment, but circumvents conan's ability to provide integration with other systems, so not suitable for community.

But, sub components is going to change the game a bit. I think package_info could potentially increase in complexity, but there might be a case for omitting install/export from cmake and relying solely on conan generated modules from cpp_info. And in that case, an auto-generated cpp_info would still have the same usefulness, if it had the right implementation.

I found #2387, which has similar discussion I think.

Questions

Thank you for taking the time to read/respond to my issue. I am a bit of a novice with this tech, so if you see any statements that hint towards under-informed characterizations on my part, please correct me.

I had a few other questions if you had the time.

• In #5408 your last comments regarded performance issues? Is that the main remaining challenge with that pull request? Perhaps it is best that I ask there?
• Do you have a feel for average size of package_info methods? Is it generally 20 lines or less like my random sampling determined? Do you know of any complicated outliers and what drives them to be such?
• Is there a particular package that you think serves as a good example of large cmake project with conan and is also good conan citizen by filling in all package metadata?
• If it were possible to auto-translate cpp-info from a build system (any build system), what would the key components of that solution to you? For example, if you could pass a flag to cmake and it would automatically generate a json dump for you of all targets and all their properties, would that be something considered usable, rather than injecting cmake code?
• There are at least a few other package description/models out there. pkg-config, libman, cps. I don't know much about these things other than they exist, but if a build system produced such package descriptions, could they be useful sources of information to conan? Are there any issues/discussions tracking how these things interact with conan?

[grifcj]

I spent more time with this. TLDR; didn't work, but I figured I'd document.

I did advance the general strategy and the following was sufficient to require
no edits to consumer project's cmake scripts. So the only impact to consumer
was using python module made available in build_requirement in their script.

• CMake install (project must have implemented cmake install)
• A project-template is built in temp dir and imports targets via find_package
• Installed targets passed to cmake function to translate interface properties
  into yaml file, via configure/generate
• Package yaml file and use it later to populate cpp-info

The helper function would read target properties as defined at configure time
via get_target_property. If you read target properties via generator
expressions (as I did in initial post) you can get aggregated values, which is
not what you'd want when trying to translate to componentized cpp_info model
(at least not all the time)

Frameworks and System Dependencies (Follow-up from above)

• Using find_library for frameworks as seems to be official suggestion results
  in target_link_library argument that always ends in .framework, so frameworks
  can be readily identified as such.

• System libraries more problematic. It looks to be a problem of elimination to
  establish that a token such as 'rt' 'pthread' isn't a target or some other
  possible input argument. A positive test for system library didn't seem
  readily available unless one wants to keep master list of common system
  libraries on various platforms. Also, there are variations such as system
  libraries specific via flags or via linking to interface libraries.

Why I gave up - Usage requirements composition via interface targets

Unlike other libraries (i.e. sub-components in project) you would need to
aggregate usage requirements from interface libraries into your component. So, this
presents situation where we would need to aggregate properties for some library
targets and not others.

At configure time, it's possible to read target types and determine whether
you're dealing with an interface library. And perhaps one could write output
properties for interface targets to yaml and aggregate on the backend while
reading...except for the following

target_link_libraries(lib PUBLIC $<GENERATOR-EXPR:...>)

Generator expressions can't be handled at configure time and such expressions
could evaluate to an interface library. It does not appear possible to iterate
(or do general selection logic) with generator expressions, so some type of
property handling appears out in that scenario. But it's not like you'd really
want to do that if it were possible, given the meta-programming like soup that
would result.

Closing

I think closing this due to not enough value for complexity and/or intractable implementation as surmised by @memsharded seems reasonable.

Course, perhaps there is a magic way that escaped me, but it seems that the following aren't really good solutions to this problem

• Parse cmake config modules directly (too complex, basically re-implementing cmake)
• Use file-api from cmake (not enough info exposed)
• Translate to cpp_info by exposing properties via cmake code (see above).

[memsharded]

Hi @grifcj

Thanks very much for following up and documenting so well your conclusions.
If there is other way, it also escaped me, but seems unlikely to exists, I'd say it is just too complicated for the value.

Good work anyway, thanks for your contribution again!

[KerstinKeller]

Hi, I'd like to reopen and revisit this issue.

If there is other way, it also escaped me, but seems unlikely to exists, I'd say it is just too complicated for the value.

Maybe right now it does not matter so much, but the more packages will be used, the more work will have to be put in maintaining all conan center recipes. (Think about the effort to sync recipes with the original repos, possibly patch sources, ...).
Conan recipes duplicate information which is actually an "outcome" of the recipe (e.g. not an input), and which is available already via the actual build system.

Therefore I think it is a very promising path (longterm) to think about how to extract packaging information from build systems instead of duplicating that information in Conan recipes.

So I tried to do the same approach that @grifcj suggested but also got stuck using the CMake file API, because it did not contain all relevant information.
After opening this thread to ask about how to possibly access the necessary information in a parsable format on the CMake discourse forum, this CMake issue has been opened, which also discusses possible C++ package descriptions:

• pkg-config's .pc files
• The Common Package Specification (CPS)
• A paper from Richard Smith presented to the SG15 C++ committee: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p1767r0.html
• libman, and the spec here:
  https://api.csswg.org/bikeshed/?force=1&url=https://raw.githubusercontent.com/vector-of-bool/libman/develop/data/spec.bs

I think it could possibly be very interesting for Conan to follow approaches or even contribute (from all the experience gained by CCI) to a standardized package description, since Conans cpp_info is basically also a package description.

If at one day, there were a standardized C++ package description, Conan would have to focus a lot less on writing build systems generators (might even be obsolete, if build systems were able to consume such a standardized package description).

[memsharded]

Hi @KerstinKeller

Thanks for your feedback. We totally agree this is the way to go, and we are really looking forward for the community moving in this direction, and we will also certainly contribute as much as possible.

I am not reopening this issue, because the feature request here is clearly different, it request deducing the information from cmake files, which is totally unfeasible for us, and what you are suggesting is actually different. Please submit a new issue for starting a new conversation around this topic.

For a chance of having something useful, it would be needed that the build systems would be willing to generate and consume that information. We could do things, for example generate some intermediate files (the best approach IMO is libman) from the package_info() method. But without the build systems following these files conventions, it will be just another extra layer for us, complicating our life and ours users without any direct benefit. As much as I would like to see build systems agreeing on this, at the moment seems complicated, and out of Conan scope. We will probably try to contribute proposals to C++ SG15 group when possible, that might be a better forum for this.