Conan: Support a modern cmake generator

[ruipires]

The regular cmake generator supports (by design) only cmake 2.8.

It would be nice to be able to have a generator leverage the new cmake features, namely transitive usage requirements (https://cmake.org/cmake/help/v3.0/manual/cmake-buildsystem.7.html#target-usage-requirements).

Note: there is some overlap with this existing issue https://github.com/conan-io/conan/issues/168 , particularly in some comments/analysis that were already done there.

[memsharded]

I have been reviewing this possibility, is something that I would like, but I still have many doubts about the viability:

• What happens to multi-libs packages, as boost? Boost defines in its package_info() something like: self.cpp_info.libs = ["regex", ...] with all the libs inside boost.
• What happens to packages that contains both libraries and executables? What happens to packages of header-only libraries?
• There are packages even mixing some static and shared libraries.
• Packages do not internally model the dependencies between their libraries, and such information would be totally

The issue is that for the definition of cmake IMPORTED targets, we need to know the type of each library, in STATIC, SHARED, INTERFACE, but we can hardly have this information from the package, without introducing a great burden on the package creator. So if there was a 1-to-1 match between packages and libraries, that would be easier, but with current approach of typical C and C++ projects of bundling many libraries, it seems difficult. Any suggestion about how conan would model that, and how the UI for the package creator would look-like?

What could be possibly made is the transitivity. Now, in the conanbuildinfo.cmake there are two things:

• A set of global variables like CONAN_INCLUDE_DIRS which are transitive for the full dependency tree.
• A list of package variables, like CONAN_INCLUDE_DIRS_BOOST, that only contains the values for such package, but not its transitive dependencies

Conan could maybe create CONAN_INCLUDE_DIRS_BOOST_TRANSITIVE, which would include, in order, the include directories for boost and its transitive dependencies. In this way, the use/include/link/flags of a certain particular package instead of the whole dependency tree would be really straightforward.

[ruipires]

I'm not sure this would pan out, since I'm not really aware of all the intricacies of conan, so I'm just replying without really knowing if this would be feasible with or without changes to how packages are created in conan.

• for multi-lib packages, I'd hope something similar to how Qt handles it would be possible (check https://www.kdab.com/modern-cmake-with-qt-and-boost/). Qt5::Widgets is a library. I'm not sure how that is implemented in cmake, but I think it is only available for imported targets. This way the top would be the package name (e.g. boost) and the imported libraries (if any) would be represented with sub-targets (not sure on the nomenclature): e.g boost::regex.
• For packages with both libraries and executables, I'm not really sure. Maybe they could both be imported? I'm not aware of how this is handled with the current cmake generator. For header-only libraries I think it would be enough to generate a cmake target that just had include information set on it via target_include_directories.
• not sure how we could support packages with both static and shared libraries. Maybe importing two targets, one for each version ?
• interdependencies between multiple targets in the packaged libraries would be indeed lost.

I wasn't really thinking of introducing changes to the conan packaging, so I have no suggestions at this point on how to add the missing information.
I was really only thinking of something like what you suggested in you CONAN_INCLUDE_DIRS_BOOST_TRANSITIVE example, but with an extra target already defined with the transitive properties already associated and ready for use.

I'm happy to help in any way I can.

[memsharded]

• What I mean, is that the package creator should always model the dependencies of the internal libraries. Lets take the boost example, it has some compiled libs, like regex, coroutine, thread, filesystem... They have dependencies between them, so e.g. (this is an example, not real), coroutine might depend on thread, which in turn depends on a header library for preprocessor. The only way that a consumer can use/link against Boost::coroutine is if the package explicitely models such dependencies between libraries. In the case of boost this is so complicated, that there exist an application dedicated to obtain and extract those dependencies.
• When I am talking about packages containing both static and shared libraries, I mean they can contain an A static library, a B shared library, a C header only library and some D1, D2 executables, with dependencies between them again. This can only be managed if the package creators explicitely models all of they output artifacts. Note that now in many cases it is not necessary, you can copy the artifacts with cmake install and collect the library names with tools.collect_libs

The problem is that it is not an isolated case, all reference packages have this multi-lib approach: Qt, OpenCV, Boost, Poco... And they will basically will not work with the targets approach, because as you said, "interdependencies between multiple targets in those package libraries will be lost"

Yes, I think that a target per package would be very cool, but up to my knowledge, that cannot be done, IMPORTED targets must be either STATIC, SHARED or INTERFACE. If there is some way we could just associate some include directories, library directories, libraries names, compile flags to a cmake target to be used in our projects, that would be amazing. Do you know any workaround around this limitation? Thanks!

[abuszta]

There is possibility to create for *-config.cmake files which could contain something like this for e.g. package/0.1:

add_library(package INTERFACE)

set(PACKAGE_LIBS)
foreach(_lib ${CONAN_LIBS_PACKAGE})
    find_library(_lib_path NAMES ${_lib} PATHS ${CONAN_LIB_DIRS_PACKAGE})
    list(APPEND PACKAGE_LIBS ${_lib_path})
endforeach()

set_target_properties(package
    INTERFACE_INCLUDE_DIRECTORIES "${CONAN_INCLUDE_DIRS_PACKAGE}"
    INTERFACE_COMPILE_DEFINITIONS "${CONAN_DEFINES_PACKAGE}"
    INTERFACE_LINK_LIBRARIES "${PACKAGE_LIBS}"
)

Then if path to package-config.cmake is in CMAKE_PREFIX_PATH, you can simply write in your CMakeLists.txt:

add_executable(hello main.cpp)

find_library(package CONFIG)
target_link_libraries(hello PRIVATE package)

All compile definitions and include directories, and link libraries (even if there are multiple ones) will be provided.

Having CONAN_INCLUDE_DIRS_*_TRANSITIVE could also extend this config file to contain also transitive dependencies on package level (just add find_package(dependency_name) and proper name to INTERFACE_LINK_LIBRARIES).

[memsharded]

Yes, maybe the idea of using an INTERFACE target per package could be good enough.
Also, it might not be necessary to create them in *config.cmake files, we could add them directly in conanbuildinfo.cmake, maybe inside a macro:

include(${CMAKE_BINARY_DIR}/conanbuildinfo.cmake)
conan_basic_setup()
conan_create_package_targets()

add_executable(hello main.cpp)
target_link_libraries(hello PRIVATE package)

Basically no need to find anything, as the paths and everything are already clearly defined.

[SSE4]

+1 for this, currently I create files .cmake for each package, for instance boost.cmake has the following contents:

if(conan_boost_cmake_included)
  return()
endif(conan_boost_cmake_included)
set(conan_boost_cmake_included TRUE)

add_library(boost INTERFACE IMPORTED GLOBAL)

set_property(TARGET boost PROPERTY INTERFACE_LINK_LIBRARIES ${CONAN_LIBS_BOOST})
set_property(TARGET boost PROPERTY INTERFACE_INCLUDE_DIRECTORIES ${CONAN_INCLUDE_DIRS_BOOST})
set_property(TARGET boost PROPERTY INTERFACE_COMPILE_OPTIONS ${CONAN_DEFINES_BOOST})

and then I just use:

include(boost) ... target_link_libraries(hello boost)

all .cmake files have similar content (difference is only in the package name), so it should be pretty trivial enough to write such cmake 3.x generator.

[memsharded]

Perfect @SSE4 , thanks for the feedback.
Let's go and automate this with such approach!

[ruipires]

I'm trying to implement a test solution based on the suggestion by @SSE4 , since it was the one closer to what I had in mind (never really grokked cmake's *config.cmake files).

I got it to work (still not fully automated), but I can't seem to find a way to get the transitive dependencies from the conans model (I'm looking at build_info.py).

In my test I'm using sqlite3pp and wanted to just link with it (and not reference its two immediate dependencies: sqlite3 and silicium).

[memsharded]

Yep, it might need some internal scaffolding to expose the transitive dependencies easily. Will try to help when possible with that, and update so you could finish your implementation with them. Thanks very much for helping with this!

[ruipires]

My pleasure!

I'm still not familiar with the code, but if you tell me roughly where they are, I can try to get them exposed, at least just an initial poc implementation.

[memsharded]

I am working on this, just a question for @SSE4
How do you manage the following variables?:

CONAN_CXX_FLAGS_{dep}
CONAN_SHARED_LINKER_FLAGS_{dep}
CONAN_EXE_LINKER_FLAGS_{dep}
CONAN_C_FLAGS_{dep}

Maybe put DEFINES in COMPILE_DEFINITIONS and FLAGS in COMPILE_OPTIONS? What about the linker flags? Interface targets doesn't seem to have a value for those. Thanks!

[SSE4]

@memsharded I really didn't use neither compiler flags nor linker flags in my conan receipts. I personally think 3rd-party components shall provide just headers, libraries and rarely definitions (like BOOST_ALL_NO_LIB or LIBXML2_STATIC), but overriding generic compiler/linker options from the 3rd-party might be confusing or even dangerous - I prefer developer to build dependencies and application with compatible compiler/linker flags.
however, it's pretty logical to set CONAN_CXX_FLAGS & CONAN_C_FLAGS into COMPILE_DEFINITIONS. for LINKER_FLAGS, I guess they can safely go into target_link_libraries (per http://stackoverflow.com/questions/33290879/cmake-setting-a-linker-flag-on-a-static-library-to-be-used-by-the-consumer-whe).
by the way, could you point me to some receipt(s) which set custom CXX/LINKER flags?

[SSE4]

for DEFINES into COMPILE_DEFINITIONS - I've initially tried that, and it worked like a charm with Visual Studio. however, when I've tried it on OSX, I've got compile errors because of parsing issues (I've got some command line like -D"-DBOOST_ALL_NO_LIB" where -D was appended twice, as result clang failed to parse resulting command line. that's why I've used COMPILE_OPTIONS in my example.

[memsharded]

I am having some trouble to define the libraries directories for INTERFACE IMPORTED targets, so I still have to fallback to link_directories(${CONAN_LIB_DIRS}). Is there any way I can set the link directories for target only to ${CONAN_LIB_DIRS_mypkg}?

[ruipires]

Looks like you shouldn't have to: https://cmake.org/cmake/help/v3.4/command/link_directories.html

"(...) Pass these absolute library file paths directly to the target_link_libraries() command. CMake will ensure the linker finds them."

Not sure if this applies to imported targets, the docs are not very clear.

[memsharded]

I have been able to make an initial proposal, in my branch: https://github.com/memsharded/conan/tree/feature/cmake_targets

Still lacking a few things, like checking cmake>3.0, refactors to avoid repeating cmake stuff, etc, but it passes tests, including diamond dependencies, etc.

The core cmake is something like (for each dependency {name}):

add_library({name} INTERFACE)
set_property(TARGET {name} PROPERTY INTERFACE_LINK_LIBRARIES ${{CONAN_LIBS_{uname}}})
set_property(TARGET {name} PROPERTY INTERFACE_INCLUDE_DIRECTORIES ${{CONAN_INCLUDE_DIRS_{uname}}})
set_property(TARGET {name} PROPERTY INTERFACE_COMPILE_DEFINITIONS ${{CONAN_DEFINES_{uname}}})
set_property(TARGET {name} PROPERTY INTERFACE_COMPILE_OPTIONS ${{CONAN_CFLAGS_{uname}}} ${{CONAN_CXX_FLAGS_{uname}}})
set_property(TARGET {name} PROPERTY INTERFACE_LINK_FLAGS ${{CONAN_SHARED_LINKER_FLAGS_{uname}}} ${{CONAN_EXE_LINKER_FLAGS_{uname}}})
target_link_libraries({name} {deps})

The important change in the model is that I had to add a new field to CppInfo to store the ordered transitive dependencies for each dependency, so they can be specified in the target_link_libraries().

TODO: Implement serialization and deserialization from conanbuildinfo.txt

Please, if you could try to run from sources, check with your projects and give feedback, that would be great. Thanks!

[ruipires]

I had some issues with the current implementation.
I tested only under OSX.

Using sqlite3pp and spdlog as my immediate dependencies in a simple test project.

sqlite3 was not being linked (resulting in undefined symbols). This seemed to be because the imported library had the same exact name as the real library name -lsqlite3.
Changing the imported name to something else fixed this issue.

I also had a similar problem to @SSE4 , where I had extra "-D". Somehow cmake was applying the defines via an actual define in some file (need to look further into this).

Also had a few warnings about not finding libraries in some files (for the header only libraries).

target_link_libraries was also complaining about this being an imported target, moving the deps to INTERFACE_LINK_LIBRARIES seemed to help, but I did not take the time to make sure that was the correct way to do it in cmake.

When I can spare some time I'll try and fix the issues I found.

@memsharded : thanks for the initial implementation!

[memsharded]

Excellent, thanks for trying and your report.

Ok, then I will let you work one these cmake issues, and when I have some time I will also help with more scaffolding, as refactoring the common cmake stuff, protecting it for cmake<3.0, adding some specific tests, etc. Thanks!

[ruipires]

Hi,

I got around to making the needed changes to make this work with my (simple) use case.
You can check them out here: https://github.com/ruipires/conan/tree/feature/cmake_targets

Please let me know what you think.

I had to rename the imported libraries to x:: as a workaround for the clashes with the original concrete libraries (chose x to stand for eXternal).

Fixed the -D issue (did not run the conan regression yet, so I really don't know if I broke some other behaviour).

The warnings are also happening with the cmake generator in master, so I think that should be addressed elsewhere (solution seems simple, if we can check for empty directories).

[memsharded]

The x:: seems fine, I'd use probably another name, as conan:: or the package name if possible, but is fine. Also the other changes.

The problem I see is the change for -D, I think it will break current behavior and tests, but I don't know a good way to fix and support both approaches. Maybe it is necessary to create a new field. I'll run the tests when possible and will report back. Thanks for your work!

[memsharded]

I am resuming on this task, lets see if we can make it to next conan 0.17.
I have merged @ruipires proposal into my branch, I am working on it.

To deal with the -D, I propose just checking it, should be the more robust:

defines = []
        for d in deps_cpp_info.defines:
            if d.startswith("-D"):
                defines.append(d)
            else:
                defines.append("-D%s" % d)
        self.defines = "\n\t\t\t".join(defines)

I have also changed x:: by CONAN_PKG:: I think it is a bit more clear what the target is.

I am running the test suite forcing the use of targets in every test, and it it passing. Changes are pushed, if someone wants to try with my branch (https://github.com/memsharded/conan/tree/feature/cmake_targets).

Missing things:

• Refactor common cmake code, clean it up.
• Duplicate some tests to run with targets, besides the normal tests (without manually forcing the whole test suite to use them)
• The global link_directories(${CONAN_LIB_DIRS}) is still there in conan_flags_target_setup. I have not been able to move it to the imported targets. Any idea?

[SSE4]

@memsharded as I understand, usage of link_directories is not recommended (per https://cmake.org/cmake/help/v3.0/command/link_directories.html)
instead, find_library shall be used in conjunction with target_link_libraries

[memsharded]

Yes, I know it, but cannot find a way to apply it to define the INTERFACE IMPORTED target. Any specific advice about how to use the find_library & target_link_libraries for that? Thanks!

[SSE4]

@memsharded the following should work:

if(conan_zlib_cmake_included)
  return()
endif(conan_zlib_cmake_included)
set(conan_zlib_cmake_included TRUE)

find_library(ZLIB_LIBRARY NAME ${CONAN_LIBS_ZLIB} PATHS ${CONAN_LIB_DIRS_ZLIB} NO_DEFAULT_PATH)
message(STATUS "ZLIB_LIBRARY ${ZLIB_LIBRARY}")

add_library(zlib INTERFACE IMPORTED GLOBAL)

set_property(TARGET zlib PROPERTY INTERFACE_LINK_LIBRARIES ${ZLIB_LIBRARY})

set_property(TARGET zlib PROPERTY INTERFACE_INCLUDE_DIRECTORIES ${CONAN_INCLUDE_DIRS_ZLIB})

P.S. there probably should be loop over CONAN_LIBS_ZLIB here in general case

[memsharded]

Great, that made it, thanks! I have implemented a loop, everything seems to work:

foreach(_LIBRARY_NAME ${{CONAN_LIBS_{uname}}})
    unset(FOUND_LIBRARY CACHE)
    find_library(FOUND_LIBRARY NAME ${{_LIBRARY_NAME}} PATHS ${{CONAN_LIB_DIRS_{uname}}} NO_DEFAULT_PATH)
    set(CONAN_FULLPATH_LIBS_{uname} ${{CONAN_FULLPATH_LIBS_{uname}}} ${{FOUND_LIBRARY}})
endforeach()

add_library({name} INTERFACE IMPORTED)
set_property(TARGET {name} PROPERTY INTERFACE_LINK_LIBRARIES ${{CONAN_FULLPATH_LIBS_{uname}}} {deps})

It is pushed to my branch https://github.com/memsharded/conan/tree/feature/cmake_targets

@ruipires, @abuszta, if you could please try the branch with your projects, that would help a lot.

TODO:

• refactor common cmake code
• condition the targets code to CMake>3.X
• proper testing, ensure the standard test suite passes with cmake 2.8

[SSE4]

@memsharded I just realised that it makes sense to check result of find_library.
for instance, in certain cases CONAN_LIBS_{name} may contain some system libraries, like ws2_32, shlwapi or dl, pthread, etc. in that case I believe just library name shall go into INTERFACE_LINK_LIBRARIES.

[memsharded]

Thanks @SSE4 , I just submitted a full proposal with that fix, as well as other pending tasks. Also rebased my branch: #726.

Testing on my branch still very welcome, the more we test, the best will be released into next release. Thanks!

[ruipires]

Done some testing on my very simple project.
Sadly, the -D issue is still posing some problems.

Here's the error I got:
`In file included from :357:
:1:9: error: macro name must be an identifier

define -DBOOST_USE_STATIC_LIBS 1`

From what I can tell your fix is ensuring the -D is prepended.
Looks like cmake is expecting command line define flags if I call target_compile_definitions(app PRIVATE ${CONAN_DEFINES}) directly, but actual define names (with no -D) for set_property INTERFACE_COMPILE_DEFINITIONS.

Couldn't find any reference to this behaviour on their documentation.

That's why I resorted to the use_define_token=False hack.

Maybe adding another list like CONAN_<PKG>_COMPILE_DEFINITIONS with the prepended -D stripped out would make it cleaner/more explicit?

[memsharded]

Yes, I like your proposal to make it explicit, I have updated my branch with it: https://github.com/conan-io/conan/pull/726/commits/cf08d1e455b0d624fadb2dcceb01359ac8cd379a

The PR is passing CI https://github.com/conan-io/conan/pull/726, but if you can try for your project, please tell me.

[ruipires]

As of 21695773863f73f91168a0016084f73ed062195f, it now works for my simple project. I'll try to do some tests with a more complex use case.

[memsharded]

I have updated the new cmake with targets in the conan develop branch, to fix bug reported by @jgsogo, and also for a refactor, defining the targets inside a macro instead of at global scope. If you are able to try, please update from conan develop branch, thanks!

[memsharded]

This seems mostly implemented by #726, and already in develop, labeling as fixed, will be released in 0.17.