Conan: Making requirements available for cmake find

[danielgindi]

To help us debug your issue please explain:

• [X] I've read the CONTRIBUTING guide.
• [X] I've specified the Conan version, operating system version and any tool that can be relevant.
• [X] I've explained the steps to reproduce the error or the motivation/use case of the question/suggestion.

Conan version: 1.9.2 (due to python 2.7 requirements from other modules)

My question could be very simple, as it looks like I'm missing something.

I have a package X (actually caffe2) that I'm trying to create a package for.
Now it has multiple requirements, that I can define dynamically in the conanfile.py, set cmake options accordingly etc.
But no matter what I do - the package can't find its dependencies and marks them as not available.

I've tried generators = ["cmake" , "cmake_find_package"]
And it creates both conanbuildinfo.cmake and the Findxxxx.cmake files.
But it's not available for the package's main CMakeLists.txt, and not included in anyway.
Is it my job to modify the CMakeLists.txt during the build stage to include conanbuildinfo.cmake?
Or is there an option I'm missing?

Thanks!

[uilianries]

Hi @danielgindi ! Did you post your project on Github or Gitlab? Could you please share the link? We could help you :smile_cat:

But no matter what I do - the package can't find its dependencies and marks them as not available.

Did you include conanbuildinfo.cmake and run conan_basic_setup() for caffe2 package?

You could try a wrapper as we did for Cereal or inject directly in cmake as we did for Paho. The first option is cleaner and looks better I think, but the result is the same for both approaches.

Regards!

[Johnnyxy]

If you use the cmake_find_package generator you have to issue a CMake find_package (...) command in your CMake somewhere.

find_package (zlib) # looks for a Findzlib.cmake
target_link_library (MY_TARGET PRIVATE zlib::zlib)

Conan sets the CMake variable CMAKE_MODULE_PATH with the path/directory of the Findxxxx.cmake files.
The CMake command find_package looks into those paths if it can find a Findxxxx.cmake and executes it. After that you can work with the libraries etc. that are defined by the Findxxxx.cmake.
This way nothing else is need to integrate Conan into CMake.

Another approach would be the normal "cmake"-generator. With that you have to include the conan generated conanbuildinfo.cmake and execute it. this is typically done at a top-level-CMakelists.txt-file.
Have a look here.

[danielgindi]

Thanks guys for the answers :-)
I haven't pushed it anywhere yet, as it's a work in progress... Huge library!

Anyway - with the cmake_find_package generator it did not work, as find_package calls in the cmake files found nothing.
I tried with injecting into the CMakeLists.txt file, which kind of worked. It found some of the packages, but some of them not.
Wrapping in a pre-baked CMakeLists.txt sounds much cleaner.

I think I should try with the cmake_find_package - problem is they don't get recognized. Should I set the CMAKE_MODULE_PATH manually?

[uilianries]

cmake_find_package is limited, it does not find transitive dependencies, but the most important point is the module name. Usually we distribute our packages using lower case and maybe your cmake tries to find some capitalized name and it will not be able to be found, because cmake_find_package will generate the Find.cmake according the package name. For example, our Boost package is named boost, thus we will provide Findboost.cmake as well, instead of FindBoost.cmake as spectated by most of cmake projects.

Maybe you will need add some conditional forcing your cmake file to include all cmake files provided by Conan, or you will need to patch your cmake to find the correct target name. I know, this is boring, but we have some explanation in a blog post.

[Johnnyxy]

Steps to debug

Conan CMake command line

Check what Conan calls CMake with.

cmake = CMake(self) # replace with your call
self.output.info(self.name + ": " +" using cmake ({command line: " + cmake.command_line + "})")
raise Exception("DEBUG: stopping Conan")

Now you get a long output with the actual CMake parameters (for CMake "configure" stage).
There you should find the following:
-DCMAKE_MODULE_PATH="PATH_TO_YOUR_CONAN_INSTALL_FOLDER"
There can be more (with a semicolon delimiter) but normally this is the only one.

CMAKE_MODULE_PATH content

Check the content of CMAKE_MODULE_PATH right before your first find_package.

message (STATUS "CMAKE_MODULE_PATH == ${CMAKE_MODULE_PATH}")
message (FATAL_ERROR "DEBUG: stopping CMake")

The variable should consist of the content from the Conan-CMake call with maybe additional paths.

find_package call

Check your find_package call. You want to use the "module" mode of the command. See the CMake docs for more informations.
The Findxxxx.cmake files contain your dependencies. CMake looks for Find<package-name>.cmake. This means you have to make sure that your <package-name> in find_package (<package-name>) looks exact the same name as the Find<package-name>.cmake-file has. Propably there is a type-o with upper/lower case.

Additionally put a "REQUIRED" into your find_package to make sure it stops right there when it does not find the dependency.

Conan Findxxxx.cmake content

Cou could check if the content of the generated Find-files is correct. Check if:

SET(<package-name>_LIBRARY_LIST) # should list all library names you need
SET(<package-name>_LIB_DIRS) # should list all paths you need

[uilianries]

Also, you could set cmake args to trace or debug:

    def build(self):
        cmake = CMake(self)
        cmake_args = "--trace" # or could be "--debug-output"
        cmake.configure(args=[cmake_args])
        cmake.build()

ref: https://docs.conan.io/en/latest/reference/build_helpers/cmake.html#configure

[danielgindi]

So I've come up with this handy little function:

    def add_find_package_case(self, file_name, name, toNames):
        with open(file_name, 'r') as f:
            content = f.read()

        appendix = ""

        if not isinstance(toNames, list):
            toNames = [toNames]

        for toName in toNames:
            for key in ["FOUND", "INCLUDE_DIRS", "INCLUDES",
                        "DEFINITIONS", "LIBRARIES", "LIBRARIES_TARGETS",
                        "LIBS", "LIBRARY_LIST", "LIB_DIRS"]:
                appendix = appendix + "set(" + toName + "_" + key + " ${" + name + "_" + key + "})\n"

        content = content + "\n\n" + appendix

        with open(file_name, "wb") as handle:
            handle.write(content)

        # For case-sensitive file-systems, keep all known casings available
        for toName in toNames:
            try:
                shutil.copy(file_name, os.path.join(os.path.dirname(file_name), "Find" + toName + ".cmake"))
            except:
                pass

Calling it like that:

        self.add_find_package_case("Findopenblas.cmake", "openblas", "OpenBLAS")
        self.add_find_package_case("Findopencv.cmake", "opencv", "OpenCV")
        self.add_find_package_case("Findsnappy.cmake", "snappy", "Snappy")
        self.add_find_package_case("Findgflags.cmake", "gflags", "GFLAGS")
        self.add_find_package_case("Findopenexr.cmake", "openexr", "OpenEXR")

Allows caffe2 to find the packages correctly.
The weird thing is gflags is being searched for as gflags, looking for a file named Findgflags.cmake, but then it looks for GFLAGS_FOUND - upper case.
So this function keeps both casing in the Findxxxx.cmake file, and on non-windows machines it will also keep multiple filenames for case-sensitive search.

Maybe this could be supplied in the conan-tools. Seems like's it's a must for large CMake projects.

[Johnnyxy]

So you say it was a casing (lower/upper letters in package name mismatch) problem.

On Windows this should not be a real problem as Win32-API (not the filesystem NTFS) is not case sensitive. With Windows 10 one could turn on case sensitivity but that is not the point here :).

Anyway I strongly advise you _not to circumvent case-sensitivity_. Most filesystems/operating systems are case-sensitive. If you equate two filenames which differ only in their filename case, you will run into severe problems sooner or later that you have to workaround with much pain.

If you need a package that has a different name, then you should use Conan's alias functionality.

[danielgindi]

The problem is deeper than that; It's not just a filename case issue, it's how they are exported.
For example gflags has a filename Findgflags.cmake, and exported as gflags_FOUND. But consumers of gflags are looking for GFLAGS_FOUND, case sensitive.
I guess that on some systems there's a built-in Findgflags.cmake that exports GFLAGS_FOUND, but on Windows it won't be available, and you take it from conan. Conan in it's turn will export it with lowercase variables.

An alias won't fix that, as conan always exports for find_package in lowercase letters.

[Johnnyxy]

If Conan applies case-consistency I think this is better so far. If someone now needs different packages that in turn are the same like

find_package (zlib)
or
find_package (ZLIB)

then this is no conan problem instead more of a CMake-Find-Modules-problem.
If one needs to support mulitple package-names the solution with a proxy-target could apply here (see).

[danielgindi]

Hey @Johnnyxy I'm really not complaining about conan, but about CMake. Cases sensitive variables in CMake is a really bad idea (especially given function names are insensitive!).
I could complain about the CMakeLists.txt file makers (i.e in caffe2 repo and other repos), but they do not really have a choice, they only do what works for them in the CMake world.
Conan trying to apply consistency there is really blessed.
But I found out that very often there's a need to patch those cmake files to get things working.

The above proposed function really solves most of the disagreement between conan and cmake, and between cmake and itself.

[Johnnyxy]

@danielgindi Sorry I did not imply that you are complaining :).

Can you elaborate who is checking this the GFLAGS_FOUND variable?
Your CMakeLists or the CMakeLists of the third party libs?

I want to understand if and how one could workaround such a problem with Conan or CMake functionality. The proposed function is one solution. I am just thinking:

Could there be another (Conan or CMake) solution? :).

[danielgindi]

It's all about third party libs. If it was my own lib, I would have written the CMake file to be bullet-proof to begin with. I'm not happy with considering only happy-flows...

Well as for another solution- I've looked at the CMake docs, and did not find anything else that could assist me. Even looked for a way to make variables case insensitive, but that's a no-go too.
As for a conan solution - you guys are the experts :-) I'm finding here about conan features that I didn't know about.

Anyway my creativity today only went so far as to add aliases inside the Findxxxx.cmake files.
If we wanted to avoid that hack, maybe the conan generator itself could benefit from a new feature, where it would be possible to specify aliases.

[Johnnyxy]

@danielgindi your implemented function essentially replaces the given string with another string in the generated Findxxxx.cmake files.
As alternative you could use the Conan tool-function tools.replace_in_file().
I do not know if the string search of that function is case-sensitive, but I do not think it has a custom string comparison.

If you have to do more sophisticated logic then of course a manual implementation is needed.

Best regards :)

[danielgindi]

@Johnnyxy nope, the function above does not do a replace. It creates aliases inside the Findxxxx.cmake files.

Example from a (now working, finally) caffe2 recipe:
Findgflags.cmake:

message(STATUS "Conan: Using autogenerated Findgflags.cmake")
# Global approach
SET(gflags_FOUND 1)
SET(gflags_INCLUDE_DIRS "C:/Users/Daniel/.conan/data/gflags/2.2.1/bincrafters/stable/package/6552a0da13846f89f03c064d8e31c69781115e0a/include")
SET(gflags_INCLUDES "C:/Users/Daniel/.conan/data/gflags/2.2.1/bincrafters/stable/package/6552a0da13846f89f03c064d8e31c69781115e0a/include")
SET(gflags_DEFINITIONS )
SET(gflags_LIBRARIES "") # Will be filled later
SET(gflags_LIBRARIES_TARGETS "") # Will be filled later, if CMake 3
SET(gflags_LIBS "") # Same as gflags_LIBRARIES
.
.
.
.
set(GFLAGS_FOUND ${gflags_FOUND})
set(GFLAGS_INCLUDE_DIRS ${gflags_INCLUDE_DIRS})
set(GFLAGS_INCLUDES ${gflags_INCLUDES})
set(GFLAGS_DEFINITIONS ${gflags_DEFINITIONS})
set(GFLAGS_LIBRARIES ${gflags_LIBRARIES})
set(GFLAGS_LIBRARIES_TARGETS ${gflags_LIBRARIES_TARGETS})
set(GFLAGS_LIBS ${gflags_LIBS})
set(GFLAGS_LIBRARY_LIST ${gflags_LIBRARY_LIST})
set(GFLAGS_LIB_DIRS ${gflags_LIB_DIRS})

And on a case-sensitive file system, it will also duplicate Findgflags.cmake into FindGFLAGS.cmake so different find_package statements will still find the package correctly.

[Johnnyxy]

Ah I see :).
So in summary this is a general problem with third party CMakes. And can be used to workaround their CMake-variable-casings shortcomings if they rely on CMake variables instead of CMake target names.

[danielgindi]

Right :-)
There are also some cases where target names are incorrectly cased, as some Findxxxx are lower cased by conan but upper cases in other envs.

[uilianries]

We have a similar problem for Folly, the solution was apply a patch when running Conan: https://github.com/bincrafters/conan-folly/blob/6315917428541f4fddb132e408eae0b1c76831a0/folly.patch#L9

[danielgindi]

@uilianries Applying patches is fine for a small library, but not for a huge library like caffe2 and others, where there are dozens of cmake files, and dozens of third party subfolders with their own dozens of cmake files :-)