Dlib: Seamless integration with package managers

[Morwenn]

Hi,

I tried to package dlib so that it works with Conan: it's mostly good and most of it worked on the first try, but the try_compile sanity checks in find_libjpeg.cmake and find_libpng.cmake make it impossible to package dlib if those libraries are only installed through the package manager and not directly in the system directories.

Expected Behavior

The CMake support for Conan allows to download libraries in ~/.conan and automatically changes variables such as CMAKE_PREFIX_PATH to point to the Conan directories, which allows find_package to find the required libraries in the Conan directories. If the CMake files don't do anything specific, everything works smoothly enough.

Current Behavior

When I try to use the CMake/Conan support to package dlib, the libraries (libjpeg, libpng, etc.) downloaded by Conan are correctly found by find_package, but the subsequent sanity checks in find_libjpeg.cmake and find_libpng.cmake fail.

The reason is that the aforementioned tests use try_compile to compile the test projects in test_for_libjpeg and test_for_libpng, and those call do not propagate the CMake environments to the callees (specifically the variables that tell CMake where to find its packages), so their find_package calls fail if libjpeg and libpng are only installed through Conan and don't exist in the system directories.

It should be possible to forward the required variables somehow, probably through the CMAKE_FLAGS argument of try_compile.

Steps to Reproduce

If you're familiar enough with Conan, here is a stripped down version of the recipe I used to fetch, compile and install dlib (requires Python 3.7):

import os
from pathlib import Path

from conans import ConanFile, CMake, tools


class LibDlibConan(ConanFile):
    name = "dlib"
    version = "19.18"
    url = "https://github.com/davisking/dlib"
    generators = "cmake"
    settings = "os", "arch", "compiler", "build_type"

    _source_subfolder = "source_subfolder"
    _build_subfolder = "build_subfolder"

    scm = {
        "type": "git",
        "subfolder": _source_subfolder,
        "url": "https://github.com/davisking/dlib.git",
        "revision": f"v{version}"
    }

    def source(self):
        # Tell CMake about the subdirectories
        tools.replace_in_file(
            Path(self._source_subfolder) / "dlib" / "CMakeLists.txt",
            'project(dlib)',
            f'''project(dlib)
                include(${{CMAKE_BINARY_DIR}}/../conanbuildinfo.cmake)
                conan_basic_setup()'''
        )

        # Avoid try_compile issues at the cost of fewer checks 
        tools.replace_in_file(
            Path(self._source_subfolder) / "dlib" / "cmake_utils" / "find_libjpeg.cmake",
            "if(JPEG_FOUND)",
            "if(NOT JPEG_FOUND)"
        )
        tools.replace_in_file(
            Path(self._source_subfolder) / "dlib" / "cmake_utils" / "find_libpng.cmake",
            "if(PNG_FOUND)",
            "if(NOT PNG_FOUND)"
        )

    def requirements(self):
        self.requires("libjpeg/9c")
        self.requires("libpng/1.6.37")
        self.requires("zlib/1.2.11")

    def build(self):
        cmake = CMake(self)
        cmake.configure(build_folder=self._build_subfolder)
        cmake.build()

    def package(self):
        self.copy("*.a", "lib", keep_path=False)
        self.copy("*.lib", "lib", keep_path=False)
        self.copy("*.so*", "lib", keep_path=False)
        self.copy("*.h", "include/dlib", os.path.join(self._source_subfolder, "dlib"))
        self.copy("LICENSE.txt", "licenses", os.path.join(self._source_subfolder, "dlib"), keep_path=False)

    def package_info(self):
        self.cpp_info.libs = tools.collect_libs(self)

As you can see I worked around the issue by skipping the sanity checks for libjpeg and libpng entirely, but I would rather keep said checks if possible. Forwarding the appropriate CMake variable to try_compile instead would probably help to keep the checks while being package manager-friendly :)

Having an installation step in CMake to simplify the process would be nice too, but that's a different issue and I seem to recall that it was already discussed somewhere else.

• Version: 19.18
• Where did you get dlib: GitHub releases
• Platform: Ubuntu 18.04 64-bit
• Compiler: irrelevant

[davisking]

You can pass arbitrary arguments to cmake in try_compile by using CMAKE_FLAGS. For example,

try_compile (thing binary_dir source_dir 
   CMAKE_FLAGS  "-DCMAKE_PREFIX_PATH=/whatever/you/want"
)

This is what is done in some of the more elaborate try_compile invocations in dlib/CMakeLists.txt.

There is also a cmake install target. It runs just like any other cmake script's install target. I'm not really sure what exactly you are doing, as I'm not familiar with how conan is setup. But you have to be mindful that the preprocessor switches that are on when dlib is built are used by client code. Normally cmake takes care of this for you, but I get the impression none of the cmake artifacts that enable it to do this are being output by this conan script. Alternatively, if you ran the install target it would generate a config.h file that ensures the right preprocessor settings are propagated to client code, but that doesn't seem to be happening either.

[Morwenn]

Oops, I realized my mistake for the install targets: a usual way to package CMake projects in Conan is to make a top-level CMakeLists.txt file (often called CMake wrapper) that sometimes configures a few variables and then includes the project to wrap through add_subdirectory (the example above was stripped-down for simplicity, and of course it didn't include the part that actually mattered for that...). In the case of dlib it apparently made dlib conclude that it wasn't the main project and that it shouldn't provide the install targets. Explicitly passing -DDLIB_IN_PROJECT_BUILD=OFF from Conan made them available and working again.

I will try to play with CMAKE_FLAGS a bit to fix my libjpeg & libpng issue. Once I get it to work, is it something you'd want as a pull request?

[davisking]

I don't know enough about conan to know what to do with this script. It doesn't go into a conan repository server or something?

[Morwenn]

If you have Conan installed you can open a terminal in the directory containing that script and launch it with something along the lines of conan create . davisking/testing (what goes after the dot doesn't matter much to reproduce) and it will download and build the version 19.18 of dlib, and install it under ~/.conan/data/dlib. If you don't have libjpeg or libpng installed on your system it should be enough to reproduce the issue.

[davisking]

Sure, but that doesn't seem like something that should live in the dlib repository. Since if you have the dlib repository, you have already downloaded dlib.

[arrufat]

I am sorry to interrupt but what's wrong with using CMake's built-it "dependency manager"?
I find it much more convenient to manage the dlib dependency.
The only drawback is that you need CMake 3.14 (but you can get it with pip easily, anyway).
This is an self-contained example. I think it's pretty neat.

cmake_minimum_required(VERSION 3.14)
project("demo" LANGUAGES CXX)
include(FetchContent)
FetchContent_Declare(
    dlib
    GIT_REPOSITORY https://github.com/davisking/dlib.git
    GIT_TAG        v19.18  # you can even use master
)
FetchContent_MakeAvailable(dlib)
add_executable(demo src/main.cpp)
target_link_libraries(demo PRIVATE dlib::dlib)

Again, sorry if this is a bit off-topic... But maybe dlib documentation could be updated to provide this info for "modern CMake" users.

[Morwenn]

@davisking Oh, I didn't mean to propose the Conan recipe for dlib, I only shared it as a mean to reproduce the error I had with the libjpeg/libpng tests and to give some context. The only thing I'd like from dlib would be the proper the forwarding of CMAKE_BUILD_PREFIX, CMAKE_INCLUDE_PATH and CMAKE_LIBRARY_PATH to those tests :)

The reason I'm using Conan is that we've got like 60 libraries (and growing) in our stack so far, and a good chunk of them simply doesn't use CMake: some do use CMake, but some use autotools, some use bazel, others are simply downloadable hearders & binaries, etc. So far the model has been incredibly useful to manage everything in a uniform manner (and to benefit from recipes for trickier packages maintained by the community).

[davisking]

No worries. You should submit a PR that updates the try_compile statement to forward these things. It's a lot easier if you do it, test it, and submit it than if someone else puts up a PR, you then later test it, find that some other thing is missing/needed for conan, you ask about it, they guess what the solution is, etc.

[davisking]

@arrufat That's a good idea, I just added a note about it to https://github.com/davisking/dlib/blob/master/examples/CMakeLists.txt.

[arrufat]

@davisking thanks for adding this to the CMake examples.
However, FetchContent_MakeAvailable was added in CMake 3.14, should I make a pull-request to update the example, which says 3.11?

Reference: https://cmake.org/cmake/help/latest/module/FetchContent.html

[davisking]

You sure? It’s in the docs for 3.11: https://cmake.org/cmake/help/v3.11/module/FetchContent.html

On Oct 24, 2019, at 8:49 AM, Adrià Arrufat [e-mail] wrote:


@davisking thanks for adding this to the CMake examples.
However, FetchContent_MakeAvailable was added in CMake 3.14, should I make a pull-request to update the example, which says 3.11?

Reference: https://cmake.org/cmake/help/latest/module/FetchContent.html

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[arrufat]

Yes, FetchContent is available since 3.11, but FetchContent_MakeAvailable since 3.14.
That's what they say in https://cmake.org/cmake/help/latest/module/FetchContent.html:

_The following shows a typical example of declaring content details:_

FetchContent_Declare(
  googletest
  GIT_REPOSITORY https://github.com/google/googletest.git
  GIT_TAG        release-1.8.0
)

_For most typical cases, populating the content can then be done with a single command like so:_

FetchContent_MakeAvailable(googletest)

_The above command not only populates the content, it also adds it to the main build (if possible) so that the main build can use the populated project’s targets, etc. In some cases, the main project may need to have more precise control over the population or may be required to explicitly define the population steps (e.g. if CMake versions earlier than 3.14 need to be supported). The typical pattern of such custom steps looks like this:_

FetchContent_GetProperties(googletest)
if(NOT googletest_POPULATED)
  FetchContent_Populate(googletest)
  add_subdirectory(${googletest_SOURCE_DIR} ${googletest_BINARY_DIR})
endif()

[davisking]

Ah, good catch @arrufat. I just updated the docs :)