Conan: [build system] Using toolchains and tools by invoking cmake instead of conan build

[pepe82sh]

To help us debug your issue please explain:

• [x] I've read the CONTRIBUTING guide.
• [x] I've specified the Conan version, operating system version and any tool that can be relevant.
• [x] I've explained the steps to reproduce the error or the motivation/use case of the question/suggestion.

Versions

• Microsoft Windows 10 Pro - 10.0.16299 Build 16299
• Conan 1.3.3
• Cmake 3.10.3
• arm-none-eabi-gcc 4.9.3
• CLion 2018.1

Setup

I am in an embedded environment and use the cross compiling facilities provided by conan. Specifically I deploy the toolchain (compiler and cmake toolchain) in a conan package and set self.env_info.CONAN_CMAKE_TOOLCHAIN_FILE accordingly.

Problem to be solved

I use CLion like suggested here. CLion will use cmake to build the project, and it is not possible to use cmake build for that purpose or use virtualenv. This might still work in most environments, but not when I need a specific toolchain. Even defining the toolchain manually does not solve the problem, as the compiler is not in PATH.

Suggested solution

Rather than using conan build to set PATH correctly and invoke cmake with -DCMAKE_TOOLCHAIN_FILE=..., it should be possible to outsource that into conanbuildinfo.cmake:

set(ENV{PATH} "C:\\Path\\to\\package\\bin;$ENV{PATH}")
set(CMAKE_TOOLCHAIN_FILE "C:/Path/to/toolchain.cmake")

This way the PATH and toolchain are set even when using cmake, and calling conan build isn't a necessity anymore.

Limitations of suggested solution

This only seems to work when the project() function is called after include(${CMAKE_BINARY_DIR}/conanbuildinfo.cmake) and before conan_basic_setup()

[pepe82sh]

I just tried conan 1.4 and the cmake_paths generator. That one would be more than enough for my usecase, but I have no chance of using the toolchan file. I propose the following:

• Add a variable CONAN_ENV_PATH_LIST that contains a list of all path variables
• Add a variable CONAN_CMAKE_TOOLCHAIN_FILE that has the toolchain file

CONAN_ENV_PATH_LIST could be iterated and added to the PATH like that:

foreach(PATH_ITEM ${CONAN_ENV_PATH_LIST})
    set(PATH_SEPARATOR ":")
    if(WIN32)
        set(PATH_SEPARATOR ";")
        string(REPLACE "/" "\\" PATH_ITEM ${PATH_ITEM})
    endif()
    set(ENV{PATH} "${PATH_ITEM}bin${PATH_SEPARATOR}$ENV{PATH}")
endforeach()

That part could even be provided outside conan, as long as CONAN_ENV_PATH_LIST exists.

Same goes for CONAN_CMAKE_TOOLCHAIN_FILE. One could set CMAKE_TOOLCHAIN_FILE himself, as long as CONAN_CMAKE_TOOLCHAIN_FILE exists.

This solution would be as unintrusive as possible and could fix my problem. It'd not be ideal, as I have to provide additional code instead of using just conan, but at least I could use conan at all.

[memsharded]

Hi @pepe82sh

Thanks for the detailed feedback.

I am not sure if I understood the issue completely.
Just in case, do you now about the conan virtualenv? The flow with it would be to activate the virtualenv after the conan install and before calling cmake ..., so the environment variables would be there, no need to have cmake defining them.

In any case, I think there might be a use case here that needs more investigation, will have a look.

[pepe82sh]

Hi @memsharded

I do know about virualenv and was able to use it when there is no IDE involved. Unfortunately I can't get CLion to run cmake inside the virtualenv. The CLion-Way to do such thing is to put it inside the CMakeLists.txt, and run cmd /c enable.bat as an external process. That doesn't work, because it'll run inside its own cmd process and the environment will be gone after the command finished.

I am not a python person, but I forked conan in order to make better suggestions (didn't commit anything so far). From what I can see my problem boils down to "I need the environment variables inside cmake without calling conan build". There are 2 ways to achieve that that I can think of right now:

1. The non intrusive way
   For each environment variable, we add a CONAN_ENV_VARNAME variable inside cmake. That wouldn't hurt anyone, and give the opportunity to use them in special cases like mine.

2. The more sophisticated way
   We set each environment variable inside cmake using $ENV{VARNAME}. That might become interesting with lists like PATH, where you could add items twice when invoking conan build. From what I see that could be solved by adding another variable via conan build and test if that is set before setting the environment within cmake.

In any case CONAN_CMAKE_TOOLCHAIN_FILE is special, as it is not only an environment variable, but something that should be transformed to CMAKE_TOOLCHAIN_FILE inside cmake. That'd reqire either a special treatment by conan, or by the user. Nonetheless there would be a solution for my problem however it is approached.

[pepe82sh]

I just figured that I can write my own generator based on cmake_paths and provide that as a package. That should enable me to fix all my problems without the need of changes in conan and should be the cleanest solution for everyone involved. I'll leave this open and paste the generator I came up with once I'm finished. Shouldn't take long though.

[lasote]

Hi @pepe82sh. I understand the issue. I think your idea of a generator that sets the env and include the toolchain declared in CONAN_CMAKE_TOOLCHAIN_FILE could work.

I'm a little worried about the cmake_paths generator, that would change the simple scope it was designed for, a better name for the generator now could be cmake_toolchain, because the principal use case is to be used as a toolchain in the command line.

Let me think about it and try it, and of course, if you write a generator tell me, it has general interest I think.

[pepe82sh]

Hi @lasote. I get your point and agree to your thoughts for a generator shipped with conan. I came up with my own one, which includes the functionality of cmake_paths. I'd say for a conan generator that could be dropped, as one could use cmake_toolchain as an addition to cmake_paths or cmake. include(cmake_toolchain.cmake) has to be called before project(...), so it might be a good thing it's an include on it's own.

Here's my custom generator (I'll add support for CONAN_XXX_ROOT for my specific usecase, but I think thats not interesting for a official conan generator):

import os
import platform
from conans.client.generators.cmake import DepsCppCmake
from conans.model import Generator


class cmake_toolchain(Generator):

    append_with_spaces = ["CPPFLAGS", "CFLAGS", "CXXFLAGS", "LIBS", "LDFLAGS", "CL"]

    def _cmake_escape(self, input: str):
        return input.replace("\\", "\\\\")

    def _get_cmake_environment_setters(self):
        ret = []
        for name, value in self.conanfile.env.items():
            if isinstance(value, list):
                if name in self.append_with_spaces:
                    # Variables joined with spaces look like: CPPFLAGS="one two three"
                    value = self._cmake_escape(" ".join(value))
                    ret.append("set(ENV{{{name}}} \"{value} $ENV{{{name}}}\")".format(name=name, value=value))
                else:
                    if platform.system() == "Windows":
                        value = os.pathsep.join(value)
                    else:
                        value = os.pathsep.join(['"%s"' % val for val in value])
                    value = self._cmake_escape(value)
                    ret.append("set(ENV{{{name}}} \"{value};$ENV{{{name}}}\")".format(name=name, value=value))
            else:
                value = self._cmake_escape(value)
                ret.append("set(ENV{{{name}}} \"{value}\")".format(name=name, value=value))
        return ret

    def _get_cmake_toolchain_setter(self):
        ret = []
        if "CONAN_CMAKE_TOOLCHAIN_FILE" in self.conanfile.env:
            toolchain = self._cmake_escape(self.conanfile.env["CONAN_CMAKE_TOOLCHAIN_FILE"])
            ret.append("set(CMAKE_TOOLCHAIN_FILE \"{toolchain}\")"
                       .format(toolchain=toolchain))
        return ret

    def _get_cmake_paths(self):
        ret = []
        build_paths = self._cmake_escape(DepsCppCmake(self.deps_build_info).build_paths)
        # We want to prioritize the FindXXX.cmake files:
        # 1. First the files found in the packages
        # 2. The previously set (by default CMAKE_MODULE_PATH is empty)
        # 3. The "install_folder" ones, in case there is no FindXXX.cmake, try with the install dir
        #    if the user used the "cmake_find_package" will find the auto-generated
        # 4. The CMake installation dir/Modules ones.
        ret.append("set(CONAN_PACKAGE_PATH {build_paths})".format(build_paths=build_paths))
        ret.append("set(CMAKE_MODULE_PATH ${CONAN_PACKAGE_PATH} ${CMAKE_MODULE_PATH} ${CMAKE_CURRENT_LIST_DIR})")
        ret.append("set(CMAKE_PREFIX_PATH ${CONAN_PACKAGE_PATH} ${CMAKE_PREFIX_PATH} ${CMAKE_CURRENT_LIST_DIR})")
        return ret

    @property
    def filename(self):
        return "conan_toolchain.cmake"

    @property
    def content(self):
        toolchain_setter = self._get_cmake_toolchain_setter()
        env_setters = self._get_cmake_environment_setters()
        cmake_setter = self._get_cmake_paths()

        return os.linesep.join(env_setters + cmake_setter + toolchain_setter)

[lasote]

Hi! Thanks for sharing. We will discuss it and if we agree we will introduce the generator to the Conan code base for 1.5.

[pepe82sh]

Thanks @lasote! That is an interesting feature indeed that I didn't know about! Don't know if I'll need it yet, but definitely nice to know!

[puetzk]

This set(CMAKE_TOOLCHAIN_FILE $ENV{CONAN_CMAKE_TOOLCHAIN_FILE}) feels like a feature that would be a very good fit for https://github.com/conan-io/cmake-conan as well...

[lasote]

We have been thinking about the $ cmake vs conan build and we think it deserves a deeper analysis. The env vars is only a part of the problem, (thanks for bringing it to the surface), but all the vars adjusted in the CMake() build helper (and other build helpers) are not applied only using cmake. We are delaying this until we find a better global approach to solve it.

[AndreyNautilus]

I have a similar issue: cmake's CMAKE_CXX_STANDARD variable is not set by conanbuildinfo.cmake when I try to build my package locally with conan install followed by cmake calls:

conan install . -s cppstd=11
cmake .
cmake --build .

The build fails with the reason: c++11 features are not enabled.

If I use conan commands only, everything works fine:

conan install . -s cppstd=11
conan build .

but I don't want to run the whole build method, I want to build just a small bit of my project.

Using pure cmake after conan install could be handy if you don't want to test the build method, but you only want to force a recompilation (to test that you don't have a syntax errors for example) during development.

Here is a minimal example of what I mean
conan-cxx-test.tar.gz

[memsharded]

Thanks @AndreyNautilus for the feedback.
Yes, what you report is totally aligned with this issue, definitely something that should be improved.

[pepe82sh]

BTW: I improved the example for a generator to a point where I think it should be complete (Environment and flags sent when Cmake is called). It's not fully tested, but I'd appreciate any thoughts from you guys. link

[peter-moran]

@pepe82sh I've tested your ConanCmakeToolchainGenerator for my own project, and it was exactly what I needed. It would be awesome to see it packaged with Conan.

[jgsogo]

There is a POC ongoing about toolchains: https://github.com/conan-io/conan/pull/5919