Conan: CMake with git submodules: installing packages required by dependencies

[JPGygax68]

In several of my projects, I'm using git submodules in combination with the CMake add_subdirectory() command to be able to work on the source of both the main project and one or more of its dependencies. Because the main project and its dependency often grow together, this is more than a convenience - anything else would involve tons of manual updating.

If now one of the submodules is using Conan for its own dependencies, how can I have conan install those?

[memsharded]

Not sure if I got the issue. Can't you move to the submodule folder and run conan install there? Then the submodule CMakeLists.txt would typically include the generated conanbuildinfo.cmake file.
It is possible to script this task too, I have some preliminary work to do it in CMake syntax, check: https://github.com/memsharded/cmake-conan. But I would say that a simple python script in the root of your project could be eimpler and better.
Maybe I would need to know more about the project to understand this properly. If it is not OSS and you don't want to disclose more details, please send an email.

[JPGygax68]

The problem with your suggestion is probably that I'm doing my building inside Build_XXX subdirectories (filtered out via .gitignore - I believe this is the most usual way to use CMake, even though it's not "out-of-source" in the strict sense). This makes it rather difficult to find the correct directory where to call conan install from, and even more difficult to then specify the correct relative path to the source directory.

The cmake-conan script however looks very interesting, I'm going to give it a try and report back.

[memsharded]

Maybe, if cmake is the one altering or creating your build directories, it is not something bad if you decide to have the conanbuildinfo.cmake, elsewhere, inside an alternate directory. So you could:

$ cd submodule
$ mkdir .conan && cd .conan
//add ".conan" pattern to .gitignore
$ conan install ....

then in your submodule CMakeLists.txt:

include(.conan/conanbuildinfo.cmake)

Ok, the cmake-conan, as I said, is very preliminary, specially incomplete in the detection and handling of cmake compilers, versions and settings in general, and how they are passed to conan. Shouldn't be a problem if you don't have to address many different configurations.

[JPGygax68]

Please forget my answer to your previous suggestion - that was a load of dung, there is only one build directory to do conan install from for each module, so your solution is perfectly straightforward.

The need to repeatedly specify the options for each of the submodules could be addressed by a script as you suggested, however, may I suggest that this be made into a feature? It would be very nice to be able to just specify the submodules as a list in the conanfile, e.g. for conanfile.py:

submodules = "external/gpcfontrasterizer", "external/libGPCFonts"

[memsharded]

No worries!

About your suggested feature, I am not very convinced. I'd say that it introduces some extra complexity to conan codebase, while the solution of the external script is totally straightforward. What I am saying is it could be a very large complexity for the feature added value, it is very important to keep the conan codebase as lean as possible. Would it recurse for each subdirectory for conan install only? Or maybe for conan build too? It seems that probably for other commands, as export, etc.

It is also not clear at all how this would behave for packages. Probably you are thinking mostly in consuming packages, but what if you add this submodules field to a package and upload it? When you depend on it, and install it, it will retrieve the git submodules and recurse into each one to do what? You are already installing/building the main repo which has the submodules inside. It doesnt make much sense to build the submodules, they should be conan packages themselves, shouldn´t they?

Also, because this feature comes from the fact that a different way to handle dependencies is being mixed. One of the best things of conan, is that it largely avoids using (the nightmare, IMHO) submodules. The relation of the parent module to the submodule is typically a dependency, that is best managed through the dependency manager. I am not very sure how your mixed scheme would behave.

I am not saying no yet, but I want to make sure to think about all the implications, and corners that this feature could have, and fully understand the use case that we are trying to solve. Probably, a more "lightweight" feature that was able to repeat a conan command over a set of folders, passed as parameters could have less complexities. What do you think?

[JPGygax68]

You make very valid points. I had indeed thought about consuming packages only, not creating and maintaining them.

I agree with you that submodules (which do have some nightmarish qualitites, though I still find them very helpful) are a different way of handling dependencies, and that this should not overlap with conan; so no, I don't think the proposed submodules feature should have any effect on package creation or upload.

I do like the idea of a "lightweight" feature, that does not clutter up the conanfile with stuff that doesn't belong there.
I'm not so thrilled maybe with the idea of having to specify a folder list, even though that could be put into a script, because scripts are not portable (though Python could be used since conan makes that mandatory).
I think there is something to be said for a "recurse" option, that would simply traverse the directory tree in search of conanfiles and install all the requirements it finds. Though I seem to recall from the docs that requirements can depend on settings, so the recursion would likely be less simple than I just made it sound.

[memsharded]

Ok, great, I will have a detailed thought about a lightweight alternative, recurse option or similar, then. Thanks!

[JPGygax68]

Thanks very much! But please consider this lowest priority. Submodules _are_ the coder's responsibility, not that of the package manager tool.

Maybe a document/article/blog post about the different stages of software building would help more than a new feature: how a language-specific package manager squeezes itself in as a new obtain dependencies step, right before the holy trinity of configure - make - make install, with perhaps a few thoughts on how deal with submodules in that new step.

(Also note that "submodules" are not necessarily _git_ submodules: any kind of nested subproject faces the same question regarding how to obtain its own dependencies.)

[memsharded]

Totally agree! Sure, low priority, and the idea of the post could be good, indeed. Thanks again!

[jgsogo]

I'm marking this issue as obsolete and consider closing it in a few days. Conan approach to developing into several libraries at the same time is the experimental feature called Workspaces: https://docs.conan.io/en/latest/developing_packages/workspaces.html, any feedback about it would be more than welcome.

Thanks for your positive feedback, @JPGygax68, and hope you are using Conan!