Conan: [workspaces] Workspaces should be Super Builds

[kenfred]

These comments were relevant as of version 1.18.

We were early adopters of Workspaces and we continue to rely it for our workflow. Unfortunately, the changes introduced in 1.13, have broken our workflow and we're stuck at a previous version of Conan. I have seen and commented on issues that are components of workspaces and deal with the issues I'm going to discuss (#4840, #4004, #4803, #4751, #5291, #4802, #4800) but I feel like we need an issue to discuss the design as a whole. My main focus is workspaces, but I'll share some opinions on editables as well, since they are closely related. Here is some feedback.

TLDR: Layouts, editables, and the current CMake workspace generator feel off the mark to me. I need more of a super build that retains the features we know and love from Conan.

Layouts

Layout files allow you to manually provide the paths Conan needs to generate conaninfo files. This is motivated by the fact that most packages in development have not had an install step and, therefore, the paths to libs, headers, and bins are somewhere in the build directory. Dependent packages need an alternate package_info to tell them where to find the goodies they need in the build directory.

Unfortunately, layout files are problematic for several reasons:

• Layout files can conflict with reality. Layout files allow you to specify externally what is happening internally, and the two can easily be in conflict. Layout files give the illusion that you can direct Conan to copy the output files to those locations, when actually you're simply telling Conan where you expect those files to end up in the build directory. In other words, layout files are presented as a remapping of output file locations, when actually they force Conan to look at particular location, whether the files are there or not.
• It defeats the convenience of package_info. The ability of a package to fill out its own info is a tremendous power of Conan. It allows dependent packages to automatically pick up paths and other features. This is analogous to modern CMake's use of target_link_libraries, which passes along include directories, library paths, and usage requirements. Using layout files, package_info is overridden with a manual file the consumer has to provide with some internal knowledge of the inner-workings of the package.

Conan does provide a way to specify this layout within package_info. IMHO, this is much preferred, since it is the recipe that knows where the output files live. (See above - the user shouldn't have to assume these locations externally). The issue I have with this is the reliance on the self.in_local_cache switch. The condition should not be whether the recipe is being consumed from the local cache (in fact a recipe shouldn't know about the cache at all). The condition should be if the consumer is consuming from the build directory or the package directory. There is a direct CMake analog for this as well: package layout config files. Look at the following CMakeLists.txt:

cmake add_library(ClimbingStats climbingstats.cpp) target_include_directories(ClimbingStats INTERFACE $<BUILD_INTERFACE:${CMAKE_CURRENT_BINARY_DIR}/generated> $<INSTALL_INTERFACE:relative/path> $<INSTALL_INTERFACE:$<INSTALL_PREFIX>/$<CONFIG>/generated> )

You can see that the include directory is specified for both scenarios: if you're consuming from the build folder or if you're consuming from the install folder. CMake will generate a configuration file for each that contains all of the necessary include paths and usage requirements. (The actual config file generation is not shown in that snippet, but it involves CMake export and install functions).

CMake is right to rely on the package's CMakeLists.txt to define how to consume the package. Similarly, Conan should do the same within the recipe. The ability to do this in package_info is sufficient, but we need a better condition than in_local_cache and a way for workspaces to trigger the in-build condition.

• New syntax for the same mechanism. Given that package_info and the full power of python is already available. Why introduce Jinja templating? Obviously this is necessary if you're going to do layout files in the ini format, but this only reinforces the notion that this logic should be in the recipe itself, not provided externally with new syntax.
• Out-of-source builds are broken. Layout files hardcode paths. There are some Jinja template expressions, but these are quite limited. Right now, the only way to do an out-of-source build is to blindly go up directories in relative path (ie. ../build). Not only is this dangerous (could overwrite an existing directory), it doesn't allow the user control over where the build directory is. This is especially critical when the user wishes to build more than one configuration at the same time with multiple build folders. CMake does things relative to CMAKE_CURRENT_BINARY_DIR or INSTALL_PREFIX and Conan should follow suite.

Editables

Editables allow you to short-circuit Conan to point to some local copy of a recipe instead of fetching it from the local cache. It does this globally for any other recipe that refers to it. As a developer who works on many things at once, this is problematic. It is too big of a hammer to globally override Conan to point to a alternate recipe for any and all dependent packages that reference it. I will happily continue to use workspaces, which explicitly states the intent for co-developed recipes and leaves the others alone. Please consider workspaces as the non-global editable and keep that functionality.

CMake Generator

The CMake Generator creates a cmake script that will add_subdirectory on the source folders (defined in the layout file). The actual generated script isn't very useful.

When you're developing recipes, you rely on conan build to perform non-trivial build steps. It is very useful to be able to reuse the settings and options in the recipe during the build. Even for end consumers, we use conan build as recommended in Mastering Conan. By using the generated cmake script, you lose this ability.

User/Channel

Why has user and channel been added as a requirement to the workspace file? When you are specifying the local path to the package in the workspace file, user and channel (and even version number) are meaningless.

Some Suggestions

I can see workspaces being used in two modes:

Super Build

This is the primary use-case. I have unique packages that I own and want to develop, sometimes together, sometimes individually. I temporarily create a workspace file that indicates my intent to develop them together. When I issue a conan workspace command, instead of generating a script with a bunch of add_subdirectory calls, it would generate a script with custom commands or similar that would invoke conan install and conan build. By doing this, you would gain:

• All of the conan build arguments, settings, options still work
• The packages don't need to be CMake anymore - you've enabled heterogeneous package co-development through the shared language of Conan
• The generated super build (can be a CMake script) can ensure that things rebuild when upstream package source changes
• You could potentially also call conan package as part of the custom command and then all of the special build folder mapping goes away
• Layout files are completely unnecessary. conan build already knows where and how to build the source, package_info already knows how to communicate paths and artifact info.
• Editables are unnecessary. Since workspaces are easier to specify without layouts, it should be preferred to make the non-global choice.

Self-Managed Build

In some cases, the source code of multiple packages are always developed together and their relative source locations are deterministic. This would be the case in a monorepo or git submodule situation. (This happens to be how I am using workspaces). In this case, I have my own CMakeLists.txt that knows where to add_subdirectory and a root conanfile.py that builds the monorepo via conan build. I also have other conanfile.py files scattered throughout the monorepo because packages within the monorepo should declare their own dependencies and remain independent, even if they are source controlled together. Finally, I have checked in a workspace file, because these packages are necessarily developed together. The workspace is critical to building the monorepo.

In this scenario, what I need from Conan is to walk all the dependencies and install all of the third-party packages like it normally would in a conan install. This is something I could do prior to Conan 1.13, although there were a few warts.

I think the Self-Managed Build is really just a special case of the Super Build where I don't bother to use the generated super build script and instead use my own CMakeLists.txt.

To help us debug your issue please explain:

• [x ] I've read the CONTRIBUTING guide.
• [x ] I've specified the Conan version, operating system version and any tool that can be relevant.
• [x ] I've explained the steps to reproduce the error or the motivation/use case of the question/suggestion.

[jgsogo]

🙌 Thanks a lot for your detailed overview and valuable feedback, also for gathering together all the issues related to workspaces.

I agree with you on most of your comments. We know we need to [re]define workspaces again, there are many issues related to them and it is a very valuable feature. Many of the issues are related to layouts, and your intuition about layouts being defined inside the recipe are in the right direction, I would like @memsharded to summarize here the work related to them before entering into details.

[monsdar]

Being able to work on multiple components at once is one of the main requests I get from our developers. Having workspaces "just work" would be awesome.

[kenfred]

@jgsogo Thanks for being receptive to my comments. I'm interested to know what is in store for the redesign.

As it is being redesigned, is there a stopgap that would allow it to work like pre-1.13.2 workspaces? I would like to get current with my conan version, but I need to be able to do workspaces without layout files. Perhaps we could revert back to that? Or enable the old or the current with a configuration option?

[kenfred]

@jgsogo I'd support the cmake generator to do a super build using externalproject_add as suggested in #3034. I saw your response to this in #3302, but I think its worth another look. externalproject_add seems like a great fit for this!

Specifically, I don't think we should mind the fact that externalproject_add uses import targets instead of the real cmake targets. In fact, this is the thing that will allow you to do support other build systems (a heterogeneous build tree) with the cmake workspace generator. It is also more true to how conan will operate when the packages are not joined in a workspace. externalproject_add has its own stamp files and DEPENDS arguments so that it will know to rebuild when one of the dependencies changes.

[petermbauer]

Another problem with the add_subdirectory approach is that one has to use e.g. CMAKE_CURRENT_BINARY_DIR instead of CMAKE_BINARY_DIR everywhere which is IMO not that common and leads to some migration effort.

[mbodmer]

https://github.com/conan-io/conan/issues/3034#issuecomment-477997115

[wpalfi]

Very interesting. There seem to be many quite different ideas and use cases for workspaces. E.g.

• use conan as much as possible, if your architecture is already based on conan (as proposed here)
• avoid using conan as much as possible (as I proposed in #6844)
• lightweight workspaces, proposed in #5566
• ...

The conan team should aim for a generic solution that enables all these use cases by just providing the required information and interfaces without actually creating any targets and without assuming any standard workspace layout:

• list of "editable" packages (together with source folders, etc.) in conaninfo.cmake or conanworkspace.cmake for custom workspace creation
• interface for linking binary packages to editables (e.g. by specifying the cmake-targets in the conanfile)