Conan: Bad developer experience with relation to unit tests integration

[liberforce]

Build systems like autotools, CMake or Meson have a specific step for running tests. However, unit tests look like second class citizens in Conan. Here's a list of problems I found about this.

"test" command is for testing the package

Conan understands "test" as "test the package", but conan is not only a packaging tool, it's also a meta-build system. This is a bit counter-intuitive for the newcommer.

"test" and "build" are different things and should not be run in the same step

Conan documentation gives examples where the tests are run inside the build() method of the package recipe. However, building and testing are semantically separate things.

Here's a specific example of the problems it induces: I have a library that takes a good about of time to build (30min). I called the unit tests in the build() method like advertised. The tests failed on Windows because of an environment problem. This problem fixed, I have the choice between:

• building from scratch again and hope the tests will not fail this time, which makes me lose ½ an hour, maybe more if it fails again
• run conan create with the --keep-build option, because I know I don't need to rebuild because the problem was not with the binaries but the environment. This won't work however, because the build() method won't be called.

With separate steps, I could have fixed the environment problem and just run my tests again.

No, I don't want to test that by hand, because I already did and it worked, and that's my continuous integration I want to test now.

Code duplication for tests setup

In the example above, the environment problem was that the library has dependencies, and on the Windows build, I need to run imports() to have the dependencies available at the time I call the tests. I already had that code, but it was in the test_package recipe, so now I have to duplicate that code, because maintaining a base class for both recipes looks overkill.

Binaries duplication ad nauseum

As I now have to call imports in the main recipe too, this means that when I will build the software, It will copy all the dependencies twice: once for the main recipe, to be able to run unit tests, and a second time in the package test recipe, to be able to test the package. With big projects, that doesn't seem to scale well, as this seems like a lot of wasted resources to "copy, test, then erase". I understand this is partly due to how this is handled on Windows, but maybe having to do the imports twice could be avoidable ?

Using Conan 1.4.1 on Ubuntu 14.04, Ubuntu 16.04 and Windows 2016 Server.

• [X] I've read the CONTRIBUTING guide.
• [X] I've specified the Conan version, operating system version and any tool that can be relevant.
• [X] I've explained the steps to reproduce the error or the motivation/use case of the question/suggestion.

[memsharded]

Conan understands "test" as "test the package", but conan is not only a packaging tool, it's also a meta-build system.

I challenge that assertion. Conan is a packaging tool. Calling the build system does not convert it into a meta-build system. To be a meta-build you have to be able to generate build scripts which conan clearly does not.

test and build are semantically different

Yes, from the build system point of view. For a package manager build means "build" the package, not an actual compilation of source code. Sure you can do whatever you want, from fetching some binaries from the internet, from building binaries from sources. Running unit tests in that build is good too if you want.

When unit tests fail, that is not a problem of the packaging. That should be debugged in a local flow, not creating and recreating packages. You would be typically working in your IDE or with your tools, in user space.

No, I don't want to test that by hand, because I already did and it worked, and that's my continuous integration I want to test now.

The CI integration process will run from scratch again, having it separated provides no utility for that case.

Using imports, and testing from other places is not feasible in the general case, as that would only access the public package interface. But most times, unit tests need access to internal implementation details. So they can only be built and run within the build done by the build system. test_package is not for running unit tests, indeed, they are for checking that the package was correctly created.

I don't know if that would be what you could be looking for, but maybe a conan build --tests (to complete --install, --configure, --build that already exist) is the feature you would like to have? That would make sense indeed.

[liberforce]

I challenge that assertion. Conan is a packaging tool. Calling the build system does not convert it into a meta-build system. To be a meta-build you have to be able to generate build scripts which conan clearly does not.

From https://en.wikipedia.org/wiki/List_of_build_automation_software#Meta-build :

A meta-build tool is capable of building many different projects using a subset of existing buildtools. Since these usually provide a list of packages to build, they are also often called package managers.

If you don't consider conan as a meta-build system, that is the source of my misunderstandings on the semantics used.

When unit tests fail, that is not a problem of the packaging. That should be debugged in a local flow, not creating and recreating packages.

Right, this is not a packaging problem. But in a CI pipeline, this is a step like any other.
Even on a local flow I would call Conan to build the software, as that's what allows me to find the dependencies. Sure, I can leave conan, go to the build directory, do my changes locally and call make, but that sucks. I'm changing source inside a build directory that can be erased and lose my changes. The workflow I'd expect, is to be able to do my changes in the sources, call conan build --keep-build again, have it build incrementaly, run my tests, and generate the package if everything is ok. By writing this I see a problem: as --keep-build implies --keep-source, my changes wouldn't be taken into account. Does this mean there's no way to perform incremental builds by just calling conan create and I should call conan build?

The CI integration process will run from scratch again, having it separated provides no utility for that case.

Not when you want incremental builds.

Using imports, and testing from other places is not feasible in the general case, as that would only access the public package interface. But most times, unit tests need access to internal implementation details. So they can only be built and run within the build done by the build system.

Good point.

I don't know if that would be what you could be looking for, but maybe a conan build --tests (to complete --install, --configure, --build that already exist) is the feature you would like to have?

I wasn't aware of --install nor --configure. Yes, --test (without an 's' to keep consistent with other options that are verbs) is exactly what I'd expect. I would help not having to rebuild the software from scratch to re-run the tests if the problem was just in the environment.

[memsharded]

A meta-build tool is capable of building many different projects using a subset of existing buildtools. Since these usually provide a list of packages to build, they are also often called package managers.

If you don't consider conan as a meta-build system, that is the source of my misunderstandings on the semantics used.

I think that in C++ we have a different semantics for meta-build, and meta-xxx given by 2 important aspects: TMP in which there is code generating more code, and CMake, which all agree is a meta-build system and its main characteristic is that it generates scripts and files for other build systems, but typically not calling such build systems. With these semantics already in place and widely adopted in the C++ community, then I think it doesn't make sense to define Conan, or other package managers as meta-build systems.

Does this mean there's no way to perform incremental builds by just calling conan create and I should call conan build?

Totally. conan create is designed to run a clean build each time. It has to be seen as a pure package creation step, not a development or debugging step. In the same way, you don't develop features in CI. For that, the conan local development flow is the way to go.

I wasn't aware of --install nor --configure. Yes, --test (without an 's' to keep consistent with other options that are verbs) is exactly what I'd expect.

Perfect! You are totally right, that totally makes sense, lets do it!

[memsharded]

Fixed. The conan build --test will be available in conan 1.6.

[cthiebault]

Sorry to wake up old thread like this one but where did you put your unit tests?
Under /src folder ? But if so, they will be published in the package right?