Conan: Challenges with new commands

[solvingj]

Here are a few nuisances I ran into when trying to run through the new workflow from scratch. They are based on a subjective assumption that the workflow for the new commands would emulate the conan create workflow, just in a step-by-step way. Under this assumption, the below seemed to violate the principle of least-astonishment, in addition to forcing new, manual, and undesirable steps to get the expected and desired behavior.

1. conan source . --source-folder=tmp/source doesn't clean the tmp/source directory

Cleaning source directory was an implied behavior in the conan create workflow. When using conan create, after we got the source method right, we could use -k --keep-source and essentially move on to testing the build method. With the new commands, the entire purpose of the conan source command is to verify through trial-and-error that the source() method gets the correct files. So, forcing the user to manually delete the contents to re-run the conan source command for the common case is extremely inconvenient. Also, cleaning it automatically would is more consistent with the perceived behavior of the source() method in a conan create execution.

1. conan build .... does not copy contents of --source-folder to --build-folder

So, after getting the conan source command working as described in item 1 above, users are intended to move on to the conan build command. All existing recipes I know of have build() methods which assume that the sources have been copied to the build directory, thus do not work with the conan build command as-is. Because of this, packagers must do one of two undesirable things to use the new workflow with existing packages.

1. Manually copy the contents between testing of the conan source and conan build commands.
2. Pass the directory which we used in conan source as both the --source-folder and --build-folder arguments of conan build.

The first is undesirable because it's manual, the second is undesirable for the same reason conan create doesn't perform builds in the source directory. We're going to run the build method repeatedly while testing, and we always want to start with a clean build directory, and pull in clean sources at the start.

Copying the contents should be the default for at least two reasons.

1. It is consistent with how conan create works
2. It is more practical than current behavior for trial-and-error troubleshooting of the build() method.

If you think copying the contents is unnecessary and inefficient, and could be skipped in many cases, I don't disagree. However, the recipe would need to have been written with that in mind. Thus, a new option for conan build such as --no_copy_source would be a reasonable addition, and consistent with conan create. This would allow recipes which have no_copy_source=True to skip the copy.

This is all i have so far, will continue testing the workflow from here (which is currently wrong as it doesn't work as explained):
https://bincrafters.github.io/2017/11/10/Updated-Conan-Package-Flow/

[lasote]

Thanks for the feedback!
About the 1) totally true, is a pain to remove the folder, probably we should remove it by default, but it can be dangerous if you pass the wrong --source_folder. A confirm parameter?
About the 2) Probably you are right too, We thought that copying the source to build folder will couple the process, because it's just a simple copy that can be done manually, and this way is more flexible. The --no_copy_source is a good point, but probably a parameter to the conan build command to avoid the copy could keep the "decoupled" mechanism irrespective of the recipe. WDYT @memsharded ?

[solvingj]

1. I definitely see the concern. So, rather than a confirm parameter, might as well leave the default as-is and add --delete, --clean-source, or whatever. Same safety, more explicit.

2. I see, yes it's a bit awkward to have the behavior dictated by the command, when conanfile.py has an explicit no_copy_source attribute. Questions come up like: "Can the user override the recipe attribute from CLI?", "Does it override by default, or would we need to add some --override param, and how ugly is that?" I could see multiple behaviors making sense, which makes for tough choices. This one will take some thought, but I have faith you'll find a nice combination.

3. Whatever you decide, I feel that conan build should feature the same auto-cleanup policies/arguments as conan source.

One thing I can tell you for sure, the idempotency of conan create felt like a brilliant and critical behavior. It cleans up source and build directories, and the defaults are right for nearly 100% of our OSS use cases. It feels perfect. Currently, without the automatic cleanup and copies in the new commands, it just feels like a step backward (for the sake of flexibility). the way it stands, I will continue to use conan create ... -k for convenience, even just to re-test the package method unless the build is super long. I can type very little, not worry about any housekeeping, and alt-tab until it's done.

[sztomi]

the workflow for the new commands would emulate the conan create workflow, just in a step-by-step way

If this is true (and this was my perception as well) then conan create should be implemented in terms of calling each "small" command (and doing performing no additional steps). If it's not implemented that way then breakage is possible in the future even if current problems are fixed.

[memsharded]

I think there is a huge inconvenient here, and is the main reason why I think a local copy of the sources has to be avoided at all costs:

• The source code is obtained always from source-control. EIther with a exports_sources, the code being exported is in the same repository than the recipe, or with the source() method, that typically runs a fetch from some version control system
• Local development flow is exactly for that: for development. We expect the users modifying the source code, building, testing, and modifying the source code again.
• If a copy is made for building, as the conan-cache does, then the build system, and the build will be always executing from the copy. The IDEs will edit the copied sources.
• All the modifications won't reflect in source-control, and will be overwritten when the process is fired again.

So local development flow is in this sense wildly different, and in this case, I think building the source code that is under version control and the user is modifying is a high priority. In the conan-cache, the criteria is different, is basically the opposite: making sure the process is repeatable and independent of possible changes. Build processes that affect the source code (like code generators, or writing a config.h) will ruin the common source code base if it is going to be compiled for different configurations, that is why a clean copy of the source code for building is the safe default.

[memsharded]

Regarding issue number one:

conan source . --source-folder=tmp/source doesn't clean the tmp/source directory

Yes, I understand this one, and I think it is fine to change it to match that behavior

I think that the underlying issue in the above approach is that we have different views on what conan build is for. I think it is mainly for developers wanting to work on their libraries, changing the source code of the library and building and rebuilding it with their IDEs. Your feedback suggest you are on the other side, and concerned only about packaging, but not for code modifications. So the trade-off can be more clearly formulated:

• If conan build makes a code copy, it emulates better what is done in the conan cache, but the developers working on their code will have a worse experience. Failures here have the result of developers losing code.
• If conan build does not make a code copy, it is the opposite. Failures here while creating packages are very mitigated by the fact that final packages will be created in local cache (or in CI) with conan create

As conan build shouldn't be a substitute for proper package creation with conan create and in the local cache, IMHO its main purpose is to make easier developers (changing code) life. In turn, it is also useful for package creators, even if not exactly the same as conan cache, because it is a tool for debugging, but the real packaging is done later. But the code-copy approach basically disables any natural flow for developers editing their code.

[solvingj]

Ok, then perhaps we just misunderstood the intentions of the commands, and we are using a square peg for a round hole. On the surface, the commands appear provide a general-use mechanism for stepping through a recipe method-by-method (and they did in the past), but it sounds like they are now intended (and tailored) for the specific flow you described (the conanfile.py being "in-source", and export_sources being used). Indeed, this workflow is distinct from our normal "third-party OSS" flow.

We had found a successful workflow using the old versions of the commands, so our suggestions were to add additional flags to approximate that. However, since those seem to be disagreeable, I had thought of an alternative which might be better for everyone anyway.

Generally speaking, the "third-party OSS" workflow is characterized by trial-and-error on the three package methods (and their corresponding folders in the cache):

1. source()
2. build()
3. package()

For this workflow, the conan create is actually almost perfect the way it is. Currently, we have the --keep-source option which everyone likes and understands. It provides a logical and efficient mechanism for us to go from step 1 to step 2 (safely skipping the source() method after we're done with it). The only thing we lack today is a concise way to run the package() and test() methods without the build() method. Thus, please consider simply expanding on this mechanism for step 2 and step 3. The implementations might raise some valid questions, but from a UX point of view this approach definitely has merit (and might be the most straightforward). Specifically, the suggestion is for the conan create command to have the following three related flags (the first already existing):

• --keep-source
• --keep-build
• --keep-package

--keep-build implies --keep-source
--keep-package implies both --keep-build and --keep-source

Note that while the flag names indicate keeping the respective directories on-disk, their behavior would/should be consistent with that of -keep-source today. I.E. keep-source skips the source() method completely, and so keep-build would skip the build() method completely, etc. The --keep-package method might seem to be of dubious purpose, but in fact it would provide a precise mechanism for troubleshooting only the package_info() method and test_package together.

This approach has some advantages over modifying the other commands as we originally requested. However, either approach would let us have a streamlined workflow for "third-party OSS" packaging which we are lacking at the moment. If you have another approach that is equally streamlined, of course it would be welcomed.

[memsharded]

I think this approach might make more sense, at least from the UI seems more consistent. Haven't checked implementations details yet, which might be tricky. Wdyt @lasote?

If there are implications, maybe a --keep={source, build, package} makes more sense.

[uilianries]

I agree with @solvingj

Add --keep-xxx won't inflict as a break change. This approach will help to keep the same commands, just adding new behaviors.

But this individuals steps are not only complicated for us ...
Some weeks ago I presented Conan in a local meetup. I used conan create as the "Holy grail" of C++ package manager. However when a guy asked how to build step-by-step, I just said "it's more complicated".
As we know, the commands are very explicit, each command needs specific arguments and, in many cases, I need to read the documents to understand.

I think current state can be a barrier for beginners as well. Thinking about UX, I've heard from some co-workers that Conan is not friendly when something goes wrong. In fact, many times we create new recipes based on trial-and-error, as @solvingj mentioned. It's because we almost need to re-run all steps using create -- create is a really great command, it has no problems.

IHMO, provide a easy way to execute each step of package creation would be great for many developers, including newbies.
So, regarding that presentation, It would be great if I had the opportunity to answer that guy with "very easy, just run three simple commands".

[memsharded]

However when a guy asked how to build step-by-step, I just said "it's more complicated".

But it is important to know what is exactly the pain, and what we are trying to solve. With the proposed --keep-xxx functionality, we still don't get the build package step-by-step. It is actually the opposite, you cannot just call the source(), then the build(), then the package(), you are basically resuming from the point you consider it worked fine last run.

If we are talking about the local flow, actually I don't think it is that bad, depending on the approach could be something like this:

$ git clone ... project && cd project
$ mkdir build && cd build
$ conan install ..
$ conan build ..
$ conan export-pkg .. Pkg/version@user/channel

Then it is important to note that this will not be an exact reproduction of what is happening in the local cache, mainly because of the above explained reason of code duplicates for code developers (not only package creators).

So you think the --keep-xxx is what those guys and your colleagues are asking for? Or more the local flow?

[uilianries]

I think that --keep-xxx is the answer in such case.

I see --keep-xxx as a good solution when I want to create a project that includes a recipe for some remote project (i.e. zlib, OpenSSL), and as you said, resuming from the point that I consider it worked fine on last run.

In the case of my co-workers, we use recipe and sources in the same project. But, many of our projects were migrated from svn, using the old externals style (a general folder with all the binary that you need). I helped a lot by creating recipes and explaining how Conan works. However, when someone tries to create a recipe from scratch, they spend some time trying to resolve the errors generated during the package flow. They use conan create as general command, so if they have some problem during package or package_info stage, --keep-build could help to save time.

On the other hand, newbies are not familiarized with the tool, so they expect something automagic. In this case I think both answers. I could say, "use conan create, it's like magic!" but to run step-by-step, you need to:

$ conan export foo/testing
$ conan install bar/0.1.0@foo/testing --build=bar
$ conan test test_package bar/0.1.0@foo/testing

or present your solution (good solution),
But our examples need to use command + arguments, more information to decorate.

or even say, "you can execute all steps using conan create, or step-by-step, but if you need to skip some step to fix some point, you have --keep-xxx". So now, I could be focused to use create as main solution, and discover other commands over time.

[solvingj]

Agree with the consolidated flag.

@uilianries makes several great points. conan create is so convenient that it sets the bar for ease-of-use very high. So much so, we should pause for a moment to celebrate such a brilliantly designed feature.

The "magic" in the conan create command comes by virtue of it being "Idempotent". So, regardless of previous runs you know you get a clean start, and it will run to the end (or ctrl+c), and you know where everything goes every time. Users never have to manually create, clean, or delete directories. This is a HUGE deal.

Also, for the record, I personally liked several aspects of the new commands which I talked about in the blog post:

1. Consistent syntax
2. Direct alignment with conanfile.py methods
3. Precise testing for specific methods
4. Doing trial-and-error work outside conan cache

But, in practice these advantages come at the cost of bringing back several of the old manual steps to which conan create eliminated. I.E. creating and managing temp directories, manually copying files, user error, etc. conan create with these new flags simply offers a superior UX: more efficient, simpler to understand.

As for the team, I know that several of our most experienced packagers have always just taken the simpler path and used conan create for everything, because it is mentally easier than checking the syntax of conan build, conan package and conan test. Even though it means conan needlessly re-builds a library multiple times, they don't care because they'll alt-tab to something else for 30 seconds. However, I do know these people use and love --keep-source after they learn about it, and that's how I know they'll use the new flags.

[memsharded]

--keep-build has been implemented, will be in 1.1

[uilianries]

This is really great! Thanks!

[solvingj]

This is super nice, thank you!!

[memsharded]

This is ready for release in 1.1

[solvingj]

What are the plans on --keep-package (intended simply to run the test_package on the package in the local cache).

[solvingj]

I guess that in this case, it's reasonable to break away from the conan create method and use conan test test_package <package_ref>.

[memsharded]

But you have conan test for that use case