Conan: Dealing with a lot of dependencies in development state

[Phibedy]

I am not sure how to handle dependencies in development state. Changing the version number for each little commit is quite painful if you have 40+ dependencies. As the packages can not depend on the latest version of a package.
It would be great if a package could depend on <foo>/*@<bar>/<wasd>. It would search for the latest version in all remotes. Instead of just *, supporting 1.* would be nice, too.

Another idea would be to force conan to check remotes and if some hash has changed the package gets downloaded even if the current version is the same as the remotes one.
For example:

• pushing package foo/1.0/dev
• conan install foo/1.0/dev -fc installs the package
• conan install foo/1.0/dev -fc would notice that the package is still the same (nothing happens)
• minor changes to package foo/1.0/dev
• pushing package foo/1.0/dev
• conan install foo/1.0/dev -fc overwrites the locally installed package

It would be useful for testing some new algorithms on a robot and the robot could just call conan install .. & ... without the need to change all version numbers each time you changed one line of code.

[monsdar]

This has been discussed in #15, but #101 and #308 could also be of interest to you. Feel free to participate, as they're still open for new ideas and discussion.

[Phibedy]

Thank you, somehow searching for "latest" haven't shown me 15 or I just missed it :)

[memsharded]

There is a feature that is quite undocumented, that is the --update option:

$ conan info --update

This will check for updates, packages with the same version that have been updated. This is a very good solution for development, you can have "develop" version if you want, or "develop" channel for given major versions

$ conan install --update

This will effectively replace local versions for upstream newer ones. So most of the functionality you are commenting is already there, sorry, our fault we failed to document it properly.

Regarding the "latest" version, lets continue the conversation in #15.

[abuszta]

@memsharded Would it be possible to make --update remove packages built using this old recipe? I have situation where --update downloads new version of package from remote right now, but then it uses old binary package (I'm using 0.12):

package/1.0@abusz/testing: Updated!
...
package/1.0@abusz/testing: Already installed!

[memsharded]

Yes, the thing is that the recipe might be updated, but it is not necessary to remove the binaries, they are fine! It is very annoying for the package creator, especially if dealing with very large packages. Imagine this scenario:

• I write a recipe for Boost
• I build 30 different packages for Win, my machine on fire for so many hours.
• I upload them.
• Then I realize, or been reported, that my package_info() is lacking a definition of a imaginary BOOST_MYFLAG, a flag only necessary for consumers. That flag does not affect at all the binaries created.
• I can just fix my recipe and upload it again. The binaries are still fine.

That is why the binaries are not removed by default, they are updated if the upstream is updated. It is the responsibility (with a great power...) of the package creator to manage that.

If you are in doubt and want to make sure, you can always run a conan remove package/1.0@abusz/testing, that will for sure remove the binaries from the local cache.

Does this make sense? Thanks very much for your feedback!

[abuszta]

So maybe consider adding --update-and-remove-binaries-if-updated (whatever name fits)?

...or maybe better idea - add possibility to marking package as "rebuilt-needed" on remote (or even rebuilt-needed but for specific configuration only), so when client does update for this package, he knows that he needs to rebuild, because e.g. someone has broken this package's 1.0 recipe before and now he fixed it. The version stays the same, but it's built now correctly with some different configuration switch etc.

My use case is that I'm working on some project_x and still tweaking dependency recipes (e.g. recipe for package_a/1.0 and package_b/1.0). Let's say that package_a and package_b recipes seem to be working on all platforms. I'm using them fine on OSX for project_x.

I'm triggering CI to build project_x on all other platforms (Windows, Linux, ImaginaryOS...) and I notice that package_a is not working with project_x on Linux because someone have overlooked something in package_a's recipe. I would like to upload fixed recipe to remote and rebuild project_x (which will update and rebuild package which was broken). I don't want to remove and rebuild package_a and package_b every time because it takes e.g. 45 minutes for each of them. I would like to rebuild only updated.

Now I would need to ssh to CI and remove this package manually because it will be never rebuilt.

I propose adding e.g. --build=updated next to --build=missing. What do you think about that? Is it hard to implement?

[Phibedy]

What you want is something like I asked in #500, some package_info that can be processed by a computer (for example json format) would do the job easily.

[memsharded]

Ok, I see.. Thanks @Phibedy for pointing this, yes, I think they are practically the same. Will try to allocate some time to check what could be done. Thanks again!

[kenfred]

I have a requirement to be able to reproduce the build for every changeset, so the proposed options of a moving recipe or a version range will not suit me. Therefore, I'm experimenting with techniques to handle the following:

1. Reference exact changeset hashes of dependencies (similar to how Git Submodule saves hashes) and install via Conan.
2. Ability to co-develop packages. Package A depends on Package B and both need to be developed in parallel.
3. Some way of preventing Package A from committing while there are uncommitted changes in Package B. (Similar to how Git Submodule/Hg Subrepos require the sub repo to be committed before the dependent repo is committed).
4. Some way of automatically tracking the referenced hash in Package A when Package B is committed.
5. Some kind of cross repo dependent pipelines for CI. The CI server needs to pass Package B before passing Package A. Every changeset of every repo must pass its smoke test.

I see Requirements 3 - 5 as beyond the scope of Conan and will require some custom tooling. However, if you have any bright ideas on any of those points, please pass them along.

The immediate needs are requirements 1 and 2 and I would expect Conan to have ways to accomplish them.

For 1, I'm experimenting with a multi-recipe technique. One "normal" recipe is stored inside the repo, but is not uploaded to the Conan server (or only occasionally for releases). On the server is a special recipe that accepts the changeset hash, tag or revision string as an option, which is used to clone the repo, checkout a particular change, and export the "normal" recipe to the local store.

from conans import ConanFile

class FooConan(ConanFile):
    name = "Foo"
    version = "revision"
    settings = "os", "compiler", "build_type", "arch"
    options = {"revision": "ANY"}
    default_options = "revision=tip"
    description = "Revision Tracking Package for Foo"
    url = "<url>/foo"
    license = "MIT"

    def source(self):
        self.run("hg clone <url>/foo") # clone the repo

    def build(self):
        self.run("hg update -Rfoo -r%s" % self.options.revision) # checkout at hash
        self.run("conan export foo <uname>/%s" % self.options.revision) # export the recipe found inside

This will clone the repo to the source folder. Then for each revision option, a unique build folder copies the repo and checks out the repo at the desired version. It then exports the "normal" recipe found within the repo at that changeset. It writes to a unique channel because there may be multiple installations of the same version, potentially one for every changeset. I'd rather change the version so the installed package is foo/<hash>@kfred/development, but not yet sure how. (Note, the local store may be littered with these "changeset packages", but I don't intend to upload them to the server. They can be pruned occasionally).

The consumer would have something like:

[requires]
Foo/revision@kfred/development

[options]
Foo:revision=<hash>

Obviously I'm abusing the recipe system here. Is there a better way to do this?

For 2, I'm not quite sure what the best way to go is. One thought is to develop right in the build folder of the changeset package. That's a problem if multiple consumers happen to point to the same changeset package. Alternatively, I clone a separate copy of the repo and change the generated conaninfo file to point to my co-developed repo. Co-development is common enough of a use-case that Conan may have a better way to do this. Any suggestions?

Without these, I'll have to resort to a hybrid approach of submodules and conan. However, for various reasons, I want to avoid submodules.

The remaining requirements will have to be accomplished with git/hg hooks and some custom Conan integration.

Thanks for your attention!
Ken

[lasote]

Hi Ken,

I have no magic solution for your workflow. Maybe the conan alias could be interesting at some point.

About the 2). Yes, co-development of conan packages is one of our priorities, very demanded lately. We are working on some ideas, but nothing released yet (it will take some time). The general idea is to indicate conan somehow that a dependency is being edited, so the generated files (like conanbuildinfo.cmake) can point to a local directory.

[kenfred]

Thanks lasote for your response and your efforts.

As you kick around solutions for co-development of packages, please consider that we'll need ways to checkout the exact hash of the dependency repo (item 1 of my list), which could be on a feature branch of a repo and not necessarily the latest change on the master branch.

If you buy the above, then it would be convenient to have some sort of automatic updating of the hash in the recipe when I commit the dependency repo (item 4). That keeps me from having to manually inspect the git hash and updating my conanfile. Otherwise there is a danger that I'll commit the repo depending on the previous dependency hash.

If the tool has the smarts to be aware of the dependency repo, we could create a git hook to to prevent committing if there are changes to the dependency repo (item 3).

These convenience features are what makes Git Submodules powerful. You may ask, "Why are you reinventing Git Submodules?" There are a number of issues with submodules that Conan naturally alleviates:

• Submodules result in redundant copies of repos in a dependency tree. This duplication of repos creates cloning performance issues and can be confusing for developers in knowing which version of the submodule to build and co-develop. Indefinite nesting can also easily exceed path character limits. The recommended way to avoid indefinite nesting is using a "shell" repo as the parent and disallow code repos from having submodule themselves. This super repo concept presents obstacles to certain workflows, particularly in a continuous-integration environment where each submodule needs to be independently verified with automated builds. Conan brings a much needed flattened list with conflict resolution.
• Conan can reason with the dependency graph beforehand because the dependency metadata is available from the server. With submodules, you progressively learn the dependency tree as you recursively clone. There are some opportunities lost here, particularly from the CI server.
• Submodules break the distributed nature of git. Submodule URLs are committed to the parent repo, so although you can pass around the parent repo, the subrepos are forever centralized. If the subrepos migrate to a new URL, your git history is broken. With Conan, you don't have to commit the URL of dependencies to the repo. The URL is stored in the recipe on the Conan server or the source code is snapshotted in the recipe itself and no URL is required. If a migration happens, the recipe can be updated transparently to your old git changes. As long as you have some Conan server (local or remote) that can provide the dependency, you can safely checkout any git changeset and build away.
• Conan is SCM agnostic. Not all repos are git.
• Consistent metadata - if I'm using Conan already, I'd prefer a consistent way to track dependencies.

One suggestion/request: if your solution to (2) involves cloning the dependency repo locally, don't place it in the output directory (i.e. CMAKE_CURRENT_BINARY_DIR or where you run conan install). It is common to nuke the entire directory to force a clean build and it would be very easy to accidentally nuke the changes to the dependency repo. Rather, I suggest putting it in a flat directory peer to the conanfile (similar to npm/yarn node_modules) which can be added to .gitignore.

I'll continue to monitor this issue. Is there another issue number tracking this development I can watch?

Thanks,
Ken

[marco-m]

Hello @lasote, is this issue the best one to track progress for the "co-development" use case you are mentioning? If not, I am happy to open one :-)

[lasote]

I've created the development_flow label. The co-development feature is not a unique issue, I think the best way is to check the tag and stay tuned to coming pull requests related.

[marco-m]

Thanks!

[memsharded]

This issue is the same as https://github.com/conan-io/conan/issues/1377, so I think this can be closed, and continue conversation there.