Conan: [Question] What is the best way to handle multiple feature branches for multiple packages with Conan?

[Aalmann]

Hi @lasote and @memsharded,

We have a big discussion within the team what is the best solution to handle feature branches for packages which depends on each other. We hope that you can suggest the best way to handle it with Conan.

Starting point

Let's take your "say"-"hello"-"chat" example from https://github.com/conan-io/examples.git workspace feature. "chat" -requires-> "hello" -requires-> "say".

developer task

The implementation has to be refactored for all packages (e.g. in your example the name of the chatting person has to be shown). This will be done in feature branches.

infrastructure

Not sure, but I think it doesn't matter if it is a single mono-repo or three different repos. Let's say there are three git repos.
Additionally there are three CI jobs which depends on each other and which creates each package.
Job_say --> creates "say" --> triggers Job_hello
Job_hello --> creates "hello" --> triggers Job_chat
Job_chat --> creates "chat"

Probably not the best solution to have that dependency defined in CI but that is not necessarily important. The only important thing is, that there are three jobs, one for each package to be more flexible.

Requirements

Before merging the feature branches into develop branch the features have to be completely tested in combination. Just unit testing "say" is not enough for merge, also "hello" and "chat" have to be build and tested completely.

There has to be a reproducible and safe way to define that special "feature branch" graph for build and testing.
CI and development workflow have to be the same (otherwise one will get something like "locally it works, don't now why CI fails" ;)

Current approaches

Approach 1: Adding branch information to "version" or "channel"

For develop and release builds the "normal" version and channel (testing or stable) will be used.
The CI get's the branch information from current scm, checks branch name and calls conan create <scm_dir> user/testing_feature_xyz for "say". This will create "say/0.1@user/testing_feature_xyz". Having the branch information within the "version" is nearly the same approach, so further description is just for channel.

After upload to server there will be a recipe and all packages (build with different settings) with the new reference.

Problems

"hello" has to know the custom channel when building the feature branch. Could be done by "hard-coding" it into the "hello" recipe, but that seems not to be the best solution, especially for merging into develop branch. And "chat" has to know the custom channel of "hello" too.

...
    name = "hello"
    requires = "say/0.1.@user/testing_feature_1"
...

and

...
    name = "chat"
    requires = "hello/0.1.@user/testing_feature_1"
...

Patching the scm versioned recipes locally may IMHO not be a solution (e.g. with "find&replace") for CI. But it would be feasible with a lot of scripting.
For development workflow it could work but is very error prone (create "say" with custom channel --> patch "hello" recipe --> create "hello" with custom channel --> patch "chat" recipe --> create "chat" with custom channel).

Approach 2: Like approach 1 but with additional Alias package

See description of approach 1. Additionally an alias package would be created which points to the required feature_branch package.

Question 1: How should the alias package be named especially when there are more than just one feature branch plus develop branch which points to <package>/<version>@<user>/testing?
Question 2: Which job should create the alias and should it be uploaded too?
Question 3: How to get rid of old packages/cleanup? Having a weekly cron job which checks the current existing branches?

Problems

Assuming alias package points to "normal" testing channel which is referenced in recipe <package>/<version>@<user>/testing there will be naming clashes / differences between local alias and uploaded "develop branch" package. Also error prone for development workflow, conaninfo.txt may contain different information and not sure about recipe hash. And last but not least, it sounds false.

Assuming alias package is something like <package>/<version>@user/testing_alias or something else, one will get the same problems as described in approach 1.

Approach 3: define an "auto-set"-option for scm_reference within the recipe

Adding an "scm_reference" option in the following (or probably better) way:

    name = "say" # same for "hello" and "chat"
    options = { "scm_reference": "ANY"}
    default_options = "scm_reference=develop" # should point to an always existing branch (develop)
    default_channel = "testing" # required for dev mode an do avoid Conan exception when calling self.channel

    def configure(self):
        if not self.channel == "testing":
            # another channel is used so we delete scm_reference
            del self.options.scm_reference
        elif self.develop:
            # channel is testing and recipe is in development or create mode
            from conans.tools import Git
            try:
                # try to get the branch name (fails with "conan create <path_to_conan_file> user/testing")
                self.options.scm_reference = Git().get_branch()
            except:
                try:
                    # fallback if no branch is checked out. Use the commit (fails with "conan create <path_to_conan_file> user/testing")
                    self.options.scm_reference = Git().get_commit()
                except:
                    self.options.scm_reference = tools.get_env("BRNACH_NAME", "develop") # get BRNACH_NAME from CI
        else:
            # channel testing used and recipe is consumed
            pass

This would end in one recipe and conan reference which can address multiple feature branch packages for testing channel. The recipes do not have to be changed locally (neither CI nor development workflow) and it seems to be more the Conan way of defining "variants" of packages.

Problems

If there is a recipe change in a feature branch, all other packages will be outdated.
Changing / updating requirements in a branch will end in undefined behavior (assuming the outdated packages could be downloaded, [not sure if that works with conan]).

Approach 4: Something like Conan package/recipe revisions

Looking at the documentation at https://docs.conan.io/en/latest/mastering/revisions.html?highlight=revision it might not fit to the described use case and the requirements because if I got it right the full reference like "lib/1.0@conan/stable#RREV:PACKAGE_ID" has to be given in the recipe.

Question

What do you think is the best way to handle that?

Thanks.
Best Aalmann

Edit: We use the scm-auto feature which will always outdate the recipe because of different revisions.

[jgsogo]

Hi!

I'm sorry, I don't have the answer 😞 , I just want to purpose another approach:

Approach 5: One remote per feature branch

Without changing the references of the packages or requirements, you can configure Conan in your CI with different remotes depending on the branch you are running. The feature feature_1 could create a repository in Artifactory and then, CI running branches feature_1 will add that remote to Conan (first place and maybe remove testing/stable remotes) and consume/upload packages from/to that one. Of course, you will need to propagate the remotes information from one job to another, but I think it would be easier/safer than editing the recipes themselves or introducing an option to handle this.

This approach needs a lot more thinking, it may have many corner cases. Let's see what others suggest.

[Aalmann]

Hi @jgsogo ,

what a pity that there is no good solution available.
Concerning your approach: We already have two repos. One for testing and one for release (and one additional for third_party stuff). What you describe sounds a bit like an additional staging repo!?! Yes, could work, but there are some issues with that approach:

• The remote has to be always the first in the list (as you already described). Feasible, but then you will have at least two remotes with the same references (staging and testing). First one in staging is used by Conan, but this has to be ensured for every future Conan version. Not sure if you can guarantee this at all?!
• Because of other dependencies in testing-, release- or third_party-repo, the remotes can't be removed.
• The staging repo has to be cleaned after merge. Otherwise one could probably get other "staged" references from this remote which might not be the expected ones. Let's assume that there is also a requirement from "chat" to "goodbye" which is not part of refactoring and which has to come from testing repo.

Probably there are some more issues with that approach.

But thanks a lot for your reply.

[jgsogo]

No solution yet, or not a solution that comes out of my mind right now, but this is something we want to address, so all the feedback related to this is so important.

• The remote has to be always the first in the list (as you already described). Feasible, but then you will have at least two remotes with the same references (staging and testing). First one in staging is used by Conan, but this has to be ensured for every future Conan version. Not sure if you can guarantee this at all?!

Yes, the iteration order of the remotes is stable, so this shouldn't be an issue. A new reference will be retrieved from the first remote that contains that reference. If your _staging_ repo is the first in the list, it will be the one used.

• The staging repo has to be cleaned after merge. Otherwise one could probably get other "staged" references from this remote which might not be the expected ones. Let's assume that there is also a requirement from "chat" to "goodbye" which is not part of refactoring and which has to come from testing repo.

I'm suggesting one _staging_ repo per feature, and this staging repo will only be added to Conan in those CI jobs related to the feature. After merging the feature, this staging repo won't be used anymore, so it can be removed or not.

Even more, there should be a new staging repo for each feature (for each branch) within each library. So, for a new branch featA, one developer will make a PR to say repo, it will trigger CI as you described in your first post (I'm adding a couple of stages):

• Create staging repo say-featA
• Job_say (add remote say-featA) --> creates "say" (uploads to say-featA) --> triggers Job_hello
• Job_hello (add remote say-featA, will retrieve say from it) --> creates "hello" (uploads to say-featA) --> triggers Job_chat
• Job_chat (add remote say-featA, will get say and hello from it)
• (if successful, if merged,...) Remove staging repo say-featA

The staging repo needs the name of the library too because the same featA could be a branch in another library, someone could make a PR to hello and it should trigger CI following these steps:

• Create stagin repo `hello-featA``
  
  
  
  • Job_hello (add remote hello-featA, say will come from _testing_) --> creates "hello" (uploads to hello-featA) --> triggers Job_chat
  
  
  • Job_chat (add remote hello-featA, will get say from _testing_ and hello from this staging repo)
  
  
  • (if successful, if merged,...) Remove staging repo hello-featA
  
  

I hope not to seem too vehement, I just wanted to expose this approach in as much detail as possible. The perfect solution will come from teamwork 💪

[Aalmann]

Ok got your approach.
Because of high infrastructure interaction (create and delete remote) this should be a conan and artifactory/ bintray feature.

Something like conan remote add testing https://url and conan upload <reference> -r testing -push-stage=say_feature_1 and conan move <reference> -r testing -pop-stage done in the last job.

Just as an idea for further discussions.

[Aalmann]

Have a look at nexus concept for maven. https://help.sonatype.com/repomanager2/staging-releases/managing-staging-repositories

It should be quite similar.