Conan: [scm] decoupled "auto" deductions

[Adnn]

After successfully introducing Conan for a first project in our organization, we start to gather some ideas and suggestions.

One important aspect that allowed us to automate steps in our gitflow based approach is the SCM feature, and more specifically its ability to auto the revision (and repo origin, even though we don't use that aspect atm).

Yet, we could see some drawbacks to the current approach:

• Put all the pressure for implementing new scms (eg. SVN) / features on the Conan contributors. An example of feature that was missing in 1.5 was the ability to update git submodules (that we had to hack in build, where it does not really belong)
• Seems to create a second code path: SCM does not seem to be called in the source() method of a parent class, so it is hard to customise / override (for example on a conditional option).

Here is our suggestion:
Instead (or in addition) of having a very specialized/not customisable SCM feature, isolate the automatic deduction features to a separate conanfile attribute. For example, export_deduced:

 class FooConan(ConanFile):
     name = "foo"
     ...
     export_deduced = {
         "revision": auto,
         "repository_url": auto,
         ...
     }

    def source(self):
        self.run("git clone {} clone_folder".format(self.export_deduced.repository_url))
        self.run("cd clone_repo && ... ") # Whatever specialized action the conanfile implementer needs to do, that `SCM` feature did not account for

The advantages we could see for this approach:

• The logic remains very similar to what exist for SCM: the commited recipe contains the auto keyword, and the value is hardcoded in place when the reciped is exported
• This has potential to be extended to other _categories_ of values, beyond SCM scope
• The conanfile implementers keeps control over the source step, and is responsible for implementing the details for his specific environment (eg. submodules) instead of having to wait/put pressure on Conan contributors ; )

[Adnn]

Thank you @lasote for reviewing this suggestion !

[jgsogo]

Maybe I haven't fully understood your comments and request, but let me answer you and we can start talking from here. Thanks for your feedback!

(talking about the auto functionality ⬇️ )

SCM objective is to capture the recipe _snapshot_ in order to be able to reproduce the build. This logic implies that SCM resolution has to be a consequence and not a cause of the recipe sources being compiled. It also applies to the _local_ repository (the one where the recipe is), so we already have the repo, no need to clone (it will be the same URL) neither to check out a revision (it will be the same currently checked out).

(without the auto keyword ⬇️ )

SCM needs to know explicitly what to clone and in this case it will work as the source method.

---

btw, SVN was released in v1.8, we need more feedback around it so we would like to hear from you if you test it.

[Adnn]

Thank you for giving time to this suggestion.

It seems we see eye-to-eye with this statement:

SCM objective is to capture the recipe snapshot in order to be able to reproduce the build

This is exactly why our organization started using the scm feature: we wanted to be able to export recipes for each commit on develop branch, with recipe versioned on the short commit hash (e.g. project/abc123d@com/develop). The scm feature is making sure to write the actual commit hash within the exported recipe. (saving us to write a script to do it ourselves ; )

---

Yet, it is our opinion that the current implementation is actually coupling two different features:

1. Deducing auto values, and replacing them with the actual literal values in the recipe, at export time
2. Making sure tu use the scm structure to source the repository, at install time (thus, using the values deduced by the first feature).

It is also our opinion that it would be beneficial to both Conan developers and Conan users to provide both features separately.

Basically, currently scm can deduce the repo origin  and current commit's hash. Now, we would like to use these values in our recipe, but without using the implicit sourcing feature (2.), and it seems we are not able to do that.

Also, maybe the deduction mechanism (1.) can later be generalized to other kind of values, values that will not necessarily participate in getting the sources.

So the "features" we are suggesting is really to separate mechanism 1.
Somehow allowing a data member like export_deduced in the initial post. This data member would be a dictionary whose auto value are replace with literal deduced at export time. Then, those values could be used by custom recipe code in the way that the users see fit.

[jgsogo]

Thanks for the detailed explanation. As I'm not aware of all the collateral implications it may have, let me mention @lasote and @memsharded so we can get more valuable insights about this.

I'm also moving this to milestone v1.10, as we are trying to close v1.9 as soon as possible and this is something that will require further agreement.

[niosHD]

FWIW, I also think that having a self-sufficient feature, independent from scm, for performing generic replacements within the recipe is very valuable. In one of my projects I currently have to generate a conanfile.py from a template to achieve the same functionality. However, this is quite messy and requires yet another python script. Given such a feature, the same functionality could directly be implemented within the recipe.

For reference, at the end of the initial post in #3183 I actually also proposed something along this line. Additionally, I prototyped the functionality (named conan_expand) in #3255 some time ago but it seems this approach was not the correct one. Still, I am definitely interested in what you guys come up with.

[lasote]

We've been thinking about this and probably what you are looking for is an export() method where you could use the Git() or SVN() to extract the data. You could write a file with the data, export it and use it however you wanted in the source() method.
Related to #2456

[lasote]

I've been playing again with this and you can access perfectly to the scm attribute even from a parent class. You the only thing you would need is to avoid Conan to do the automatic checkout of the code? I think I missed the point of the issue. Even for implementing new SCM types you would need to know how to extract the data, so I think the only way to really achieve that is to use an export method.

[Adnn]

Hi @lasote ! And thank you for investigation that aspect.

I think giving the recipe implementers access to an export method, where they could "record" any information at export time into the recipe is a very good idea.
The point of the issue was to make the auto deduction mechanisme more generic in the way they can be used (avoiding the automatic checkout, and using exported values in other places of the code), as well as more easily extensible to record other kind of values. I think the proposed approach would solve both!

[lasote]

I think this feature #4088 from @jgsogo could, not solve, but help with this. Please take a look, you can write a tiny function to "resolve" the URL and commit, so you could implement your custom login and reuse it. e.g:

import os
from conans import ConanFile
from conans.client.tools.scm import Git
def get_revision():
    here = os.path.dirname(__file__)
    git = Git(here)
    return git.get_commit()

def get_url():
    here = os.path.dirname(__file__)
    git = Git(here)
    return git.get_remote_url()

class MyLib(ConanFile):
    name = "issue"
    version = "3831"
    scm = {'type': 'git', 'url': get_url(), 'revision': get_revision()}