Conan: [SCM][revisions] Supporting the `scm revsion` as package `version`

[niosHD]

Disclaimer: I discussed this issue on Slack and post it now here to keep track of it. Thank you @lasote for the great discussion!

Problem

I currently have several recipes which "manually" capture the git revision based on James' idea when exporting/creating the package. Given that conan now has scm support I want to get rid of my custom code and use the upstream feature instead.

However, the way I use my scm revision is apparently not supported by the current scm implementation. Namely, when building packages from the development branch, I directly use the scm revision as package version. See https://gist.github.com/niosHD/dc669dcb6c8d92b51e3d9c6bab774e90 for a sketch of one of my conanfiles.

Background

My reasoning on why I want to use the scm revision as version in non-release packages is as follows. For me a version should uniquely identify a package. However, version numbers are only assigned when a release is made. Subsequently, non-release builds (e.g., CI, builds on the developer machine of a certain feature branch, ...) typically have no real version.

Sill, as conan really needs a version, some kind of version has to be generated. This can either be done by generating a new unique version (e.g., something like a build number) or by using some arbitrary fixed string (e.g., branch name, the next expected version as working title, ...) which results in continuously replacing the package. (Note that package revisions will dramatically improve the usefulness/reproducibility of the second option.)

We opted for the first approach since it enables us to also package/test/distribute non-release versions in a reproducible way via conan. However, instead of generating an arbitrary build number, which again does not directly convey what has been packaged, we use the scm revision which states exactly what is in the respective package.

Given this approach, we basically then use two types of dependency management. The first one is similar to a git submodule approach where a very specific (scm) version is required by a package. An example for such a dependency is clang's dependency on llvm where both repositories are typically always incremented in lockstep. Using the scm version as package version enables us to easily manage these kind of dependencies. Of course CI builds are used to verify that we get working results.

The second dependency type is more dynamic. For some libraries, where the API is more or less stable, we simple want to depend on the latest sucessfully CI tested version of this library (in a certain git branch). There we use conan's package alias, which gets updated by our CI server after each successful build, with the branch name as version. The using such an alias target is then done requiring libfoo/develop@... or libfoo/feature-x@... (we use gitflow on some projects) and means that the actually used version depends on the build time of the consuming package. However, given that the alias packages are transparently resolved, the resulting package is still fully reproducible and records the actual versions. Furthermore, CI is used again to ensure that we get working results anyway.

[lasote]

I have been thinking about this and maybe we can use the scm feature normally and for capturing the commit, use directly the internal SCM() or Git() classes. I'll assign it to the 1.6 to give it a try, maybe it doesn't require any feature to make it work, but maybe to make it easy.

[niosHD]

Sorry, I did not really understand what you are proposing, but thank you for looking into it.

As it turns out, I currently can only use the scm module when the version field is removed from the recipe. However, this requires that I have to provide the scm version, as part of the full package reference, on the command line which is error prone. Furthermore, it completely obsoletes the scm module in the first place. Therefore, any improvement is highly appreciated!

[lasote]

Hi! Please take a look at https://github.com/conan-io/conan/pull/3160 There is a test capturing the branch and the commit using the Git tool.
I think using this example together with the scm feature should be possible.

[benjaminseow]

Hello all, I have a similar use case with this.

Right now I specify a package something like this:

class TestgitConan(ConanFile):
  ...
  branch = "my_branch"
  version = branch
  scm = {
    "type" : "git"
    "url" : "[email protected]:conan-io/conan.git"
    "revision" : branch

I am hoping that Conan will automatically follow the branch, and update the package when it detects that there is a change. I think we do not need the previous versions of the package, as consumers following a branch are expected to keep up with the changes, otherwise they should use the tagged versions.

However this current method doesn't work, as Conan doesn't know that the remote has been updated. The digest comparisons do not detect any difference.

Perhaps the files with scm should always git pull / git fetch --tags to determine if there are any changes?

I'm guessing where to do this is in conans/client/cmd/export.py, where "The stored package has not changed".

Anyway, may I know what are the plans for the git_branch update? And how is this intended to be solved?

[niosHD]

@benjaminseow I may be wrong but your example reminds me a little bit of #3150. Maybe you can also comment there if it is indeed the same issue.

[benjaminseow]

Thanks, it does look similar. I will comment there too.
On Wed, 8 Aug 2018 at 7:00 PM, Mario Werner [e-mail]
wrote:

@benjaminseow https://github.com/benjaminseow I may be wrong but your
example reminds me a little bit of #3150
https://github.com/conan-io/conan/issues/3150. Maybe you can also
comment there if it is indeed the same issue.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/conan-io/conan/issues/3052#issuecomment-411368079,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ALZ2VHtDelnCzOAL7zXaZAMckrtmCxHFks5uOsTLgaJpZM4Un2cn
.