Conan: generating version from git

[Prinkesh]

i have the conan package recipe and sources in the same git repo , in order to generate the version from git, currently i have to export some intermediate file with the version information while exporting the package recipe as the build happens in conan local store and the whole git directory cannot be exported as that can be huge

Is there any way the version information can be exported without generating the intermediate file ?

[memsharded]

The problem is that by design, the package has to know the version, as there is no general way to know the version from git (or from other vcs, git is not the only one): would it be a tag? the current branch? What happens if you have uncommited changes? So the creation of the package recipe is totally decoupled from any vcs. It is true that in the source() method you can git clone, but you might see it as a download(), as it is just a way to retrieve source code files, not real operation/sync with a vcs.

Can you show the workflow and file to generate the version? Because it is not just a file, it should be assigned to the version field in Conanfile, right?

[Prinkesh]

I modified the _parse_module method in loader to differentiate when the class is loaded for export and other tasks

If its called by export call then version information is generated ( by VersionInfoGenerator) and saved in a file named .conan_version_export which is always in the exports list and for other calls the version is loaded from this file by VersionInfoLoader()

 def _parse_module(self, conanfile_module, consumer, filename, is_being_parsed_for_export = False):
        """ Parses a python in-memory module, to extract the classes, mainly the main
        class defining the Recipe, but also process possible existing generators
        @param conanfile_module: the module to be processed
        @param consumer: if this is a root node in the hierarchy, the consumer project
        @return: the main ConanFile class from the module
        """
        result = None
        for name, attr in conanfile_module.__dict__.items():
            if "_" in name:
                continue
            if (inspect.isclass(attr) and issubclass(attr, ConanFile) and attr != ConanFile and
                    attr.__dict__["__module__"] == filename):
                if result is None:
                    result = attr
                else:
                    raise ConanException("More than 1 conanfile in the file")
            if (inspect.isclass(attr) and issubclass(attr, Generator) and attr != Generator and
                    attr.__dict__["__module__"] == filename):
                    _save_generator(attr.__name__, attr)

        if result is None:
            raise ConanException("No subclass of ConanFile")

        # check name and version were specified
        if not consumer:
            if not hasattr(result, "name") or not result.name:
                raise ConanException("conanfile didn't specify name")

            if not hasattr(result, "vcs"):
                raise ConanException("conanfile didn't specify vcs .Values can be [git, svn , None]")

            if not is_being_parsed_for_export and vcs is not None:
                #then load the version from cached file else user specified the version manually leave it as it is
                result.version = VersionInfoLoader(os.path.dirname(conanfile_module.__file__))

            if is_being_parsed_for_export:
                if result.vcs is None:
                    if not hasattr(result, "version") or not result.version:
                        raise ConanException("conanfile didn't specify version")
                elif result.vcs == "svn":
                    raise ConanException("autoversioning for vcs type : svn not implemented")
                elif result.vcs == "git":
                    if not hasattr(result, "use_tags") or type(result.use_tags) is not bool:
                        raise ConanException("autoversioning for vcs type : git \n conanfile didn't specify correct versioning mechanism use_tags [True/     False]")
                    if result.use_tags:
                        #Use git tags as version
                        result.version = VersionInfoGenerator(os.path.dirname(conanfile_module.__file__), vcs = "git", use_tags = result.use_tags).          get_version()
                    else:
                        # check if version specified has major and minor version
                        if len(result.version.split('.')) != 2:
                            raise ConanException("git versioner expects only the major and minor version in conanfile version.")

                        result.version = result.version + "." + VersionInfoGenerator(os.path.dirname(conanfile_module.__file__), vcs = "git", use_tags =     result.use_tags).get_version()
                    #raise ConanException("autoversioning for vcs type : git not implemented" + result.version)
                else:
                    raise ConanException("unknown vcs type in conanfile")

            if not hasattr(result, "version") or not result.version:
                raise ConanException("conanfile didn't specify version")

        return result

[memsharded]

I see, you have gone deep into the conan code :)

We are using in some places environment variables, I think for this case they could be handy. You might be able to automate that in your conanfile without requiring modifying the _parse_module, which eventually could be a bit more complicated to maintain.

So the basic thing we are doing with environment variables is for users and channel in test_package, but would be the same for the version

import os
conan_version = os.getenv("MYPACKAGE_VERSION", "1.0")

class MyPackageConan(ConanFile):
    name = "MyPackage"
    version = conan_version
    ...

Then in the command invocation, get the MYPACKAGE_VERSION with some git-fu, and assign to the environment variable.

A more elaborated version would be to execute the parsing code in the recipe:

import os

def parse_version():
      conan_version = os.getenv("MYPACKAGE_VERSION", "1.0")
      if os.path.exists(".git"):
            output = StringIO()
            ConnanRunner()("git describe ... ", output)
            conan_version = output.get_value().some_regex_or_parsing()
       ....


conan_version = parse_version()

class MyPackageConan(ConanFile):
    name = "MyPackage"
    version = conan_version
    ...

From conan 0.13 it is possible to share python code between recipes, just by creating a python package with the code you want to share. It is still an undocumented feature, but it is there, so maybe the parse_version() could be a use case for it.

Just some ideas to think about, I don't say that these would be the only approaches, you already have done a lot of work, so you might know better.

[Prinkesh]

But then I won't be able to know if the package is being exported or some other action is being done

[lasote]

But only during the export command you have the .git folder present, right? So, you could use a get_version method: if your sources are present parse git or whatever and generate a version file , otherwise read the version file.

[memsharded]

Yes, maybe something like:

def parse_version():
      conan_version = os.getenv("MYPACKAGE_VERSION", "1.0")
      if os.path.exists(".git"):
            output = StringIO()
            ConnanRunner()("git describe ... ", output)
            conan_version = output.get_value().some_regex_or_parsing()
            save("version_file.txt", "%s" % conan_version)
       elif os.path.exists("version_file.txt"):
             conan_version = load("version_file.txt")
       ....
       return conan_version


conan_version = parse_version()

class MyPackageConan(ConanFile):
    name = "MyPackage"
    version = conan_version
    exports = "*version_file.txt"

[Prinkesh]

I went with modifying the _parse_module for exporting the version information, while loading i am using the method you suggested.
I think it would be a good option to have a method called export in conanfile which is invoked only while exporting the package so that user can setup any intermediate information like generating version information or dynamic requirements

like in case of dynamic requirements i am getting the version of a package from some external database which won't be consistent if i don't save these values while exporting and later use the saved values.

one more case could be when building insource user could export the source folder path and later use in source method of conanfile

[memsharded]

I like the idea, I think it could be a good approach and totally inline with conan design. I guess that just setting the version to None, and letting the export method defined might be ok. It could be a bit tricky to implement, as in some places we might assume that the version is just there, and I guess this function should be executed before actually doing the export to the local cache (you need the version to do the export)

Dynamic requirements, any example? There is a requirements() method that can be use to do things like conditional requirements (they can be done in configure() too).

[Prinkesh]

Yes the function should be executed before checking rest of the attributes of conanfile .

dynamic requirements is basically to solve this #15
what i have done is exposed a command named

conan link reference alias -r remote

on running this an entry in database on remote server is created . So, now the package manager has ability to create alias like 1.1.latest / 1.1.X and the consumers can use it by using a helper method in their conanfile

suppose the helper method is

get_reference(name, alias, user, channel , remote)

this fetches the actual reference from database in remote . hence consumer is able to now reference other packages by their alias.

But there is a catch in this , what happens in the sequence of commands given below

1. manager of package named A updates a alias called 1.1.latest to 1.1.5

 `conan link A/1.1.5@demo/test 1.1.latest -r remote`

1. consumer uses this alias, exports and uploads its package named B, so B while building referenced A/1.1.latest ( which resolved to 1.1.5 )
2. now the package manager of A , changed alias 1.1.latest to 1.1.6

 `conan link A/1.1.6@demo/test 1.1.latest -r remote`

1. the issue arises now when someone installs package B

the command get_reference("A", "1.1.latest", user, channel , remote) in conanfile will return A/1.1.6@demo/test and this will shown as dependency of package B instead it should have been A/1.1.5

To avoid the above scenario from happening it is necessary to cache the output of all such commands get_reference("A", "1.1.latest", user, channel , remote) while exporting the package recipe

The final conanfile looks something like this

remote_host = "local"
user, channel = "demo", "test"
class PackageBConanFile(ConanFile):

    name = "Test"
    vcs = "git"
    use_tags = False
    settings = "os", "compiler", "build_type", "arch"
    generators = "cmake"
    exports = "CMakeLists.txt", "build.py"

    @staticmethod
    def export_requirements():
         """ This method is called only on export, and before version is checked in conanfile 
              All the requirements defined in this method by using ReuirementExporter is saved in a file 
              and exported with the package
         """
        requirement_exporter = RequirementExporter(remote_host)
        requirement_exporter.requires("PackageA", "1.1.latest", user, channel)

    def requirements(self):
        """ RequirementLoader loads requirements from the exported file in export_requirements instead of querying the remote database (as that might have changed)
        """
        requirement_loader = RequirementLoader(remote_host)
        self.requires(requirement_loader.load("PackageA", "1.1.latest", user, channel))

[memsharded]

Hi @Prinkesh,

Any new thought on this now that there are version-ranges in conan? The main problem that you were trying to solve might have been already addressed by them: http://docs.conan.io/en/latest/reference/conanfile.html#version-ranges

Please tell us, or if the possible export() method in the conanfile could have any other relevant use case. Thanks very much!

[memsharded]

I think this issue can be closed now, but please @Prinkesh, tell if otherwise, or reopen the issue. Thanks!

[steptosky]

Hello guys I have one interesting issue. I have a cmake script that generates c++ header with the project description including branch name and commit id from the git and I have the problem. When I export my sources I don't export .git folder and while building the c++ header is generated incorrectly because the cmake script can't retrieve git information.
Any ideas how to fix this issue? I would like conanfile to have "export" method and to do retrieving git information to file and then export it with the sources but it seems there is no such a method.

[memsharded]

Why not using the source() method instead to do a self.run("git clone...") instead? Then you will have the git repo, and it will work.

If you still want that functionality, really, the exports() method is not needed. As the recipe is a python script that will be loaded, you can implement any functionality and call it at the global level, something like:

def my_git_method():
     # execute some git command to retrieve the info
     # then dump it to some gitinfo.txt file
     # make it dont fail and don't create file when the git repo doesn't exist

my_git_method()

class MyPkg(ConanFile):
      exports = "src/*", "gitinfo.txt"

Please try this approach and ask if you need further help.

[steptosky]

Thx for the advice for the global function, I did not think about this approach, I will try it and I think it will work well.
P.S Working with the sources is not conveniently when you in develop mode on the same machine. For example: I work on a library and executable file that uses that library at the same time but remote repository does not have all necessary changes.
The source() is good for the releases and for the end users but not for developing or maybe I don't know how to cook it for developing :) .
I'm still trying to search conveniently approaches for the developing. If you know where I can read the best practice for such a situation, please, let me know.
Thank you.

[memsharded]

Ok, @steptosky let me know how it goes.

You are right about the development of source for packages flow. I have just added a new issue https://github.com/conan-io/conan/issues/1171, so we review the best practices, document workflows, or change or implement some functionality to help in the process. Not possible to add it to 0.22, which is already full, but will try to address this point for next 0.23. Thanks very much for the feedback.

[steptosky]

Thank you guys for the good supporting and the good product.