Conan: [question] Best approach for obtaining commit from debian changelog

[albaltimore]

Conan version 1.20.4 on Windows

Similarly to the workflow described here: https://github.com/conan-io/conan/issues/4139

My team builds for both Linux using debian dpkgs and Conan for Windows.

We are driving our versioning off the dpkg debian/changelog file. So first we release using dpkg, with some scripts which update our debian/changelog with a section of the form:

coollib (0.0.10.4) unstable; urgency=low

  * 7b84a69 awesome feature3
  * 23d7f40 awesome feature2
  * 84b4d2b awesome feature1

 -- Alban Lefebvre <[email protected]>  Tue, 11 Dec 2018 07:02:18 -0500

We have then created a shared python_requires to parse the changelog and extract the version and the commit for that version:

base = python_requires("conan_shared_python_requires/0.0.8@PORT/stable")

class CoolLibConan(base.conan_shared_python_requires):        
    name = "coollib"
    url = "[email protected]:myorg/coollib .git"  # conanfile location: in repo
    version, commitAtVersion = base.get_version_from_dpkg_changelog(name)
    scm = {"type": "git",
           "url": url,  # location of source to build
           "revision": commitAtVersion}

However, we have recently added a step in the dpkg build which not only modifies the changelog with the new version, but also updates some source file to keep the version consistent using bumpversion.
The issue I'm facing now is that the commit I'm extracting from the changelog (7b84a69 in the above example) does not include the commit where the changelog - and now some source files - are modified.
I would like to find a way for Conan to build the commit that includes the bumpversion change.

Right now, I have tried the following approach: modify the get_version_from_dpkg_changelog function mentioned above in the following way:
1) If a file debian/changelog_commit exists, read its content and use it as the commit to use in the scm command
2) If not, do some git operation to find that commit, and save it to the file debian/changelog_commit and add it to the exports so it's available when the build command starts

My main issue with my above implementation is that it has the side effect of leaving the file debian/changelog_commit in my clone, and if I don't remember to delete it, when running conan create ., it will always build from the commit in that file.

Would you have any suggestions on either?:

• delete the file debian/changelog_commit at the end of the build from the cloned repo where the create command was launched (knowing that at that point, it's building from the cache instead)
• find another way to transmit this commit information from the source step to the build step without using exports
• any other ideas (not sure I'm on the right track with my current approach)

• [X] I've read the CONTRIBUTING guide.

[jgsogo]

Hi, @albaltimore

For sure I don't have the right solution for this scenario, it is so specific and probably I cannot see the full picture you have in your company, but let me suggest to you a different approach:

I assume you have your conanfile.py file in the repo, so it is available also in the Debian context where you first generate your package without using Conan... but maybe you can still run Conan here.

In your Debian job you can write down the commit to a file and also run conan export and conan upload without creating packages (no need to commit the file, it is exported with the recipe to the Conan server, creating a new recipe revision).
Then, in your Windows, you can just conan install <ref> and Conan will take care of everything: the file with the commit is in the server, Conan will clone the repo and build the package

I would need to explore a little bit this idea, but I think it can work. It is more or less the same we are doing with SCM: we do not commit to the repo, we generate a file on-the-fly and upload it to the server, the conan-recipe-revision will be _linked_ to that commit in the repo.

[michaelmaguire]

Thanks @jgsogo but we wouldn't have the authority to add conan toolchain support to the non-Windows build environment.

[jgsogo]

You don't need to build it in non-windows, you just export and upload it, no need to build with Conan. On the Windows machine, you will install the package and build it using Conan.

[michaelmaguire]

@jgsogo yes, but we'd still need conan tools on the big corporate build farm to export and upload. Windows devs are the oppressed minority who dare not speak its name in our engineering environment ;-)

[albaltimore]

Thank you @jgsogo for your replies!

As @michaelmaguire mentioned, the approach you're mentioning would be hard to implement in our workflow. Also the Debian build and the Conan build being independent from each other, I believe we would also need to add some way to communicate the <ref> from the debian build to the Conan build (maybe via another file).

Going back to my current approach, I noticed that the function get_version_from_dpkg_changelog is called twice when calling conan create ., once when doing the exports from the git clone, and another time from the cache, to do the build. Is there a way to know which action is calling the constructor of our ConanFile class (here CoolLibConan)?

[jgsogo]

@jgsogo yes, but we'd still need conan tools on the big corporate build farm to export and upload.

Oh! I was missing that bit of information, quite important, so there is no-Conan at all on the non-Windows paltform. Then, the only way left to communicate information is a file inside the repo (the approach you are doing). I was looking for something like the SVN property $Revision$ for Git, but I cannot find anything reliable.

So probably, the only way to go (without any external server-tool) is to store that file in the repo itself, as you are doing right now.

Going back to your first comment:

My main issue with my above implementation is that it has the side effect of leaving the file debian/changelog_commit in my clone, and if I don't remember to delete it, when running conan create ., it will always build from the commit in that file.

You can create that file during the export step on-the-fly running a pre/post-export hook, that way you won't have the file in your clone and it will be recreated (with the _last_ commit) every time you export the recipe. Have a look at this hook export_metadata. Note.- I'm not sure if you can copy the file directly to the export folder, if not, you can follow the steps in this example hook and remove it in the post_export function.

Is there a way to know which action is calling the constructor of our ConanFile class (here CoolLibConan)?

If there is a way I'm sure it is not documented (maybe you can hack-detect it from some class attributes) and we will probably break it in the future, so I cannot recommend you that way.

Let me know if the hook approach can work for you. If you needed I can help you to write the hook (remember that you can share all the conanfiguration between your coworkers in a Git repo and ask them to install/sync some shared files)... I think it is the cleaner approach, but again, once you start coding it you always find some issues.

[albaltimore]

Thank you @jgsogo! I'll have a look at these hooks and get back to you if I need help with it.

[albaltimore]

@jgsogo So I was able to implement what you recommended (btw, congrats for the hook feature, it's pretty great 👍)

However, I realized that it wasn't the proper workflow for what I wanted to do. Let me explain why.
What I'm trying to do above is to extract a commit from a changelog and traverse to the child commit with git commands to be able to set it in the scm["revision"] of the conanfile.py as shown above. For some reason, I thought that the source action (here with a scm git clone) was done from the export folder, which is why we were exporting the debian/changelog. But after examining the conan source, I realized (and please correct me if I'm wrong) that a ConanFile object is created twice, once at the very beginning, and another time when doing the graph building between the export and the source step. And only the first instance is used during the source step using the scm property that was configured at the beginning.

So to summarize, it looks like I don't need hooks or any written file, but I could just do my git commands in the get_version_from_dpkg_changelog function which will succeed during the first instantiation as the run will be located in my git checkout, but will will fail (and it's ok) during the graph building phase as it will be run from the export folder.

[jgsogo]

Hi! Nice to see we are making som progress ;D

There are many scenarios to take into account when dealing with these stages (export, source, create,...), and I'm afraid it is not clear enough, even for us, and we'll try to simplify it in the future. Let me enumerate some examples:

• conan create <local/path>: it exports the recipe and run all the source, build, package,... stages. I don't want to mention here the -ks or -kb arguments that will skip some stages.
• conan export + conan install ... --build: it will export the recipe and sources and, in a different command, it is going to build the package.
• conan install ... --build: no local sources, recipe is retrieved from the remote and built

...also to note that Conan is using an internal cache to avoid loading the same file several times and, if we are using scm with auto keywords (not your case, I think) there is one optimization to take into account ("Tip" at the end of this page)

---

So to summarize, it looks like I don't need hooks or any written file, but I could just do my git commands in the get_version_from_dpkg_changelog function which will succeed during the first instantiation as the run will be located in my git checkout, but will will fail (and it's ok) during the graph building phase as it will be run from the export folder.

You can use a try/catch strategy to resolve the commit while you are in you working copy (during export) and it will fail when running in the cache because there is no repo there.

But I cannot see how it can work for a conan install ... --build (using a clean cache) because, in that workflow you won't have any sources in the cache, Conan will download the recipe, will execute the get_version_from_dpkg_changelog and fail becuase there is no repo... where is the file you get the commit from? If it is not stored with the recipe (exported together with it like the conandata.yml), how can it work?
*

[memsharded]

Hi @albaltimore

Just a quick idea, maybe the set_version() method could be used for this? It is only called at export time, and never used again when building the graph from packages in the cache. Check: https://docs.conan.io/en/latest/reference/conanfile/methods.html#set-name-set-version

[albaltimore]

@jgsogo @memsharded Thank you for your help! I've implement a solution using both of your suggestions and it works now 🙌
(Feel free to close this issue)

[jgsogo]

Perfect! Feel free to share you solution, someone might find it useful (people arrive at issues even if they are closed).

Thanks!

[michaelmaguire]

Thank you for your help. We went with the following. We would be very interested in understanding if you see any potential issues with the following snippets.

In our shared base conan_shared_python_requires:

# If the current directory is a Git repo, returns current branch
def get_current_branch():
    cmd = ['git', 'rev-parse', '--abbrev-ref', 'HEAD']
    return subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL).stdout.decode('utf-8').rstrip()

# Given a commit and a branch, returns the child commit of the debian/changelog if it exists
# Otherwise, returns the commit given as input
def get_child_commit_of_changelog(commit, branch):
    commit_range = "{}..{}".format(commit, branch)
    cmd = ['git', 'log', '--format=%H', '--reverse', '--ancestry-path', commit_range, '--', 'debian/changelog']
    child_commits = subprocess.run(cmd, stdout=subprocess.PIPE).stdout.decode('utf-8').rstrip().splitlines()
    if len(child_commits) == 1 and child_commits[0]:
        return child_commits[0]
    return commit

# For libraries also in dpkg: Return a version number and an associated commit based on our latest debian/changelog entry
#
# Looks for first (newest) entry of the form:
#
#    mypackage (0.0.8) unstable; urgency=low
#
#      * 88b2c0e7 Fix CASE with string output type
#      
# and parses the version number and associated commit.  If there is a child commit that in turn contains 
# the commit that modified the debian/changelog file itself, returns that commmit.
#
# So release cycle is: promote your updated dpkg on Linux first, then git pull and 
# build on Windows with Conan to publish a Conan Package with the corresponding version.
#
# Usage: Ensure your conanfile.py references this with:
#   base = python_requires("conan_shared_python_requires/[>=0.0.9]@PORT/stable")
# then implement a def set_version(self) method which calls this using:
#   
#     def set_version(self):
#        self.version, self.commitAtVersion = base.get_version_from_dpkg_changelog(self.name)
#        self.scm["revision"] = self.commitAtVersion
#

In the package itself:

base = python_requires("conan_shared_python_requires/[>=0.0.9]@PORT/stable")

# We cannot subclass base.conan_shared_python_requires because it defines its own self.version
# and our set_version() below cannot co-exist with that.
class nxPlatformApiConan(ConanFile):
    [...]
    # We  will override scm.revision in set_version() below before base source() method uses this attribute
    scm = {"type": "git",
           "url": url,
           "revision": "master"}
    [...]
    def set_version(self):
        self.version, self.commitAtVersion = base.get_version_from_dpkg_changelog(self.name)
        self.scm["revision"] = self.commitAtVersion

[jgsogo]

Hi!

Some considerations:

• Use the self.recipe_folder attribute in the set_version() function to find the path to the Git repository (we might change the working directory to run some methods, but the recipe_folder will be the same).
• Use the new python_requires syntax, this _legacy_ syntax will likely be deprecated in Conan 2
• Modifying the attribute self.scm in the set_version() will probably work in the next versions, but it is something we are not testing and our idea for the next versions is to restrict as much as possible the attributes available in each method (not sure about this). I'm pretty sure that we'll keep calling set_version before storing scm data to conandata.yml (Conan 2 will activate scm_to_conandata), but again, it is something we are not testing right now.

The fix, if needed, would be to call `base.get_version_from_dpkg_changelog` twice, one inside `set_version()` and another one to assign the data to the `scm = {... "revision": base.get_version... }` field. 

🤔 And yes, I realize that here you cannot use the `self.recipe_folder`, so you have to trust that the working directory makes sense for your `git` command... (https://github.com/conan-io/conan/issues/6502)

None of these points is a big change and you will ensure that the recipe works with newer versions.