Gitversion: [Feature] - Run Gitversion via a githook to pre-calculate version variables.

[dazinator]

Detailed Description

I'm thinking that gitversion.exe could have a new command for installing a git pre-commit hook into the local git repository.

This pre-commit git hook would:

1. Execute gitversion automatically upon pre-commit (I.e on the developers local workstation)
2. Update a local "version.yml" file in the repo with the new gitversion variables, and include that in the staged files with the commit.

The idea would be that, in CI build scenarions, we'd be able to execute GitVersion.exe but tell it to read the version from the file in the repo rather than calculate it.

This is only a very high level idea that I wanted to capture for feedback purposes, I haven't thought it through in much detail at present.

Context

Why is this change important to you? How would you use it?
This change would be important in:

1. Speeding up the build - gitversion could read the pre-calculated version information from the file that was updated by the installed git "pre-commit" hook being active on developer workstations. This would negate the need to call into git or do any version calculation logic during a CI build. Therefore this could also aleviate issues like we have with the msbuild task and concurrent usage of git.

How can it benefit other users?
Other uses could benefit from improved CI build times at the expense of slightly longer local commit times and an additional file change showing up in commit history.

Possible Implementation

Here is an example git pre-commit hook script. To install this script it needs to be copied to ".git\hooks\pre-commit"

#!/bin/sh; C:/Program\ Files/Git/usr/bin/sh.exe
echo "running precommit hook"

#run gitversion here to update "version.yml" file in the root of repo
echo "updated version.yml"
# Add the modified file to staging to include in the commit.
echo "version.yml" | xargs git add
exit 0

It shouldn't be hard to see how this could be completed to make a call to gitversion.exe and then dump the version variables in a "version.yml" file which would be a file that would need to be tracked by git in the repo. This would be automatically be updated then on each commit.

Then when it comes to running a CI build, we could add an argument to gitversion.exe so that it can use a pre-caclulated set of version variables from some file in the repo e.g the "version.yml" file rather than caclulating anything.

P.S I would say "version cache" but I am not sure if the version cache is comitted to the repo - I don't think it is, so I think we need to distinguish this from that.

[arturcic]

Then when it comes to running a CI build, we could add an argument to gitversion.exe so that it can use a pre-caclulated set of version variables from some file in the repo e.g the "version.yml" file rather than caclulating anything.

That means that if there is no version.yml we should execute the old way, right?

[dazinator]

Then when it comes to running a CI build, we could add an argument to gitversion.exe so that it can use a pre-caclulated set of version variables from some file in the repo e.g the "version.yml" file rather than caclulating anything.

That means that if there is no version.yml we should execute the old way, right?

Yeah!

[dazinator]

Looking at other git hooks here, there are some others that we might want to automatically update the version.yml on like "pre-merge-commit": https://git-scm.com/docs/githooks

[dazinator]

Im also not sure if running gitverison during a "pre-" operation would be problematic. Technically if the commit hasn't been made yet then we'd calculate one version behind! We can run as "post-" hooks instead, but that means you can't influence the commit to include the modifed version.yml file so options are:

1. Run as "pre-commit" and "pre-merge-commit" hooks. Auto include the modified "version.yml". Compensate for versaion number being 1 commit behind by adding 1 the version number before saving to the file.
2. Run as "post-commit" and "post-merge-commit" hooks. Don't auto include the "version.yml" just update it. Developer must stage / commit the version.yml changes themselves when they are done, before pushing to the remote.

In addition to the above we could also install a "pre-push" hook that acts a safety net:

1. Runs gitversion to calculate the version and compares that with the committed version.yml version, if the version.yml version doesn't match, abort the push with an error. This would act as a safety net to ensure this file is always up to date prior to pushing to a remote.

These are all just ideas at present. I think git hooks could potentially offer a nice model though..

[dazinator]

Run as "pre-commit" and "pre-merge-commit" hooks. Auto include the modified "version.yml". Compensate for versaion number being 1 commit behind by adding 1 the version number before saving to the file.

I'm not sure this would work, because its not just a case of incrementing a counter by 1, other variables include the commit SHA etc which aren't available until after the commit is made. So I think that would seem to rule out this option.

Run as "post-commit" and "post-merge-commit" hooks. Don't auto include the "version.yml" just update it. Developer must stage / commit the version.yml changes themselves when they are done, before pushing to the remote.

I am not sure this would work either, the act of committing the changed version.yml file is itself another commit that would change the calculated version variables and commit SHA. Therefore the values saved in it (like commit SHA etc) would be immediately outdated upon commit. Unless we can somehow have GitVersion ignore changes to this file in its version calculation logic.. Also I'm not sure about the recursion caused when adding a another commit in post-commit hook - would that cuase another execution of the post-commit hook to run for that new commit? If so, we'd have to add logic to detect a cycle (i.e a commit just made by the post-commit hook) and break out. If we did all this the flow would be:

1. Developer commits a change.
2. Post commit hook runs:
   
   
   
   • executes gitversion.exe, saves updated version.yml
   
   
   • stages version.yml and commits it as an additional commit.
   
   
3. Post commit hook runs:
   
   
   
   • detects that the last commit is just for version.yml - exits and doesn't do anything.
   
   
4. Developer pushes changes to the remote.
   
   
   
   • Note: Commit history will contain a bunch of version.yml updates that might annoy poeple. Perhaps we can find a way to extend the hook here to squash them all into a single change to "version.yml". Doing this automatically is possibly not a good idea. Not sure how best to deal with this but it would be nice if we could do something.
   
   
   • pre-push hook runs gitversion and compares output to version.yml - of different prevents the push writing a message to STDOUT indicating the problem.
   
   

Once changes are pushed, the remote CI build:

1. Executes gitversion.exe with some new flag to read from version.yml in the repo.
2. Giversion.exe reads the version.yml to populate its variables and bypasses git completely.

P.S I have been able to develop the actual githooks using csharp (https://github.com/filipw/dotnet-script), and can execute and debug it in VS code, so I have a really nice development experience for the hook itself. I'm able to run gitversion exe in pre-commit and on post commit and save the version JSON output to a file. So it's now just figuring out the actual steps to do this nicely, and working out if this would be a viable feature that people would be interested in using...

[dazinator]

Another downside of this approach is that I don't think you can enforce githooks that run on the server. Therefore if you allow commits directly via github (to readme's etc) the version number isn't going to be updated. This could potentially be solved with a web hook but now I think you are starting to ask for a bit much...

All in all I think I am beginning to go off this feature suggestion.

[dazinator]

However should it prove useful in future - here is my example githook experiment: https://github.com/dazinator/GitHookExperiment

[arturcic]

Thanks @dazinator for sharing the ideas!

[dazinator]

@arturcic I am re-opening this one because I recently stumbled accross git-notes: https://git-scm.com/docs/git-notes

I am wondering now, if we could use a POST-COMMIT hook to execute gitversion and then store the version variables as git note. Then change gitversion.exe so that it will first check for a git note containing the version variables first, and use that if found. Sadly this means using git still though... so I am not sure whether it would resolve the concurrent lock issue or not - my thinking would be grabbing a git note should be a lot quicker than walking the commits or anything..

Gonna leave it open.

Another idea - we could create a web service in the cloud that could:

1. Receive a git SHA together with the sem ver version variables - and store it.,
2. Later be used to get back the version variables for a specified GIT SHA when asked.

The idea there being, a gitversion POST COMMIT hook could calculate the version and call that cloud service to store the variables with that SHA. Later during a CI build if GitVersion.exe is executed with an additional argument, it could check with that cloud web service to see if there are already version variables held for the current SHA to get the version variables back..

I beleive many CI systems would provide the current SHA as a build variable - meaning you wouldn't need to run any git logic at all during a CI build, as long as the SHA is provided to you by the CI build environment at the start of your build then we can recover the version variables.

[arturcic]

Sound interesting.. need to think about

[dazinator]

Found a free key value pair web service:

https://keyvalue.xyz/

So thinking as a proof of concept, the POST COMMIT git hook could be configured to post a key / value pair to that website, the key being HEAD SHA and the value being a condensed form of the current Variables.json

Then make a change to GitVersion.exe (implemented behind a new command line arg) to grab the variables from there based on current commit SHA, where SHA was provided from the build server as an argument.

If I can get this working as a POC first in my own branch I'll see if it has any legs.

[arturcic]

If you can get the POC working we might consider that for version 6.0.0

[asbjornu]

So thinking as a proof of concept, the POST COMMIT git hook could be configured to post a key / value pair to that website, the key being HEAD SHA and the value being a condensed form of the current Variables.json

That sounds very convoluted. Why is an external web service needed for this? Isn't the file system, possibly manipulated through Git – with git notes as you suggested, or otherwise, – a much simpler and sufficient solution?

[dazinator]

@asbjornu
I wasn't very familiar with git notes when I stumbled accross it.
My issue with it is that notes aren't automatically pushed to a remote (or fetched unless you update the refspec):

git push <remote> refs/notes/*
# to push all notes. Fetching can be done with

git fetch origin refs/notes/*:refs/notes/*

I thought this might be annoying to the end user if they forget to push the notes branch then the CI builds wouldn't take advantage of the note and would have to calculate from scratch.- not to mention the CI build would have to fetch this additional branch every time - and I wasn't sure whether that would be a pain to set up.

So I was in two minds as to how best to solve that and started considering alternatives.

Using the file system to store version variables is also a pain because:

1. If you create this file in a PRE-COMMIT hook you can include it in the current COMMIT automatically (kind of nice I guess - but maybe not if you don't want noise in commits), however - as soon as it's committed, the current commit SHA now changes, and all it's information is now stale (1 commit behind).
2. If you create this file in a POST COMMIT hook, the file has all the current commit version variables, but it hasn't been comitted - so won't be pushed. As soon as you commit the file - it's now stale as it will be one commit behind.

Do you have any ideas to overcome these problems? Do you think there is a way to get git notes working so it doesn't disrupt the end users workflow or give them extra actions? Other than that - calling out to a webservice to dump a key value, and then retreiving it during a CI build was my next best idea.

[asbjornu]

Committing the version number is useless, as you mention. I didn't know git notes weren't pushed and pulled by default; that's very unfortunate. GitVersion is already manipulating the Git repository through normalization, though, so perhaps we can expand this to also include pulling and pushing of notes?

Git normalization should probably be changed drastically to simplify or overcome #2194 anyway.

[dazinator]

though, so perhaps we can expand this to also include pulling and pushing of notes?

I like this idea. But am I right in thinking this would only work when using GitVersion to normalise the repo i.e where the Git branch env var was set. For situations like azure pipelines jobs with the azure pipelines gitversion task - I haven't needed to set any git_branch env var, stuff just worked because the pipelines job auto fetches sources - the entire history and checks out the branch so gitversion normalisation doesn't come into play?

In terms of auto pushing up notes I can look into whether a git push hook could be installed that would also ensure notes branch was pushed - so no extra manual work for the developer

[dazinator]

Also one of the main goals of this feature is to prevent (concurrent) usage of git - as the idea is that the version variables should be readily available from somewhere and not have to use git to calculate them. GitVersionTask has issues where multiple instances of it run concurrently in a solution all using git at the same time causing a failure. If we used a notes branch to store the version variables in - we'd still have to use git to pull that branch and grab the note - I'm not sure if that is done concurrently by multiple girversion tasks would there still be an issue? If it was a file, or stored by a key value pair web service using a build variable as a key (sha) then it eliminates usage of git entirely. I think we have eliminated the idea of being able to store version variables in a file because we can't commit anything to the repo.

[asbjornu]

I'm not sure how Git would react to multiple concurrent git fetch origin refs/notes/*:refs/notes/* calls. If it handled that gracefully, I'd say it's a viable solution.

[dazinator]

I think git notes is looks promising. It's whole purpose is for storing metadata against a commit without modifying the commit.

@arturcic at the moment I have to use the post-commit hook to run gitversion.exe - get back the variables in json form, and then execute `git notes add -m "{version-variables}".

I am thinking it might be a good idea eventually to extend gitversion.exe with a seperate command for outputting the variables to a git note!? As well as restoring the variables from a git note perhaps. But I know the CLI might be being reworked soon, so I won't attempt anything like that for now, i'll just keep the logic all in the post commit hook.

[dazinator]

I've done some more work over here: https://github.com/dazinator/GitHookExperiment

It should now be possible to clone that repo, and follow the README, and see gitversion executing as a post-commit git hook inside that repository, and adding the version variables as git note. I had some issues committing a git note that contained JSON in the message - I think due to the fact that the json contains double quotes - I couldn't work out the correct escape sequence when piping it in to the git notes command through System.Diagnostics.Process.StartInfo.Arguments. What I decided to do instead, was to convert the JSON to YAML, and commit a git note with the variables in YAML format instead. This seems to work.

@asbjornu I have had a look at how we would get the notes automatically fetched and automatically pushed.

We can update the refspecs when normalising as you mentioned, to make it so notes are automatically fetched:

[remote "origin"]
    fetch = +refs/notes/*:refs/notes/*
    fetch = +refs/heads/*:refs/remotes/origin/*
        # omitted for brevity

However I have yet to find a way to make it so that notes are automatically pushed. There was some interesting discussion here but I couldn't see anything conclusive.

[dazinator]

For auto pushing notes, the best I can come up with is to create a git alias that can push the current branch but also push the notes branch.

The end to end experience for an end user would be:

1. Execute gitversion.exe's new command for installing the githook into their local repo:

gitversion install-hook # or whatever we call this command

This:

• Will deploy the post-commit hook to the ./git/hooks directory.
• Could also set up the alias for them, or we ask them to do that in the docs - its a case of running:

git config --global alias.push-notes "!git push origin refs/notes/* && git push"

• Could also update their refspecs for the origin remote for them, again we could document this in case they are working with a remote not called "origin"

[remote "origin"]
    fetch = +refs/notes/*:refs/notes/*
    fetch = +refs/heads/*:refs/remotes/origin/*
        # omitted for brevity

1. They now fetch and pull as normal, and notes would be included thanks to the refspecs.

2. When they commit a change - gitversion post-commit hook auto adds a note with the version variables in yaml format, for their commit.

3. When they push - they would use the alias:

git push-notes

Or remember to push the notes branch up first:

git push origin refs/notes/*
git push

The alias makes it slightly less of a chore. But I am aware that GUI tools won't be aware of the alias, and many people will use those to push branches. That's why the alias solution isn't perfect.

It might be possible to reduce the risk of a user forgetting to push notes - by also installing a pre-push hook, so that when they push a non-notes branch, the hook will check to see if notes have been pushed and if not, prevent the push with a suitable error message.

From the build server perspective:

1. Gitversion's normalise function will update the refspec to fetch notes, or we document how to do this for your build / ci system.

2. GitVersion.exe can be modified to check for a note for the commit being built and restore the version variables from there (I've used yaml format). If successfully restored, version calculation is not necessary - mission accomplished.

[dazinator]

Note to self: I need to investigate this thread:

So use git fetch origin refs/notes/:refs/notes/origin/ - and then merge them with your local notes

[asbjornu]

Thanks for the investigation and prototyping, @dazinator! With the new command-based CLI and (hopefully) more compartementalized architecture of GitVersion 6, it should be no problem to introduce support for git notes in a plugin project that adds a gitversion notes command (or something similar) with options to make this work.

[dazinator]

@asbjornu cool - looking forward to having a go at that when its ready then. Is the idea with plugins that they would need to be installed separately and live in a separate repo, or would this notes plugin live in the gitversion repo still and be bundled with gitversion distributions?

[arturcic]

@dazinator we did not decided yet. We're still in the brainstorming phase. You can join us on Slack channel

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. After 30 days from now, it will be closed if no further activity occurs. Thank you for your contributions.