Nerdbank.gitversioning: Build Server support: set build server's build number to git version height during build

[clairernovotny]

Might be useful to have a way to set the build number the build server uses to match the calculated version.

I realize that all build servers have different ways to do this but it is possible:

• TeamCity
• AppVeyor
• VSTS Agents

To name a few popular ones

[AArnott]

Are you suggesting that we make it so TeamCity is influenced by NBGV, or the other way around?

Build server build numbers tend to be ever-incrementing integers. But NBGV build numbers are calculated based on git commit height. So they are intended to represent different things. I think build servers determine the build number to use before they even start the build, so it's far too late for NBGV to influence what that build number is.

Most build server integration I see in packages like these let the influence go the other way: the build number determined by the build server becomes the build number used in actual version stamping of the binaries. But that's exactly why I wrote this package: I don't believe in that. I think builds should be exactly reproducible by syncing to the commit that was originally built. The build server's stateful incrementing integer shouldn't break that, IMO.

[AArnott]

I sympathize though. I have often wish the numbers aligned. But then I think about how sometimes I need to build the same commit more than once, and I realize the build server's numbering scheme makes sense to keep the 2 builds separate.

[clairernovotny]

No, I meant that NBGV would set the build number...each build server has a way of having its version set programmatically and is implemented here by GitVersion:
https://github.com/GitTools/GitVersion/tree/master/src/GitVersionCore/BuildServers

[AArnott]

Oh! That's _very_ interesting. Thanks for the tip.

[AArnott]

The work for VSTS is done via #61 and commit 923fccde2ca19f88a913808308ace119ff83b3a7

[AArnott]

Hmmm... well, I guess I don't go as far as updating the build number itself -- I set other VSTS variables though.

[AArnott]

Working on AppVeyor cloud build number support now. Online documentation suggests it's simpler than making HTTP calls the way GitVersion does.

[KirillOsenkov]

Hey @AArnott there's something I don't understand about how this works with AppVeyor. If you look at the history here:
https://ci.appveyor.com/project/KirillOsenkov/msbuildstructuredlog/history
the version jumps from 1.0.17 to 1.0.27 after your PR 23, then continues to be incremented +1 (and not by Git height), then after 1.0.33 randomly jumps back to 1.0.27, and continues to be incremented +1 from there as well.

I saw you added this:
https://github.com/KirillOsenkov/MSBuildStructuredLog/blob/master/version.json#L12-L16

How does it work? Do I need to change anything in AppVeyor settings to respect that?

[AArnott]

Hi @KirillOsenkov, excellent questions. Short answer is everything you see is by design.

the version jumps from 1.0.17 to 1.0.27 after your PR 23

Yes, AppVeyor has its own concept of build number which set 1.0.17. Then after PR 23 NB.GV took over and as your git height was 27, it jumped to that.

then continues to be incremented +1 (and not by Git height)

I'm not sure why you say it's not by git height here. Your git history shows that each of these builds are in fact just one commit away from each other (you appear to be using direct pushes and single commits for each of these changes). So it looks fine to me.

then after 1.0.33 randomly jumps back to 1.0.27, and continues to be incremented +1 from there as well.

It's not random. 1.0.33 was the last calculated git height. In the meantime, AppVeyor continues to have its own idea of auto-incrementing build numbers which each build is assigned until NB.GV replaces it. When the build fails _before_ NB.GV replaces it, the AppVeyor-assigned build number remains. So that's why it appears to be 'thrown backwards' in build numbering. You'll notice that all the ones with the backwards build numbers are failed builds. So there you go.
You may want to change your appveyor version to 0.0.{build} so that when these occur it's more obvious it isn't in the normal versioning line of build numbers.

I saw you added this:
https://github.com/KirillOsenkov/MSBuildStructuredLog/blob/master/version.json#L12-L16
How does it work? Do I need to change anything in AppVeyor settings to respect that?

It simply causes NB.GV, during the build, to execute a special command that AppVeyor interprets as a request to change the build number.
Nothing else you need to change, except perhaps that appveyor.yml version change I recommend changing in my above paragraph.

[KirillOsenkov]

Perfect explanation, thanks Andrew! I've set my AppVeyor next build number to 0.0 instead, so that this never overlaps with what NB.GV is setting.

This also explains why the build started failing as soon as it reached 1.0.27. 1.0.27 was the first build number that was set by NB.GV, and so when an AppVeyor build starts initially, it uses the AppVeyor numbering, which happened to overlap with 1.0.27 (AppVeyor "reached" or "passed" NB.GV). As soon as that happened, AppVeyor failed the build, without giving NB.GV a chance to run.

Workaround is as you say, set a different version to AppVeyor as I did.

[KirillOsenkov]

Also you're right, I'm using 1 commit per push, so incrementing actually matches git height. I jumped to incorrect conclusions.

[AArnott]

I should update the README's FAQ with your questions though... they're good ones.

[KirillOsenkov]

Yes, basically the AppVeyor version sequence as set up in the AppVeyor Settings should never intersect with NB.GV since the former takes precedence and fails the build immediately in case of collision. Setting the AppVeyor to 0.0.* is a good advice.

[AArnott]

I've documented this nuance with appveyor in the cloud build doc.

[KirillOsenkov]

Thanks. I forgot to tell you that apparently setting the AppVeyor build numbering to 0.0 was ignored, so I set it to 1.0.1000 instead.

[AArnott]

That's very interesting. OK, I'll update the guidance.

[andreasnilsen]

The implementation class for TeamCity support seems to do nothing (?), how would one go about to retrieve a generated build number from NBGV in TeamCity?

I think there's missing a call to something like should be a call to something like

stdout.WriteLine($"##teamcity[buildNumber '{buildNumber}']");

in Nerdbank.GitVersioning/src/NerdBank.GitVersioning/CloudBuildServices/TeamCity.cs?

[AArnott]

@andreasnilsen I think the deal here was I couldn't find TeamCity documentation for how to set the build number. If you know, please file a dedicated issue for it referencing the docs. Or (even better) send a PR.

[clairernovotny]

@AArnott everything you need is already in the GitVersion repo as far as build server interaction goes:

https://github.com/GitTools/GitVersion/blob/master/src/GitVersionCore/BuildServers/TeamCity.cs#L58

[AArnott]

Nice.

[appendix68]

@AArnott:
Maybe I am confused here.
When using NBGV to set version, I can see it being applied on the assemblies them self, but I don't know how this can be applied to "build.number" configuration parameter in TeamCity to be used in steps after. It doesn't seem to be doing that? Please help. :)

[AArnott]

@appendix68, during the build within a cloud CI system, special steps exist that can be taken to change or set certain variables to affect subsequent build steps. In the case of TeamCity, it appears this is done through a special format of logged messages.

It seems from that code that we set buildNumber rather than build.number that you mentioned. Also it doesn't impact cloud build variables at all unless you turn it on in your version.json file by setting setVersionVariables=true.

Does that help?

[appendix68]

@AArnott Thank you for replying.
It correct that "buildNumber" is the right parameter, but the variable within TC is called "build.number". Sorry for the confusion.
My issue is now that even with the version.json like this:

{
  "$schema": "https://raw.githubusercontent.com/AArnott/Nerdbank.GitVersioning/master/src/NerdBank.GitVersioning/version.schema.json",
  "version": "1.0",
  "cloudBuild": {
    "setVersionVariables": true,
    "buildNumber": {
      "enabled": true
    }
  }  
}

I am still not seeing that "##teamcity[buildNumber '']" line is being executed during the build.

[AArnott]

You may not see it in the build log--at least with the CI systems I've seen, they suppress logging these special messages in the exposed output. You could test it yourself locally by building your project, setting the environment variables that this file looks for as "activation" checks. Then you should see the messages logged. And if you don't see them in TeamCity, either TeamCity has changed its environment so that the extension doesn't activate, or it's suppressing the messages from the log (but still processing them, presumably).

[appendix68]

@AArnott, sadly I cannot get it to work locally either. I've set the environment variables in Visual Studio in Pre-build event like this:
SET BUILD_GIT_BRANCH="refs/heads/develop" SET BUILD_VCS_NUMBER="5988cd7e189317859ecb7466b3091d083132ba79"

Getting the version works fine, but SetCloudBuildVariables returns following error:

No supported cloud build detected, so no variables or build number set. (TaskId:25)

Am I doing something wrong again?

[AArnott]

I've never tried setting the env vars in prebuild. I always use the developer command prompt to set them manually then kick off msbuild. Does that work for you?

[appendix68]

So, I got it to work locally. :)
The issue with Teamcity is that even with the variables set, it is not working.
What we did is that we added a new Message task to output the string when AfterBuild target is executed:
<Message Importance="High" Text="##teamcity[buildNumber '$(AssemblyInformationalVersion)']"/>

I just couldn't get it to set the variables as described in your docs.
Thanks for the help though.

[AArnott]

Strange that your workaround works when NB.GV doesn't as it seems to use the same mechanism and string to emit the message. Thanks for sharing your workaround anyways, @appendix68. :)

[wyrdfish]

In vsts projects with several builds / repos its confusing that the cloud build number is just semver.
So I put a powershell step in after a solution has built that puts identifying information back into the build number
e.g.
1.2.3 to Nerdbank gitversion master 1.2.3

#place this in a build step after a solution with nerdbank.gitversioning nuget has been completed
$newVersion = $env:SYSTEM_TEAMPROJECT + " " + $env:BUILD_DEFINITIONNAME + " " + $env:BUILD_SOURCEBRANCHNAME + " " + $env:BUILD_BUILDNUMBER 
Write-Host ("##vso[build.updatebuildnumber]$newVersion") 

[AArnott]

Thanks for sharing, @wyrdfish. Do you see what NB.GV is doing as a takeback? IIRC, default VSTS build numbers don't include this information either.

[nskerl]

I too have the TeamCity problem. It doesnt seem to be setting any of the ##teamcity parameters.

I do see the messages output when running locally with the env variables set, but nothing during the TeamCity build. I did confirm the env vars are set as expected and available to TeamCity (I printed them via a powershell step).

Anything I can do to help troubleshoot?

[AArnott]

Thanks for offering, @nskerl. I don't have TeamCity to test this myself. VSTS hides these messages in its own logs, so it's believable that NB.GV is doing the right thing in teamcity as well and that the logs are just suppressing those special control messages. That leads me to suspect that teamcity is simply not respecting the commands. Perhaps the commands NB.GV is emitting are wrong, or the variable name is wrong, etc. That's what I think would be best to investigate if you can do that.

[bbeda]

Hi,

Any clue on how can I take the GIT_VERSION and use it in Jenkins to publish NUGET packages? Is there a way to read it from the file or is it accessible anywhere from within Jenkins?

Thanks

[AArnott]

@bbeda Can you explain further what you mean? What is the GIT_VERSION you're talking about? This tool produces a lot of different versions (for nuget packages, assembly versions, etc.).

When you say you want Jenkins to use it to publish nuget packages, I don't know why you need the version to publish packages. You need the version to _build_ packages, but if they're msbuild self-packing projects then the NB.GV nuget package automates this versioning for you already.

[bbeda]

@AArnott Sorry. The question is not an issue on GitVersioning, that is working fine but rather about Jenkins.

So, I am using the GitVersioning in my library which I am building with Jenkins. After build, I want to publish the resulted NugetPackage on the Nuget server. GitVersioning is generating the correct file jenkins_build_number.txt but I don't know how to get the version number from the file so I can target the specific nupkg file.

I mentioned the GIT_VERSION because I saw in the source code that it is setting this Environment Variable as well however, I couldn't manage to make that work either.

Thank you

[AArnott]

Thanks for the added explanation. I'm rather unfamiliar with Jenkins. Is it unable to simply nuget push *.nupkg, such that you have to know the exact file name? If so, the way I'd be inclined to solve it is with a powershell or other script that finds the filename and then invokes nuget push. Is that an option for you?

Otherwise, you could run dotnet tool install -g nbgv as documented here and then use the powershell script (nbgv get-version -f json | convertfrom-json).nugetpackageversion to fish out the actual package version.

[bbeda]

I have already done the first option, before each build deleted all *.nupkg and at the end published all using the same pattern. That is however not fully using the tool and is more general so I was curious to see how I could use the file or eventual environment variable NB.GV is setting.

Thank you for the second suggestion.

[connorads]

Hi @bbeda, did you manage to get GitVersioning working with CloudBuild and Jenkins? Did you find a nice way of getting the version/build numbers? I'm experiencing the same issue.

[bbeda]

@connorads - sorry, somehow I missed this. No, I did not. I ended up clearing the workspace before the build and the publish *. Have you found a way to publish the specific file?

[connorads]

Hey @bbeda so we're using Jenkins Pipeline and we just read the version number from the text file generated by cloud build.

First we enabled cloud build in 'version.json` for one of our projects.

Then when you're building in a Jenkins environment dotnet build (I believe that it checks an environment variable that is set in Jenkins environment). It will generate a text file with the version number in it jenkins_build_number.txt

Then you can just read this out of the file into an environment variable and access it in different build steps.

script { env.VERSION_NUMBER = readFile 'jenkins_build_number.txt' }

[bbeda]

@connorads thank you. That is working fine however I have other issues

1. The file contains the BuildVersionNumber but my Nuget package gets generated using GitAssemblyInformationalVersion (which has an extra 'g'). Of course I execute a little something to add that 'g' in the variable but I would like to understand how is this supposed to be working out of the box.
2. It looks like not of the Build Variables are set. By example, how can I use GitAssemblyInformationalVersion in my next steps. I have dumped the EnvironmentVariables from one of the steps using bat 'set > env.txt' and cannot see any of those in the docs.

Here are my version.json
{ "$schema": "https://raw.githubusercontent.com/AArnott/Nerdbank.GitVersioning/master/src/NerdBank.GitVersioning/version.schema.json", "version": "2.0", "cloudBuild": { "setVersionVariables": true, "setAllVariables": true, "buildNumber": { "enabled": true, "includeCommitId": { "when": "nonPublicReleaseOnly", "where": "buildMetadata" } } } } and
Jenkins step
steps { dir(path: '...') { bat(script: 'dotnet build ...sln -c Release -r win-x64', label: 'Build') } script { env.VERSION_NUMBER = readFile 'jenkins_build_number.txt' } echo("${VERSION_NUMBER}") bat 'set > env.txt' } }
This works fine: echo("${VERSION_NUMBER}")
This prints null echo("${env.GitAssemblyInformationalVersion}") along with any other things I tried
@AArnott any ideas?
Thanks

[bbeda]

@AArnott
I looked at the code and it looks like the environment variables are set for the current process - which I guess is dotnet/msbuild (where this task is running at build).

Environment.SetEnvironmentVariable(item.Key, item.Value); in SetCloudBuildVariables.cs line 63
Is it just the case of making that set the EnvironmentVariables machine wide or am I missing something?

[AArnott]

@bbeda, environment variables can only ever be set in the current process. AFAIK there's no such thing as setting an environment variable machine-wide. All you can do is change the registry so that Windows Explorer starts (next time) with more environment variables, and as Windows Explorer starts all other processes (more or less) they inherit those environment variables when next started.
So the idea of setting environment variables in other, already running processes can't be done. The technique of leaving files or other hints so that other processes will change their own environment is a sound idea.

[bbeda]

@AArnott that makes sense, thanks. Just read the docs as well, sorry for the confusion.

However, the main problem I have here is that the value put in the file is not the exact one used to name the .nupkg file so I cannot target it directly. Right now I am adding that 'g' with a script and I understand why there are different values but could there be a way to get the NugetPackageVersion out of the box?

[AArnott]

Sure. You can get any variable you want either by extracting it from msbuild properties or by running this in a powershell script:

 (nbgv get-version -f json  | ConvertFrom-Json).NuGetPackageVersion

You of course would need to first install the nbgv dotnet CLI tool.

[bbeda]

@AArnott thank you very much! That solved my problem!