Nunit-console: Master Chocolatey issue

[CharliePoole]

I'm creating this issue to coordinate our adoption and use of chocolatey. Almost all the usage will center around the console runner, so it makes sense to discuss it here. See also prior discussion on issue nunit/nunit#452, including a lot of good info about use of chocolatey from @ferventcoder.

GENERAL APPROACH

1. Although it's an over-simplification, Chocolatey is generally good for tools and their extensions, while NuGet is good for libraries referenced by user code. We will use it that way.

   • The framework will be distributed using NuGet
   • The console runner, gui (even though separate), engine and extensions will be distributed using Chocolatey.
   • The console runner, engine and extensions are __also__ distributed using nuget and we will keep doing that so long as users want it.

2. The engine directory will contain an addins file nunit-choco-addins with a relative path to pick up all extensions installed under chocolatey. That is, the set of extensions available to a chocolatey installation of the runner will be separate from extensions installed in any other way, such as using nuget.

3. To facilitate (2) all engine extensions will use an id in the form nunit-extension-*. This is analogous to our use of NUnit.Extension.* for nuget but conforms to chocolatey naming standards.

4. Any chocolatey packages that bundle multiple extensions or other components will do so by use of dependencies, not by directly including the software. This implies that nothing can be bundled unless it already has a separate package, which seems like a good principle. I'm inclined to encourage users to install just what they need because it educates them and eliminates surprises, but I recognize that people are used to getting bundles of functionality. Chocolatey may help in encouraging unbundling, since you can install a runner and multiple extensions simply by listing their ids on the command line.

5. The code for creating packages for a single module, such as the runner or an extension, really belongs in the project that creates that module. I am currently building them in a separate project in my own id, but they should be moved to the individual projects when convenient - probably after the next release.

6. Packages that bundle a number of extensions with a particular executable should be associated with the executable project. Packages - if any - that bundle multiple executables from different repositories should probably be kept separate.

STEPS NEEDED

1. Create package for the console runner

2. Create packages for each extension

3. If we want a combined package with common extensions, create that.

4. Deprecate the old nunit package (framework+consolerunner+extensions) in favor of (3) or (1).

5. Create a similar set of packages for the GUI.

STATUS

(1) and (2) are ready to test

@nunit/core-team @nunit/engine-team @ferventcoder
Your comments are solicited and anxiously awaited! 😸

[ChrisMaddock]

All sounds brilliant!

If we want a combined package with common extensions, create that.

You're already aware I'm very much in favour of a choco package which relects the current contents of the NUnit.Console NuGet package.

Deprecate the old nunit package (framework+consolerunner+extensions) in favor of (3) or (1).

A combined console + extensions package would be the closest the the existing 'nunit' package. I think that would be the preferable replacement, over a console on it's own.

Packages - if any - that bundle multiple executables from different repositories should probably be kept separate.

Given we're looking at merging nunit-distribution _back_ into the nunit-console repo (https://github.com/nunit/nunit-console/issues/218) can I suggest any choco packages which bundle the same components live in the same place? I think the other option here would be to keep nunit-distribution a separate repo, and add any combined choco packages there - but I believe @rprouse would find releases easier if everything was in the nunit-console repo.

[CharliePoole]

By multiple executable I actually meant multiple apps. For example console plus gui, if that's ever done. For single apps with extensions we're on the same page.

[ChrisMaddock]

Great!

[jnm2]

Looks good.

[CharliePoole]

The following packages are waiting for moderation in the chocolatey gallery:

• Console runner 3.6.1
• GUI runner 0.4
• All five extensions at their latest versions

[CharliePoole]

Gui, Console Runner and NUnit Project Loader extension are approved and listed now.

[CharliePoole]

Visual Studio Project loader is now listed.

BTW, the chocolatey reviewers are being very responsive. It's a matter of my finding time to do some minor fixes and submit them for each extension. At some point, I plan to write down everything I've learned about packaging for chocolatey, which is not as close to nuget as the common format might suggest. 😄

[ChrisMaddock]

All installed - works for me! 😄

[CharliePoole]

@nunit/core-team @NikolayPianikov
The last two extensions (V2 result writer and teamcity) are now available on Chocolatey.

Package names on Chocolatey differ from nuget, by design (ours and Chocolatey's Names and corresponding versions are:

• nunit-console-runner (3.6.1)
• nunit-extension-nunit-project-loader (3.5)
• nunit-extension-vs-project-loader (3.5)
• nunit-extension-nunit-v2-driver (3.6)
• nunit-extension-nunit-v2-result-writer (3.5)
• nunit-extension-teamcity-event-listener (1.0.2)

Example of installing the console runner with the teamcity extension from an administrator prompt:

C:\>choco install nunit-console-runner nunit-extension-teamcity-event-listener

[CharliePoole]

We also have Chocolatey packages for nunit-console, nunit-console.install and nunit-console.portable, created by @Roemer.

This is the package that bundles the console-runner and five extensions together. @Roemer 's implementation does so by wrapping the .msi and .zip packages. It has no ability to add other extensions. We no longer want to do chocolatey that way. Instead we want to have a package that simply depends on the other packages. Unfortunately, this package is already at 3.6.1, so we can't update till there is a 3.7 release. We will have to update the nunit-console package and deprecate the .install and .portable packages. I'll create an issue.

[CharliePoole]

Now that the concept is proven, I'm listing the remaining tasks to get all the chocolatey packaging up to date, with issue references where relevant...

[Reorganized Task List 7/28/2017]

nunit-project-loader

• [x] Create initial chocolatey package
• [x] Integrate chocolatey package with the build script (nunit/nunit-project-loader#8)
• [x] Release package as v3.6

vs-project-loader

• [x] Create initial chocolatey package
• [x] Integrate chocolatey package with the build script (nunit/vs-project-loader#8#)
• [x] Release package as v3.6

nunit-v2-result-writer

• [x] Create initial chocolatey package
• [x] Integrate chocolatey package with the build script (nunit/nunit-v2-result-writer#4)
• [x] Release package as v3.6

nunit-v2-framework-driver

• [x] Create initial chocolatey package
• [x] Integrate chocolatey package with the build script (nunit/nunit-v2-framework-driver#16)
• [x] Release package as v3.7

teamcity-event-listener

• [x] Create initial chocolatey package
• [x] Integrate chocolatey package with the build script (nunit/teamcity-event-listener#41)
• [ ] Release package

nunit-console-runner

• [x] Update chocolatey package to 3.7
• [x] Integrate chocolatey package with the build script
• [x] Release package (either new release or add to 3.7 downloads)

nunit-console

• [x] Create new nunit-console package, integrated with build script, using references to extensions
• [ ] Release package, deprecating nunit-console.install and nunit-console.portable packages
• [ ] Deprecate nunit, nunit.install and nunit.portable packages

[jnm2]

Unfortunately, this package is already at 3.6.1, so we can't update till there is a 3.7 release.

I see people do 3.6.1.20170711 all the time to fix chocolatey package issues.

[CharliePoole]

Good one! I forgot about that.

[ferventcoder]

I see people do 3.6.1.20170711 all the time to fix chocolatey package issues.

@CharliePoole @jnm2 Chocolatey 0.10.5+ implemented a true package release in the version notation. However you can't use it anywhere yet - we will work with vendors and all over the next year to ensure that when we pull it from a dark feature, it will just work everywhere. So instead of 3.6.1.20170711, it would be 3.6.1_2. Again, go with the advice from @jnm2 as this will not work pushed anywhere yet. For more information on the feature, see https://github.com/chocolatey/nuget-chocolatey/issues/11

[CharliePoole]

Hey @ferventcoder ! Thanks for the info. That's a great feature, although we will never again make a mistake that requires re-releasing a package. 😉

[rprouse]

@CharliePoole I have the Chocolatey packages compiled from the 3.7 release available. They are also in the GitHub release if you want to grab them from there. Who has access to upload them? Should I get access and learn so I can get Chocolatey updated in sync with the releases? It was easy to build and package so I don't mind updating a couple more packages.

[CharliePoole]

Did you create your own nuspec files? I haven't added them to the code projects yet and their are quite a few differences.

I have a side project that creates them temporarily based on downloading the nuget files.

I can do this first thing this morning if you like.

[CharliePoole]

It's very early here and I'm not quite awake. Now I think I understand! I guess you created the two chocolatey packages using the build.cake from the nunit-console project. Is that correct? If so, that won't work and I think it would be better to remove them from the release on github for now.

Those two packages have never been submitted to chocolatey.org, gone through validation and verification and been modified to meet their requirements. The packages I'm reporting on here (like the 3.6.1 chocolatey package that is now listed by them) are created in a side project from different code. We need to create packages that using that approved code using updated binaries. I can do that this morning.

A further minor change is that we agreed to use ".choco.nupkg" for the chocolatey packages. Otherwise, users won't know why there are two console runner packages in the release.

As a reminder, my original suggestion was to separate out the creation of the chocolatey packages but the team decided to keep them as part of each project. That's an OK decision, but not yet implemented. I added the "Integrate..." tasks to migrate code from my separate project into each individual project. I haven't worked on the Integrate task for the console yet, because I didn't want to change the build script until the release was complete.

This morning, once I have coffee, I'll use my chocolatey-packages project code to generate chocolatey packages based on your published NuGet package contents and upload to chocolatey. I can replace the uploads in GitHub if you like or leave that to you

[gep13]

@CharliePoole @rprouse @ChrisMaddock as always, please reach out if you have any questions about Chocolatey and/or Cake. Happy to help where I can.

[rprouse]

It's very early here and I'm not quite awake. Now I think I understand! I guess you created the two chocolatey packages using the build.cake from the nunit-console project. Is that correct? If so, that won't work and I think it would be better to remove them from the release on github for now.

Thanks, I haven't been following the chocolatey issues closely enough. Since they were built, I thought they were ready. I will remove the packages from GitHub.

[CharliePoole]

The nunit-console-runner package has been approved and is now on chocolatey.org. My thought would be to rename the nupkg as nunit-console-runner.choco.nupkg so it's clearer what it's for and add it to the release. Does tnat make sense @rprouse?

@gep13 Thanks for the offer! I'm using nunit-project-loader as the model of how to set up our builds so we can create the packages either with the build or separately at a later point when necessary. Feel free to chime in on any PRs!

[gep13]

@CharliePoole will keep an eye out.

[rprouse]

@CharliePoole renaming to *.choco.nupkg makes sense. I had to look in the build output to figure out the differences. I don't think using dashes over dots in the name is enough of a distinction 😺

[CharliePoole]

I'll rename it then. BTW dashes are for chocolatey.org.

[ferventcoder]

All good as long as what you push to Chocolatey community repo doesn't include the *.choco ;)

[CharliePoole]

@ferventcoder I was wondering about that. Any reason? The file type is still .nupkg.

[CharliePoole]

@ferventcoder I was wondering about that. Any reason? The file type is still .nupkg.

[CharliePoole]

I updated the check-list with items for the 3.7 release.

[ferventcoder]

I was wondering about that. Any reason? The file type is still .nupkg.

@CharliePoole nuget pack and choco pack manage the name of the file. I am unsure of what affect managing the file name would produce. As I think about it more, it may not be a good idea to rename the file here either.

[ferventcoder]

What affect changing the file name would produce on deployment to packaging servers (Chocolatey Gallery, ProGet, Nexus, Artifactory, Chocolatey.Server, NuGet.Server, etc), nor am I sure on the affect that it would have on installing the package and managing it with Chocolatey.

And yes, all of those different types of servers will be used by companies caching packages from the community repository.

[jnm2]

Hmm, seems like it would be good to specify that.

[CharliePoole]

With that degree of uncertainty, I guess we should try to solve this in some other way.

Problem 1: Developer confusion in editing the nuspec files.

I suffered from this a lot doing my editing in vscode. Having two nuspec files with the same name is a big confusion, even in separate directories. Using .choco. as part of the name solved that for me and the same .nupkg is produced no matter what you call the .nuspec file. I'll keep this.

Problem 2: User confusion with releases. This is only a problem with the GitHub release files. You get two .nupkg files with very similar-sounding names. GitHub doesn't give you a place to add a file description like we had on some earlier platforms. The first attempt to solve this was by renaming the .nupkg file in the same manner as the .nuspec. That seems like a bad idea based on @ferventcoder 's input, so I see two solutions:

1. Don't put the nuget or chocolatey packages in the GitHub release. This seems impractical, since they are sometimes used when nuget.org or chocolatey.org is not available and they represent a canonical copy of each package.

2. Put some kind of info in the release explaining what each file is for. That would be my preference.

Thoughts?

[jnm2]

Put the output nupkg into a ./choco folder?

[jnm2]

I vote against 1 either way.

[ferventcoder]

I like 2.

[ferventcoder]

Although one could test renaming the nupkg file to see how it ends up at each end point.

[CharliePoole]

@jnm2 Is there a way to structure releases in folders? I haven't been able to find it.

[gep13]

@CharliePoole since you are using Cake to build the nupkg's, you can actually do away with the nuspec file completely, and put all the required information in the NuGetPackSettings and ChocolateyPackSettings classes. This might help with the editing of the packages themselves.

Just a thought...

[jnm2]

@CharliePoole You mean in AppVeyor or Travis, or on your hard drive when you do a release?

[CharliePoole]

@jnm2 I mean in the Release itself on GitHub.

[CharliePoole]

@gep13 That's worth considering.

[jnm2]

@CharliePoole Ah... right. Never mind.

[gep13]

@CharliePoole said...
I mean in the Release itself on GitHub.

I am not aware of this being possible. The only thing I could think would be to bundle all the artifacts into a zip file, and have the folder hierarchy in there.

[CharliePoole]

When we were on Launchpad we were able to have a description of each file. That would be a nice feature.

[rprouse]

We could put all of the Chocolatey nuget packages in one appropriately named Zip file in the GitHub releases.

[CharliePoole]

@gep13 I modified the NuGet build to do everything in the cake file. The only thing I can't specify in the NuGetPackSettings is language. Not sure if we need that so badly, but it's surprising that it's not there.

[gep13]

@CharliePoole oh, interesting. Perhaps that was simply an omission. @devlead can you comment on this? Think you did the initial NuGet integration in this regard.

[devlead]

@gep13 @CharliePoole I'll submit a PR for this today, progress can be tracked in issue cake-build/cake#1699

[CharliePoole]

Super! Meanwhile, I'm just leaving the setting out.

[CharliePoole]

The NUnit project loader extension 3.6 has been released. Others to follow.

[CharliePoole]

Notes on progress...

• I updated the task list in https://github.com/nunit/nunit-console/issues/253#issuecomment-314363156 to be a bit clearer regarding what's done and what remains.

• WRT distinguishing the nuget and chocolatey packages, I took the easy route with nunit-project-loader by simply listing the purpose of each in the release notes on GitHub.

[CharliePoole]

Back from vacation and focusing on this issue and its sub-issues. Short status:

• Four nunit-supported extensions have been updated in master. One is released, the other three are awaiting PR review.

• For the teamcity extension, I'll need to find out from @NikolayPianikov how he wants to handle it.

• Integrating the creation of the nunit-console-runner package requires some restructuring of the script. I'll post something asking for your thoughts.

• The nunit-console package has to wait for all the other components to be done, so that it can reference them.

[CharliePoole]

All extensions except for teamcity have now been updated and re-released.

[NikolayPianikov]

@CharliePoole the TeamCity extension has no any significant updates and I think the release is not required for it, but I am going to test it with the last version of nunit components ASAP

[CharliePoole]

I am about to add an issue to integrate the chocolatey build with it. I think that will qualify as an update, but it's ultimately up to you. I could go ahead and do it just as I did the other four, or you can ask for it to be done differently. Or, ultimately, you could ask for changes in the PR, since it's yours. I do think it makes sense for me to do it since I'm on a roll with four others already under my belt. 😄

See you on the new issue. 😄

[CharliePoole]

@rprouse @ChrisMaddock The nunit-console-runner and nunit-console build work is now merged in #275.

Once the last remaining extension (teamcity) is ready, we will have to decide about releasing the two console packages. Do we wait for 3.8? Do we try to get something out now?

Finally, in the future, we need a process for dealing with new extension releases. Should a new extension release trigger a new release of the nunit-console-with-extensions and NUnit.Console packages? What tests do we do first, if any. How do we deal with differences between versions in the nuget, chocolatey and msi packages?

[rprouse]

Once the last remaining extension (teamcity) is ready, we will have to decide about releasing the two console packages. Do we wait for 3.8? Do we try to get something out now?

I am trying to stagger the Framework and Console releases to make management of the two easier and to drive home the fact that they are on a different release cadence. I am going to do a Framework release this month and there is only one fixed issue in the Console/Engine, so the 3.8 Console release is probably a ways out. Based on that, you might want to do a Chocolatey release with 3.7.

Finally, in the future, we need a process for dealing with new extension releases. Should a new extension release trigger a new release of the nunit-console-with-extensions and NUnit.Console packages? What tests do we do first, if any. How do we deal with differences between versions in the nuget, chocolatey and msi packages?

We might want to pull this off into an other issue, but my initial thoughts are that nuget, chocolatey and MSI should match as much as possible and be released at the same time. If you do a 3.7 chocolatey release then we might want to make an exception this one time because everything is newer and it is being released so much later.

As for the versioning and who tests. My feeling is that it should be the responsibility of the extension author to test their extension then submit a PR to this repo to update the version. The MSI pulls in the latest versions from NuGet and always uses those for the release. We could do the same for Chocolatey and the NuGet package. Doing so would signal to the extension authors that they should only be releasing tested versions and that it is their responsibility. That said, except for TeamCity, we are the authors, but I think we should setup the model as we want it to be.

My concern with leaving it to the Console/Engine team to test the extensions is that it adds a great deal of time and testing to the release process. Before release, we will need to go out and find each of the extensions, check their latest versions, update our code, test the functionality of the extension. I think that the extension authors are best suited to test the extensions.

[CharliePoole]

OK, I'll wait till the teamcity extension is ready and think about what to do. We probably need user documentation about extensions somewhere, telling them how to upgrade to a new extension version between console package releases. It's easy to do with nuget and chocolatey, harder if you are using the msi release.

For the longer term, I agree with you. I think a logical separation between the extensions and the engine makes sense. I'd like to become the maintainer of a few of them myself and it would help to have clear expectations. That is probably a separate issue, as you say, but maintainers of extensions should test against any version of the runner that they support and we should be able to rely on their having tested. I don't think we need a formal verification program since we would not be bundling any extensions from someone we didn't trust.

[ChrisMaddock]

I agree with what's said here. Testing to be the responsibility on the extension author - and we can deal with any poorly functioning extensions we are bundling as and when that situation arises.

[CharliePoole]

Status to date:

• All extensions are done
• All extensions have been released in new versions except for teamcity. @NikolayPianikov will decide whether to do a 1.0.3 or 1.1 release or to hold off and stick with 1.0.2 for now.
• NUnit console packages, both with and without extensions are done. I'll work next on releasing them.
• Once all is done, we still have some old packages to mark as obsolete.

[NikolayPianikov]

@CharliePoole I've released a new version of TeamCity extension 1.0.4, it has several fixes related to NUnit 2 tests compatibility. Also I've created and tested a choco package. Is it possible to add it to the Chocolatey repo?

[CharliePoole]

I could upload it if course, but it makes more sense for you to be able to do it. What's your chocolatey id? I'll add it.

[NikolayPianikov]

Yes, here it is: NikolayP

[CharliePoole]

I added you and it says "Pending Approval." I don't know what that means. Maybe they will send you an email to confirm the addition.

You should add your name in the chocolatey nuspec package while you think about it. It's OK if it's not in this release, but it should be in the master.

[NikolayPianikov]

@CharliePoole Everything worked out. Thank you!

[ChrisMaddock]

I believe we're tying up the remaining choco work on https://github.com/nunit/nunit/issues/452, so I'm going to close this issue.