Home: Skip Duplicate switch added to nuget.exe push command

[maartenba]

_From @lukemcgregor on October 22, 2015 21:42_

Currently if the package is already published the following occurs

Pushing Pkg 1.0.0 to the NuGet gallery (https://www.nuget.org)...
Failed to process request. 'A package with id 'Pkg' and version '1.0.0' already exists > and cannot be modified.'.
The remote server returned an error: (409) Conflict..

And a non-zero return code is returned from nuget push

I would like to add another switch to pack so that the following occurs

nuget push Pkg -SkipIfAlreadyPublished

A package with id 'Pkg' and version '1.0.0' already exists, skipping...

A zero return code is returned.

This would allow for better integration with automated build tools which produce nuget packages.

_Copied from original issue: NuGet/NuGetGallery#2743_

[maartenba]

_From @lukemcgregor on October 22, 2015 21:54_

To do this I want to edit https://github.com/NuGet/NuGet.PackageManagement/blob/dev/src/NuGet.CommandLine/Commands/PushCommand.cs But I cant work out how to build the project.

Ive tried using home (http://docs.nuget.org/Contribute/Setting-up-the-NuGet-Development-Environment) But the nuget3 solution doesn't appear to build the CLI.

Building PackageManagement directly seems to fail as it refers to a bunch of nuget packages which dont exist yet.

[yishaigalatzer]

@lukemcgregor the code lives in nuget.client.

Before you send a pull request, we should have a discussion first, and agree on a design and approach.

I have a few questions:

1. Why are you pushing directly from your CI to nuget.org, this is a scenario we do not want to support. We can't force people to do it, but CI builds show go to public private feeds and not to nuget.org. nuget.org is intended for stable or stable pre-releases of a product and not for nightly/ci builds. There is not retention policy, or facilities to support that, nor we plan to add them.
2. Why would your CI generate two builds of the same version, that seems like a pretty bad idea, unless of course you are trying to rebuild an RTM build (before publishing it to nuget).

So I'm not so sure this is something we want to add as a top level support.

Perhaps the right alternative is to support nuget.exe list -version {version} (today we only support -allversions) and then you can script your way happily with the result, but not encourage this behavior.

Thoughts?

[lukemcgregor]

My intent is not to publish every build that goes through CI, I totally
agree thats the wrong way to do it, it creates a huge mess of versions and
ruins everyones day. I think we are agreed here

What I want to do is to remove some complexity around deploying to nuget.
What I have found is that the decision to create a package is the same as
the decision to create a new version. So when you increment a version
number of a nuget package in code you are actually deciding to deploy that
package. My aim is to automate the deploy bit when the version is
incremented.

I have written some stuff to query the nuget apis and precheck before
running push but its a bit messy and it feels like it would be way tidier
built into push as a skip over this package if no one has changed the
version.
On 24/10/2015 11:57 am, "Yishai Galatzer" [email protected] wrote:

@lukemcgregor https://github.com/lukemcgregor the code lives in
nuget.client.

Before you send a pull request, we should have a discussion first, and
agree on a design and approach.

I have a few questions:

1. Why are you pushing directly from your CI to nuget.org, this is a
   scenario we do not want to support. We can't force people to do it, but CI
   builds show go to public private feeds and not to nuget.org. nuget.org is
   intended for stable or stable pre-releases of a product and not for
   nightly/ci builds. There is not retention policy, or facilities to support
   that, nor we plan to add them.
2. Why would your CI generate two builds of the same version, that seems
   like a pretty bad idea, unless of course you are trying to rebuild an RTM
   build (before publishing it to nuget).

So I'm not so sure this is something we want to add as a top level
support.

Perhaps the right alternative is to support nuget.exe list -version
{version} (today we only support -allversions) and then you can script your
way happily with the result, but not encourage this behavior.

Thoughts?

—
Reply to this email directly or view it on GitHub
https://github.com/NuGet/Home/issues/1630#issuecomment-150711110.

[yishaigalatzer]

What do you think about the list option? Then your script is as simple as

List

If errorlevel == 0 (package exists)
{
Skip
}
Else
{
Publish
}

BTW on our team we ALWAYS increment the version on every single build, makes it a lot easier to track issues even with dev builds.

From: Luke McGregor [mailto:[email protected]]
Sent: Friday, October 23, 2015 4:34 PM
To: NuGet/Home [email protected]
Cc: Yishai Galatzer [email protected]
Subject: Re: [Home] Commandline switch on nuget push to skip if already published (#1630)

My intent is not to publish every build that goes through CI, I totally
agree thats the wrong way to do it, it creates a huge mess of versions and
ruins everyones day. I think we are agreed here

What I want to do is to remove some complexity around deploying to nuget.
What I have found is that the decision to create a package is the same as
the decision to create a new version. So when you increment a version
number of a nuget package in code you are actually deciding to deploy that
package. My aim is to automate the deploy bit when the version is
incremented.

I have written some stuff to query the nuget apis and precheck before
running push but its a bit messy and it feels like it would be way tidier
built into push as a skip over this package if no one has changed the
version.
On 24/10/2015 11:57 am, "Yishai Galatzer" <[email protected][e-mail]> wrote:

@lukemcgregor https://github.com/lukemcgregor the code lives in
nuget.client.

Before you send a pull request, we should have a discussion first, and
agree on a design and approach.

I have a few questions:

1. Why are you pushing directly from your CI to nuget.org, this is a
   scenario we do not want to support. We can't force people to do it, but CI
   builds show go to public private feeds and not to nuget.org. nuget.org is
   intended for stable or stable pre-releases of a product and not for
   nightly/ci builds. There is not retention policy, or facilities to support
   that, nor we plan to add them.
2. Why would your CI generate two builds of the same version, that seems
   like a pretty bad idea, unless of course you are trying to rebuild an RTM
   build (before publishing it to nuget).

So I'm not so sure this is something we want to add as a top level
support.

Perhaps the right alternative is to support nuget.exe list -version
{version} (today we only support -allversions) and then you can script your
way happily with the result, but not encourage this behavior.

Thoughts?

—
Reply to this email directly or view it on GitHub
https://github.com/NuGet/Home/issues/1630#issuecomment-150711110.

—
Reply to this email directly or view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fgithub.com%2fNuGet%2fHome%2fissues%2f1630%23issuecomment-150716813&data=01%7c01%7cyigalatz%40microsoft.com%7cccdb46fdb4b9481e4f1b08d2dc026269%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=I8%2bBO4AVjYt9ZuhyzJlooMYTsRSeoBbz7zdjjR8B%2bHQ%3d.

[lukemcgregor]

I have noticed that quite a few of the MS teams used to use build numbers in their nuget packages. This isn't strictly semver.

I think if you do want to track explicit builds (which we don't care too much about) then you need to separate that from your semver completely.

Perhaps a good method would be to use the AssemblyInformationalVersion or a specific nuget version to do the correct Semver version.

eg:

[assembly: AssemblyVersion("MAJOR.MINOR.PATCH.BUILD")]
[assembly: AssemblyFileVersion("MAJOR.MINOR.PATCH.BUILD")]
[assembly: AssemblyInformationalVersion("MAJOR.MINOR.PATCH")]

This would let you look at the DLL deets to see the build but not propogate that to the actual package.

Personally though the only relevant referable version for software I write is one with a distinct semver number.

[vijayganeshpk]

+1 for this feature. Very helpful when trying to migrate from shared folder based nuget source to internal server. Issue occurs when the push is interrupted due to network issue and we have to retry the push again.

[darthscsi]

+1 for dealing with many packages. I have 100 packages and if a timeout happens, I can no longer use wildcards to upload the remaining. I have to try each one individually or generate a list of uploaded packages and exclude them from my set.

[csalgo70]

I wrote a power shell script to push it to our internal server.

param([String]$packageName,[String]$apiKey, [String]$source)

$latestrelease = ./nuget.exe list $packageName -source $source | Select-String -Pattern $packageName

if($latestrelease -eq $null)
{
 $srvpkg = $null
}
else 
{
 $srvpkg = $latestrelease.Line.split(" ")[0] + "." + $latestrelease.Line.split(" ")[1] + ".nupkg"
}

#set the path here to wherever you have your *.nupkg created from the current build.
$newpkg=get-childitem -path ..\ | where {$_.Name -Match $packageName + "\.\d" } | where {$_.Name -NotMatch "symbols.nupkg" }

if($srvpkg -eq $newpkg.Name)
{
  Write-Host("Skip pushing " + $newpkg + " as it already exist on server")
}
else 
{
  ./nuget.exe push $newpkg.FullName -source $source -ApiKey $apiKey -configfile nuget.config
  Write-Host("Pushed " + $newpkg.Name)
}

[dustincleveland]

+1 I have a framework of libraries that are contained in one solution. I would like to publish NuGet packages to our private server whenever the solution is merged into our master branch. Only the NuGet packages with a newer version should be deployed. Right now, I will either have to swallow the return code, or implement a Powershell script instead of using built-in VSTS tasks.

Swallowing the return code is a bad option because I will also be swallowing errors not related to duplicate packages.

Writing a Powershell script sucks because I will have to learn Powershell, then implement a solution and maintain that solution. This all costs my client lots of money.

I understand @yishaigalatzer's concern that this may promote some bad behavior on nuget.org, but there are certainly legitimate use cases. It would be nice if a good compromise could be found.

EDIT: I have since discovered the VSTS actually exposes a feature that allows you to skip duplicate packages, FTW.

[lukemcgregor]

Just on a super irrelevant side note. I was happy to see that appveyor does all of this for you with magic when you publish a nuget package via their stuff. That being said I still want to be able to do this much more easily via scripts.

[abatishchev]

VSTS uses internal API to handle this scenario. Any chance it'll be implemented NuGet-wide?

[mbedded]

Hello together,

i would love to see such a flag as well. Currently i had have the issue that the NuGet-Plugin for Jenkins marks a build as "failed" if a duplicated push causes an error (406 as you said above).

Just as an info: I created a small console-project (.NET) which verifies if a package is existing before a push will be done. https://github.com/mbedded/NugetWrapper
If a package is not existing nuget.exe push.. will be called.

I am testing this since a few days with Jenkins and till now it works fine. Please take care that it is optimised for Jenkins NuGet-Plugin. So the error-handling is minimal and a fixed number of arguments is expected. I am no big fan of workarounds because i would expect a "soft push" like "ignore duplicates" as well..
You can give it a try or fork the code so it fits your needs (or create issues and PRs) so other people can use it for now as well. I saw it at VSTS and Appveyor too, that they don't mark a build as failed if the package was already pushed.

Just my 2 cents :)

[infin8x]

@anangaur any thoughts on the prioritization of this? We have an ask on the Azure Artifacts backlog in Q2 to consider extending vstsnugetpush (which provides our "allow duplicates to be skipped" option) to x-plat and on-prem. I'd rather push towards an official nuget feature if it's available :).

[anangaur]

Makes sense. We will triage it next week.

[kilasuit]

Honestly this feels like fixing something that wasn't overly broken (though I may just be lucky and have never experienced it/a need for it)

surely this is something that should be implemented on the serverside to decide to allow/deny either a skip/republish option and therefore should be implemented within serverside code not the client utility?? especially on the serverside you could then implement ruling that would give opportunity for specific users to be allowed to republish / enforce all other users to skip (even if they are the package owner) and even be able to make this user configurable via serverside tickboxes

Thats my thoughts, which could easily be seen as completely wrong but I thought I'd throw this into the mix before a final decision is made

[anangaur]

When you are trying to push multiple package *.nupkg and do not want to be blocked when one of the pushes failed when the package is already present on the server, this option can come in handy. Especially for CI/CD scenarios when you are pushing intermediary bits to a private feed.

Example: You publish a bunch of packages (say 10) to a private feed that does not allow overwriting packages as part of a build/release pipeline. One of the package in the bunch is malformed and hence after publishing 5 packages threw an error. So the push fails. Now you fix it and re-run the release. It should just skip (and not error) while pushing the 5 packages and then push rest of the 5 packages successfully. Note that servers may or may not allow you to overwrite and this option is to handle that situation.

P.S.: I agree there are multiple ways to solve this issue on the server side or the client side or both. This is one of the ways.

[donnie-msft]

As I see it, the server already does implement its portion of this by returning a specific error code for a duplicate. It's up to the client to decide if it cares or not. In the case of NuGet client, it always interprets it as a fatal error. The new option would allow you to customize NuGet client to say "it's okay, keep going". The server doesn't (and shouldn't) care whether you want to send further requests or not.

[infin8x]

Thanks for taking this up! @donnie-msft that's right. We wrote our own custom client for Azure Pipelines to do just that. It's great to see it being taken up as a native NuGet feature.