Azure-pipelines-tasks: NuGet Publisher shouldn't abort on first conflict (409)

[thorbenw]

There are situation where I'd like to make NuGet Publisher ignore conflicts, i.e. recognize the package is already up to date and continue publishing in case there are multiple packages found.

1511 introduced a workaround, but this doesn't add that "publish if there's anything new" capability I'm looking for.

[zjrunner]

This is something we'll consider, but today we can't reliably get this information out of nuget to know the difference between an ok failure (409) and a fatal attempt. We have potential future plans to change how we push packages to allow more control here, but not in the Dev15 timeline.

[abatishchev]

+1. An ability to suppress 409 Conflict only and fail on others is very needed!

[zjrunner]

I've captured the request internally in our feature board (683673) to consider for the future.

[abatishchev]

Hey Zack, can you please send me the link to alexbat@?

[zjrunner]

I'll just post it here - nothing to hide, just not accessible to most people:
Feature 683673: Improve NugetPublisher Task
https://mseng.visualstudio.com/VSOnline/_workitems/edit/683673

(gave it a generic name so we could roll up several items in a feature and then expand them out when we take them on. Helps ensure things don't get lost in a flooded backlog over time)

[Zazcallabah]

Instead of suppressing the 409, could the build task first check if the current version of the package has already been published - and only attempt publish if not?

also, if you close these issues, there is no way for users (i.e. me) to get a notification when they have been fixed?

[abatishchev]

hi @zjrunner, do you know what is current status? the link above doesn't work anymore, at least to me.

[infin8x]

@abatishchev we're spec'ing these improvements now, hoping to start on them next sprint. Feature 564185: NuGet producer build tasks v2 https://mseng.visualstudio.com/VSOnline/_workitems/edit/564185?fullScreen=false

[infin8x]

We're adding a new option to this task called "Allow duplicates to be skipped" that should address this issue. I'm reopening this issue for now to allow for notifications when we ship the fix.

[michelkamp]

Hi ,

it would be great to have this option "Allow duplicates to be skipped" implemented. I have 1 solution containing several packages. i want to build this with 1 build plan. Now the Devs have to update all package numbers if they only have to change one package. So not a good way do doing.

This here any workaround for this now ? Should i use the nuget packager and apply only the project files / nuget. The issue in that case is that i build dot net core projects and this step looks like not working with dot net core.

Any help would be great.
thanks,
michel

[abatishchev]

The work on new version is in progress to my knowledge. For now a workaround would be either writing own PowerShell script to publish which won't fail on conflict, or create publish package step for each package independently and make them to continue on error. This will make your build orange but it's better than updating all versions each time.

[infin8x]

@michelkamp we're taking the "Allow duplicates to be skipped" work this sprint. Should be available in about a month.

[infin8x]

This is fixed by #4397.

[michelkamp]

Is it fix active already ???

Is still get an error :

==============================================================================
Task : NuGet Publisher
Description : Uploads nupkg files to a NuGet server
Version : 0.2.34
Author : Lawrence Gripper

Help : More Information

C:Windowssystem32chcp.com 65001
Active code page: 65001
Detected NuGet version 3.5.0.1938 / 3.5.0
SYSTEMVSSCONNECTION exists true
d:a_tasksNuGetPublisher_xxxxxxxxx0.2.34node_modulesnuget-task-commonNuGet3.5.0NuGet.exe push -NonInteractive "d:a3sPortalMainxxxxxxx.PortalPortal.AzureBilling.Clientbinany cpureleasexxxxxxxx.Client.1.0.70.nupkg" -Source https://xxxxxxx.pkgs.visualstudio.com/_packaging/xxxxxx-Packages-Private/nuget/v3/index.json -ApiKey VSTS
Pushing xxxxxxxx.Client.1.0.70.nupkg to 'https://xxxxxx.pkgs.visualstudio.com/_packaging/xxxxxxxxx/nuget/v2/'...
PUT https://xxxxx.pkgs.visualstudio.com/_packaging/xxxxxxxx/nuget/v2/
Conflict https://xxxxxx.pkgs.visualstudio.com/_packaging/xxxxxxxxx/nuget/v2/ 1001ms
Response status code does not indicate success: 409 (Conflict).
Error: d:a_tasksNuGetPublisher_xxxxxxxx0.2.34node_modulesnuget-task-commonNuGet3.5.0NuGet.exe failed with return code: 1
Packages failed to publish

Michel

From: Alex Mullans [mailto:[email protected]]
Sent: Wednesday, May 31, 2017 9:33 PM
To: Microsoft/vsts-tasks [e-mail]
Cc: michelkamp [e-mail]; Mention [e-mail]
Subject: Re: [Microsoft/vsts-tasks] NuGet Publisher shouldn't abort on first conflict (409) (#2119)

This is fixed by #4397https://github.com/Microsoft/vsts-tasks/pull/4397.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/Microsoft/vsts-tasks/issues/2119#issuecomment-305292986, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AGe6ssWjD4jJvyelYKfAN6aT3tKUiTssks5r_cBfgaJpZM4JSXap.

[infin8x]

It is not. We need to finish up the linked PR and get it deployed, which will happen in 3 weeks.

[elitas]

@alexmullans : Three weeks has passed, when is the deploy due?

[infin8x]

Will be finished tomorrow (Thursday).

[Abdruggi]

Any schedule when it will be available for tfs on premise?

[michelkamp]

I can confirm that fixed already !

Get Outlook for iOShttps://aka.ms/o0ukef

---

From: Abdruggi [e-mail]
Sent: Thursday, July 13, 2017 3:17:23 PM
To: Microsoft/vsts-tasks
Cc: michelkamp; Mention
Subject: Re: [Microsoft/vsts-tasks] NuGet Publisher shouldn't abort on first conflict (409) (#2119)

Any schedule when it will be available for tfs on premise?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/Microsoft/vsts-tasks/issues/2119#issuecomment-315074759, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AGe6slJ5YSWOsxKZ7w-ts7N7tyQ3y7Uqks5sNhjjgaJpZM4JSXap.

[infin8x]

Should land in TFS 2018.

[sandorfr]

The Allow duplicates to be skipped option is missing on External Nuget Server. I think this is needed there as well ;)

[infin8x]

Unfortunately, that feature relies on a custom NuGet client and APIs that are only available in Visual Studio Team Services, so we can't enable it for external NuGet servers.

[abatishchev]

It's impossible to catch HTTP exceptions and swallow those which have status 409 Conflict? I understand that internally it's more complex but in the end or the day that's what it is, isn't it?

[sandorfr]

If what @abatishchev is suggesting is not possible, maybe having the task skipping symbols automatically would be great.

[zjrunner]

@abatishchev , when using nuget.exe your response codes have historically been only 0 or 1. It was success or failure. In this case, ignoring a conflict would also ignore invalid uploads and other types of connectivity errors. To go beyond that you need to get into verbose output and log scraping which generally aren't reliable, especially across user-configurable nuget versions. Since we own the VSTS Server implementation for nuget, we can create a more direct-line to do what you're talking about and add some other efficiencies, but for other host servers we go through nuget.exe as the standard.

[abatishchev]

I see. Should we/you then go to the NuGet team and ask to expose HTTP statuses properly?

[zjrunner]

Yes, but it has been an open request in nuget since 2015 (https://github.com/NuGet/Home/issues/1630). The last workable suggestion in that thread was to list, then publish. That still is exposed to race failures. (to be fair the issue is asking for a solution, not simply better error codes. It is clear most of the folks in the thread would be fine scripting around it so you could add to it and try to clarify/redirect also)

[infin8x]

@sandorfr great feedback, we used to exclude those, and I think we could add that back to the default. For existing builds, you can add ;!$(Build.ArtifactStagingDirectory)/*.symbols.nupkg to your publish path to exclude symbols packages.

[doorman02]

Hi @alexmullans what´s the status, is there a way to skip duplicates using the nuget command line tool? I have TFS2018 installed. Thanks

[infin8x]

@doorman02 there is not, unfortunately. The VSTS build task uses a VSTS-specific tool to push packages that can understand and react to the 409.

[michelkamp]

if you use the new NuGet push build step you have a option "Allow duplicates to be skipped" described as : If you continually publish a set of packages and only change the version number of the subset of packages that changed, use this option. It allows the task to report success even if some of your packages are rejected with 409 Conflict errors.

This option is currently only available on Visual Studio Team Services and using Windows agents. If NuGet.exe encounters a conflict, the task will fail.

[snelljef]

@michelkamp Is there a timeline or workaround for TFS On-premise users?

[josh-degraw]

@michelkamp I am using TFS On-premise as well and this feature would be incredibly useful.

[mlooise]

@michelkamp Could this feature also be made available for the on-prem users? Maybe with a marketplace plugin? I'm also searching for a workaround. At the moment i'm using continue on error but always get release failures using this approach

[infin8x]

This feature is also available on-premises as of TFS 2018.

[snelljef]

@alexmullans It is? Unless it's misconfigured on my end I have to disagree. I am running TFS 2018.3 and it explicitly warns that it does not work with on-premise installs with the following text:

This option is currently only available on Visual Studio Team Services and using Windows agents. If NuGet.exe encounters a conflict, the task will fail.

I am using the NuGet task version 2.*

[infin8x]

@snelljef Right you are, my apologies. I thought we'd shipped that on-premises at some point. I've put an item on our backlog to consider when we do planning for Q1.

[chouhanaditya]

Hi @alexmullans,

I am getting same error - 'error: Response status code does not indicate success: 409 (Conflict - The feed already contains package'. I am using azure pipeline YAML task _DotNetCoreCLI@2- Pack and Publish_ for publishing nuget packages. The flag 'Allow duplicates to be skipped' or property ' allowPackageConflicts: true' doesn't seem to be working for me since I am using Ubuntu build agent. Can you please asssist me

[zjrunner]

This won't work outside of windows agents, it is making use of a custom uploader instead of nuget.exe that happened to be written in .net framework at the time. Your best bet at this point is to follow the earlier suggestions of doing a list, then the publish as needed. It looks like the issue mentioned earlier in this thread is still open (https://github.com/NuGet/Home/issues/1630), but it references some other efforts in just the last week attempting to make a fix in nuget to support this fully.

[RehanSaeed]

Any plans to make allowPackageConflicts: true work on Ubuntu or Mac? Is the only option to switch to Windows or loop over packages to call this task?

[zjrunner]

no, the custom tool beneath it won't become xplat. we're moving in a direction of encouraging the use of our auth-only task and then using raw nuget/msbuild/dotnet in batch tasks or YAML build for a cleaner flow where we're less in the way. nuget had a couple issues open at one time about directly supporting return codes so you could differentiate on the client between a failed upload and a conflict upload, I'd recommend finding and taking part in those threads to help drive a more native solution