Azure-pipelines-tasks: Nuget push to feed will show error "503 (Service Unavailable)"

[igoraresti]

Hello,

since 1st of September 2020 we are facing nuget push upload error on every try in certain package in our on-premise agent which is pushing/uploading to Azure DevOps services project feed.

Most of our packages are quite light with few depencies and always have worked fine the nuget feed updating.

There is just one large package. This package nuget push task is in separate task from the other packages and always has the same error after 1 minute and 38 seconds. The nuget push command is set with timeout 0 (infinite). The nuget package size is about 139MB.

Error logs

NuGet Version: 4.1.0.2450
System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 503 (Service Unavailable).
at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
at NuGet.Protocol.Core.Types.PackageUpdateResource.<>c.b__21_3(HttpResponseMessage response)
CredentialProvider.TeamBuild: "C:\vsts-agent_work_tasks\NuGetCommand_333b11bd-d341-40d9-afcf-b32d5ce6f23b\2.174.0\CredentialProvider\CredentialProvider.TeamBuild.exe" -uri https://sisteplant-alm.pkgs.visualstudio.com/2c4e2c04-c0b1-4990-b8d0-36382db181eb/_packaging/fd06961a-5e18-7449-9328-b7065f0faf70/nuget/v3/index.json -nonInteractive -verbosity detailed
CredentialProvider.TeamBuild: URI Prefixes:
at NuGet.Protocol.HttpSource.d__14`1.MoveNext()
CredentialProvider.TeamBuild: https://dev.azure.com/sisteplant-alm/
--- End of stack trace from previous location where exception was thrown ---
CredentialProvider.TeamBuild: https://sisteplant-alm.pkgs.visualstudio.com/
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.Protocol.Core.Types.PackageUpdateResource.d__21.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.Protocol.Core.Types.PackageUpdateResource.d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at NuGet.Protocol.Core.Types.PackageUpdateResource.d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.Protocol.Core.Types.PackageUpdateResource.d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.Commands.PushRunner.d__0.MoveNext()

[edgarrs]

@igoraresti , could try using the Nuget Auth task?
Note that you will need to update the version of Nuget to 4.8.5385 or higher.
If the problem persist after changing your pipeline to use the Nuget Auth task, please enable debugging in your pipelines (system.debug=true) and add the parameter -verbosity detailed to the nuget call to get more information about the failure.

[igoraresti]

We will try this week the proposed solution. For the moment, I have splitted up the package into smaller packages and now its working.

[sidiandi]

I can confirm this issue with NuGet Version: 5.4.0.6315

Uploading a 32 MB sized nuget package works. When trying to upload a 64 MB large packages, nuget fails after 3 retries each failing after 30 seconds:

2020-10-12T08:28:37.9867220+02:00|INF|process 11516 started: nuget.exe push C:\Users\---HIDDEN---\AppData\Local\Temp\0g3uz213.inu\upload-test-6e196556-d299-4dfa-9e34-ccf471fc9329_67108864B.1.0.0.nupkg -Source https://pkgs.dev.azure.com/--HIDDEN--/_packaging/csa-test/nuget/v3/index.json -ApiKey dummy -Timeout 3600 11516:Pushing upload-test-6e196556-d299-4dfa-9e34-ccf471fc9329_67108864B.1.0.0.nupkg to 'https://pkgs.dev.azure.com/---HIDDEN---/_packaging/55314db2-5402-4b7e-a265-f75bf28afa90/nuget/v2/'... 11516: PUT https://pkgs.dev.azure.com/---HIDDEN---/_packaging/55314db2-5402-4b7e-a265-f75bf28afa90/nuget/v2/ 11516: ServiceUnavailable https://pkgs.dev.azure.com/---HIDDEN---/_packaging/55314db2-5402-4b7e-a265-f75bf28afa90/nuget/v2/ 30528ms 11516: PUT https://pkgs.dev.azure.com/---HIDDEN---/_packaging/55314db2-5402-4b7e-a265-f75bf28afa90/nuget/v2/ 11516: ServiceUnavailable https://pkgs.dev.azure.com/---HIDDEN---/_packaging/55314db2-5402-4b7e-a265-f75bf28afa90/nuget/v2/ 30127ms 11516: PUT https://pkgs.dev.azure.com/---HIDDEN---/_packaging/55314db2-5402-4b7e-a265-f75bf28afa90/nuget/v2/ 11516: ServiceUnavailable https://pkgs.dev.azure.com/---HIDDEN---/_packaging/55314db2-5402-4b7e-a265-f75bf28afa90/nuget/v2/ 30433ms 11516:Response status code does not indicate success: 503 (Service Unavailable). 2020-10-12T08:30:10.6015319+02:00|INF|process 11516 exited with 1: nuget.exe push C:\Users\---HIDDEN---\AppData\Local\Temp\0g3uz213.inu\upload-test-6e196556-d299-4dfa-9e34-ccf471fc9329_67108864B.1.0.0.nupkg -Source https://pkgs.dev.azure.com/---HIDDEN---/_packaging/csa-test/nuget/v3/index.json -ApiKey dummy -Timeout 3600 2

[Veek1]

Hope this could help...

I had the same issue with a 451 MB sized nuget package. After 3 tries I got Error: _Reponse status 503 – service unavailable_

Looking online I discovered that it seems to be a network issue (and in some cases proxy issues).
In order to solve the issue, I uploaded the file to my Azure hosted VM and tried the push from there… it worked.

[phil-hodgson]

Azure Artifacts has adjusted some limits to help address long-running publish operations. Please let us know if you're continuing to see persistent 503 responses on a push/publish for larger packages.

[MZade]

Dear, We try to upload a package and also get the HTTP 503 error as followed.
The package is about 500 MB (microsoft.dynamics.ax.application.devalm.buildxpp.10.0.644.10018.nupkg).
Could you please help us on this one as it is a really blocking point.

C:\Temp\Temp>nuget.exe push -Source "Unified_Operations_Platform_Build_Artifacts" -ApiKey az C:\Temp\Temp\Packages\microsoft.dynamics.ax.application.devalm.buildxpp.10.0.644.10018.nupkg -Timeout 60000 -Verbosity detailed
NuGet Version: 5.8.0.6930
Pushing microsoft.dynamics.ax.application.devalm.buildxpp.10.0.644.10018.nupkg to 'https://XXXXXXXXXXXXXX.pkgs.visualstudio.com/ca07cda5-17f0-457d-a990-519d472ffff4/_packaging/8b7d901e-735e-4504-aa0a-abb64df82e9c/nuget/v2/'...
PUT https://XXXXXXXXXXXXXX.pkgs.visualstudio.com/ca07cda5-17f0-457d-a990-519d472ffff4/_packaging/8b7d901e-735e-4504-aa0a-abb64df82e9c/nuget/v2/
ServiceUnavailable https://XXXXXXXXXXXXXX.pkgs.visualstudio.com/ca07cda5-17f0-457d-a990-519d472ffff4/_packaging/8b7d901e-735e-4504-aa0a-abb64df82e9c/nuget/v2/ 159065ms
PUT https://XXXXXXXXXXXXXX.pkgs.visualstudio.com/ca07cda5-17f0-457d-a990-519d472ffff4/_packaging/8b7d901e-735e-4504-aa0a-abb64df82e9c/nuget/v2/
ServiceUnavailable https://XXXXXXXXXXXXXX.pkgs.visualstudio.com/ca07cda5-17f0-457d-a990-519d472ffff4/_packaging/8b7d901e-735e-4504-aa0a-abb64df82e9c/nuget/v2/ 149801ms
PUT https://XXXXXXXXXXXXXX.pkgs.visualstudio.com/ca07cda5-17f0-457d-a990-519d472ffff4/_packaging/8b7d901e-735e-4504-aa0a-abb64df82e9c/nuget/v2/
ServiceUnavailable https://XXXXXXXXXXXXXX.pkgs.visualstudio.com/ca07cda5-17f0-457d-a990-519d472ffff4/_packaging/8b7d901e-735e-4504-aa0a-abb64df82e9c/nuget/v2/ 141056ms
Response status code does not indicate success: 503 (Service Unavailable).
System.Net.Http.HttpRequestException: Response status code does not indicate success: 503 (Service Unavailable).
at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
at NuGet.Protocol.Core.Types.PackageUpdateResource.EnsureSuccessStatusCode(HttpResponseMessage response, Nullable1 codeNotToThrow, ILogger logger) at NuGet.Protocol.Core.Types.PackageUpdateResource.<>c__DisplayClass24_0.<PushPackageToServer>b__0(HttpResponseMessage response) at NuGet.Protocol.HttpSource.<ProcessResponseAsync>d__191.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.Protocol.Core.Types.PackageUpdateResource.d__24.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.Protocol.Core.Types.PackageUpdateResource.d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at NuGet.Protocol.Core.Types.PackageUpdateResource.d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.Protocol.Core.Types.PackageUpdateResource.d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.Protocol.Core.Types.PackageUpdateResource.d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.Commands.PushRunner.d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.CommandLine.PushCommand.d__36.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.CommandLine.Command.Execute()
at NuGet.CommandLine.Program.MainCore(String workingDirectory, String[] args)

[MZade]

@phil-hodgson, could you please reopen this topic for further improvement as the HTTP 503 error during the upload is block8bg us from using some certain nuget packages which are crucial for our build process. Thanks in advance!