Azure-pipelines-agent: NuGet restore is very slow on self-hosted agent run as service

[satano]

Agent Version and Platform

Current agent version: '2.154.3'
OS: Windows
Platform: x64

Azure DevOps Type and Version

dev.azure.com, just build agent is self-hosted.
https://dev.azure.com/krossk/

What's not working?

NuGet restore task is very slow. I have already tried Developer Community. So the setup is this:

• We use latest nuget.exe, now it is the version 5.1.0.
• We use custom nuget.config where there are two nuget sources: standard _nuget.org_ and our own nuget server. Our nuget server is not private. It is public, so from the agent view, it is the same as nuget.org, just different URL.
• Solution has 7 projects.
• The total number of restored packages is 81. Only 4 of them are hosted on our own nuget server, the rest are packages from nuget.org.
• Total size of restored packages is around 830 MB.
• The build runs on our self-hosted machine/agent.
• Our build agent(s) run as a service.
• The machine is not bad: Intel Xeon, 64GB RAM, SSD disk. And this is our build machine - there is no other software running.

NuGet restore task always takes around 11 minutes (usually between 10 and 13). And all the packages are in the local NuGet cache, they are not downloaded from the internet. I tried standard _NuGetCommand@2_ task and also simple script step running nuget.exe directly - this makes no difference.

At the Developer Community we were suggested to try changing the mode the agent is running. And this is the thing. When the agent is started manually (interactively), everything is OK. NuGet restore takes 15 seconds when all the packages are in local cache, and around 40 seconds when the cache is empty and the packages are downloaded from the internet.

When the agent is started as a service again, package restore lasts 11 minutes again.

We do not experience this slowness with the other types of steps (build, publish artifacts...). All of them are fine, just nuget restore is not working as expected.

For testing purposes, there were no steps in the build pipeline, just NuGet restore.

Agent and Worker's Diagnostic Logs

Agent_20190719-140904-utc.log
Worker_20190722-140600-utc.log

[juliobbv]

@aasim, can you take a look at this issue?

Looks like the "NuGet Restore" task can take much longer to complete when the agent is configured to run as a service vs. when it's run interactively.

From the agent's point of view, once the agent is launched, it doesn't take any specific decisions on whether it's running as any of the modes. The only difference is that running an account as a service allows running as the agent as another account. Maybe doing so causes the authentication/authorization flow in NuGet to take a slower path?

[ltrzesniewski]

I've noticed that NuGet restore is also very slow on Microsoft-hosted agents (also see #2134, which got dismissed). I guess the underlying issue is the same.

Anyway, here's some info I found out from a project of mine, in case it could be helpful:

• Caching the global packages cache does not seem to help, so the problem is apparently not tied to network speed. This is slow even though NuGet is not supposed to need network access.
• I took a binary log of a restore through MSBuild (with all packages already cached), and here's what it shows:
  
  
  
  • Most time is spent in RestoreTask, and it is definitely the bottleneck
  
  
  • The MsBuild task which invokes the _IsProjectRestoreSupported target (in _FilterRestoreGraphProjectInputItems) also seems a bit too slow, but this may be an unrelated issue.
  
  
  • The logs of RestoreTask are similar to what I'm getting on my machine, and I couldn't spot any meaningful difference except for the concurrent job count. Unfortunately, MSBuildLogViewer doesn't show the timestamps of individual log lines and I don't know if this information is available in the binlog itself, but I can check that.
  
  

[MehdiSaeedifar]

We have the same issue. It takes 5 minutes to restore packages. Any solution found?

[alexstdev]

Hi there, we seem to have the same Problem, Nuget restore sitting around 5 minutes before the logs come in, the total task takes 9minutes...

[igor-teresco]

Hi, what was taking 20-30 seconds before, now lasts for 10-15 mins - very disappointing... Please check and fix!

[Jetski5822]

Same issue here - I just installed a fresh agent and NugetCommand is taking 11ish minutes.

[weloytty]

me too, I've been pulling my hair out -- 11 minutes to restore a ~65 project sln that has been unchanged for over a year. (Build is running under a service account as well)

[daviesro]

4-5 minutes per nuget restore on our Azure build pipelines. 15 seconds on local machine.

[hoborm]

I experience similar issues as described above, however I do not think that this can be necessarily linked to azure pipelines agent. I have a restore time around 2.5 minutes on a Jenkins node (run as a service) and 10-20 seconds on my local machine. (tried with -NoCache as well) My best guess would be to poke around the nuget.exe issues.

[ArtourK]

We also experience this issue!
Some time 1 month back both Jenkins and Azure DevOps Server, nuget restore times went x10.
For the Jenkins we run the service as a user, I just logged in as that user and restarted the server.
After that the problem was solved!

For Azure DevOps ServerI have tried more or less everything I can think of! The times are not going down :/

Setting NUGET_CERT_REVOCATION_MODE = "offline" solved the problem, as the server will not use "internet".
Download only from internal nuget storage...

[anatolybolshakov]

Hi everyone, this seems to be a known issue with NuGet - please see this link for more details.
As @ArtourK mentioned - one of possible solution could be to set environment variable NUGET_CERT_REVOCATION_MODE = "offline" - it is not recommended under normal circumstances, but could help to resolve issue with longer execution time.
Let me close this at the moment since it doesn't seem to be related to agent - please feel free to ask any further questions.

[dssagar93]

What a disappointment, Restore Packages took 3 mins, Build solution took <50 seconds. Extremely slow.