Home: NuGet failed to restore packages inside docker

Created on 15 Nov 2020 · 17Comments · Source: NuGet/Home

_Forwarded form https://github.com/dotnet/dotnet-docker/issues/1670_

I had a problem when build sample dotnet app
When docker Run shell dotnet restore
There're throw some error:

Failed to download package 'Microsoft.AspNetCore.Authentication.Core.2.2.0' from 'https://api.nuget.org/v3-flatcontainer/microsoft.aspnetcore.authentication.core/2.2.0/microsoft.aspnetcore.authentication.core.2.2.0.nupkg'.
  The download of 'https://api.nuget.org/v3-flatcontainer/microsoft.aspnetcore.authentication.core/2.2.0/microsoft.aspnetcore.authentication.core.2.2.0.nupkg' timed out because no data was received for 60000ms.
    Exception of type 'System.TimeoutException' was thrown.
  Failed to download package 'Microsoft.Extensions.Caching.Abstractions.2.2.0' from 'https://api.nuget.org/v3-flatcontainer/microsoft.extensions.caching.abstractions/2.2.0/microsoft.extensions.caching.abstractions.2.2.0.nupkg'.
  The download of 'https://api.nuget.org/v3-flatcontainer/microsoft.extensions.caching.abstractions/2.2.0/microsoft.extensions.caching.abstractions.2.2.0.nupkg' timed out because no data was received for 60000ms.
    Exception of type 'System.TimeoutException' was thrown.
  Failed to download package 'Microsoft.AspNetCore.Diagnostics.2.2.0' from 'https://api.nuget.org/v3-flatcontainer/microsoft.aspnetcore.diagnostics/2.2.0/microsoft.aspnetcore.diagnostics.2.2.0.nupkg'.
  The download of 'https://api.nuget.org/v3-flatcontainer/microsoft.aspnetcore.diagnostics/2.2.0/microsoft.aspnetcore.diagnostics.2.2.0.nupkg' timed out because no data was received for 60000ms.
    Exception of type 'System.TimeoutException' was thrown.
  Failed to download package 'Microsoft.Extensions.Logging.Abstractions.2.2.0' from 'https://api.nuget.org/v3-flatcontainer/microsoft.extensions.logging.abstractions/2.2.0/microsoft.extensions.logging.abstractions.2.2.0.nupkg'.
  The download of 'https://api.nuget.org/v3-flatcontainer/microsoft.extensions.logging.abstractions/2.2.0/microsoft.extensions.logging.abstractions.2.2.0.nupkg' timed out because no data was received for 60000ms.
    Exception of type 'System.TimeoutException' was thrown.
  Failed to download package 'System.Net.WebSockets.WebSocketProtocol.4.5.3' from 'https://api.nuget.org/v3-flatcontainer/system.net.websockets.websocketprotocol/4.5.3/system.net.websockets.websocketprotocol.4.5.3.nupkg'.
  The download of 'https://api.nuget.org/v3-flatcontainer/system.net.websockets.websocketprotocol/4.5.3/system.net.websockets.websocketprotocol.4.5.3.nupkg' timed out because no data was received for 60000ms.
    Exception of type 'System.TimeoutException' was thrown.
  Failed to download package 'Microsoft.AspNetCore.Localization.Routing.2.2.0' from 'https://api.nuget.org/v3-flatcontainer/microsoft.aspnetcore.localization.routing/2.2.0/microsoft.aspnetcore.localization.routing.2.2.0.nupkg'.
  The download of 'https://api.nuget.org/v3-flatcontainer/microsoft.aspnetcore.localization.routing/2.2.0/microsoft.aspnetcore.localization.routing.2.2.0.nupkg' timed out because no data was received for 60000ms.
    Exception of type 'System.TimeoutException' was thrown.
  Failed to download package 'Microsoft.EntityFrameworkCore.Tools.2.2.4' from 'https://api.nuget.org/v3-flatcontainer/microsoft.entityframeworkcore.tools/2.2.4/microsoft.entityframeworkcore.tools.2.2.4.nupkg'.
  The download of 'https://api.nuget.org/v3-flatcontainer/microsoft.entityframeworkcore.tools/2.2.4/microsoft.entityframeworkcore.tools.2.2.4.nupkg' timed out because no data was received for 60000ms.
...

Dockerfile

# https://hub.docker.com/_/microsoft-dotnet-core
FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS build
WORKDIR /source

# copy csproj and restore as distinct layers
COPY *.csproj .
RUN dotnet restore

# copy and publish app and libraries
COPY . .
RUN dotnet publish -c release -o /app --no-restore

# final stage/image
FROM mcr.microsoft.com/dotnet/core/runtime:3.1
WORKDIR /app
COPY --from=build /app .
ENTRYPOINT ["dotnet", "dotnetapp.dll"]

File *.csproj

<Project Sdk="Microsoft.NET.Sdk" ToolsVersion="15.0">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp3.1</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Ben.BlockingDetector" Version="0.0.3" />
    <PackageReference Include="Microsoft.AspNetCore.App" Version="2.2.7" />
    <PackageReference Include="MongoDB.Driver" Version="2.9.2" />
    <PackageReference Include="prometheus-net" Version="3.1.4" />
    <PackageReference Include="prometheus-net.AspNetCore" Version="3.1.4" />
    <PackageReference Include="Quartz" Version="3.0.7" />
    <PackageReference Include="ServiceStack.Core" Version="5.7.1" />
    <PackageReference Include="ServiceStack.OrmLite.MySqlConnector.Core" Version="5.7.1" />
    <PackageReference Include="ServiceStack.OrmLite.PostgreSQL.Core" Version="5.7.1" />
    <PackageReference Include="ServiceStack.Redis.Core" Version="5.7.1" />
    <PackageReference Include="System.IO.Pipelines" Version="4.6.0" />
  </ItemGroup>
</Project>

Output of docker version

Client: Docker Engine - Community
 Version:           19.03.2
 API version:       1.40
 Go version:        go1.12.8
 Git commit:        6a30dfca03
 Built:             Thu Aug 29 05:29:49 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.2
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.8
  Git commit:       6a30dfca03
  Built:            Thu Aug 29 05:28:23 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info

Client:
 Debug Mode: false

Server:
 Containers: 46
  Running: 6
  Paused: 0
  Stopped: 40
 Images: 1338
 Server Version: 19.03.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.9.0-4-amd64
 Operating System: Debian GNU/Linux 9 (stretch)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.829GiB
 Name: debian-167
 ID: K5NH:LCRJ:AFV7:D3TN:N3IN:KE6B:RICT:7SPD:A3HP:IPGW:OOIN:6XFQ
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

\cc @JonDouglas

HttpCommunication Restore Docker 2 NeedsRepro Bug

Source

xsoheilalizadeh

👍7

Most helpful comment

Looks like throttling the number of requests with nuget.config helps and works much faster than 'disable-parallel'
(dont forget to copy it into container)

<configuration>
  <config>
    <add key='maxHttpRequestsPerSource' value='20' />
  </config>
   ...

Vfialkin on 25 Nov 2020

👍4

All 17 comments

Facing the same issue with .NET 5

abuzaforfagun on 15 Nov 2020

👍1

Adding --disable-parallel fixed my issue but it takes a long time to restore.

abuzaforfagun on 16 Nov 2020

👍4

Any updates on this ?

marcasmar94 on 20 Nov 2020

I'm also facing the same issue. My project is built on top of .NET Core 3.1 LTS.

jackabouaad on 20 Nov 2020

👍1

Any updates?

kelvin-p on 23 Nov 2020

👍1

kmute90 on 23 Nov 2020

👍1

this happens intermittently within docker for me

murugaratham on 24 Nov 2020

👍1

Same here, with latest aspnet:3.1.10-bionic
--disable-parallel helps but can take minutes to restore
Using internal repo instead of api.nuget.org gives the same results

Vfialkin on 24 Nov 2020

👍1

efrost123 on 24 Nov 2020

👍1

Looks like throttling the number of requests with nuget.config helps and works much faster than 'disable-parallel'
(dont forget to copy it into container)

<configuration>
  <config>
    <add key='maxHttpRequestsPerSource' value='20' />
  </config>
   ...

Vfialkin on 25 Nov 2020

👍4

This issue impacts my daily work because I have to wait about ~4 min for each package restore on the container, could anyone in the NuGet team have (just) a look at this issue help us to fix it, it becomes frustrating.

I also tried to mount the NuGet cache folder from host to container but it still takes a lot of time to be restored.

 volumes:
   - ~/.nuget/packages:/root/.nuget/packages:rw

@JonDouglas, @nkolev92, @karann-msft

xsoheilalizadeh on 28 Nov 2020

👍1

cc @zivkan

nkolev92 on 1 Dec 2020

@xsoheilalizadeh Can you give us some details about your setup?

Is the host and containers Windows or Linux or Mac?
When you restore on the host does it have the same problem, or only in contains?
Are you limiting the container resources (ram and/or cpu), or does it have the same resources as the host?

zivkan on 5 Dec 2020

Is the host and containers Windows or Linux or Mac?

I used both Linux and Windows on the same machine as a Host.

When you restore on the host does it have the same problem, or only in contains?

The host is able to restore perfectly without any issues, either it's from cache or NuGet server.

Are you limiting the container resources (ram and/or cpu), or does it have the same resources as the host?

No resource limitation has been set, everything is the default.

xsoheilalizadeh on 5 Dec 2020

I tried your repro (single project, small number of packages), as well as the NuGet.Client repo (large number of projects and packages). I tried on my computer with both Windows and Linux contains, and I tried on 2-core Azure VMs using Ubuntu 20.10 as the host, and once I noticed that you were using Debian 9, I also tried Debian 9 as a host. In all of these attempts, I was not able to reproduce the issue. Restore worked inside the container in all my tests.

I never used Docker before this week, so if anyone has any suggestions on how I might repro the issue, please let me know.

zivkan on 10 Dec 2020

Having the same issue on Mac with .NET 5 Docker 20.10. Any update on this? --disable-parallel works but takes 10x longer

hero3616 on 26 Dec 2020

👀1

Just to complete the working answer by Vfialkin
putting a file name nuget.config (watch the casing) besids the solution works.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <config>
    <add key='maxHttpRequestsPerSource' value='10' />
  </config>
</configuration>

So maybe the issue needs to be fixed by microsoft by changing the default nuget.config inside of mcr.microsoft.com/dotnet/sdk:5.0. So the issue is this one
https://github.com/dotnet/dotnet-docker/issues/1670

Looks like throttling the number of requests with nuget.config helps and works much faster than 'disable-parallel'
(dont forget to copy it into container)
<configuration>
  <config>
    <add key='maxHttpRequestsPerSource' value='20' />
  </config>
   ...