Home: Restore failing with "Too many open files in system" inside docker

[dsplaisted]

One of our tests in the dotnet/sdk repo (GivenThatWeWantToPublishAWebApp.It_publishes_as_framework_dependent_by_default) is failing in CI for the Linux-portable legs, which run inside docker. Restore is failing with the error "Too many open files in system".

The project which is being restored looks like this:

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>netcoreapp2.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <Folder Include="wwwroot\" />
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore.All" Version="2.0.9" />
  </ItemGroup>

</Project>

Here is a run where it is failing: https://dev.azure.com/dnceng/public/_build/results?buildId=348310&view=ms.vss-test-web.build-test-results-tab&runId=10462046&resultId=100023&paneView=debug

Here's the output for the failure:

/home/vsts/work/1/s/artifacts/bin/redist/Release/dotnet-with-previous-runtimes/dotnet msbuild /t:restore /home/vsts/work/1/s/artifacts/tmp/Release/dotnetSdkTests/It_publishes_---43300696/web.csproj -p:Configuration=Release
Microsoft (R) Build Engine version 16.3.0-preview-19426-01+faf5e5d75 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnetmyget.blob.core.windows.net/artifacts/nuget-build/nuget/v3/flatcontainer/microsoft.extensions.localization/index.json'.
  Too many open files in system
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnetmyget.blob.core.windows.net/artifacts/nuget-build/nuget/v3/flatcontainer/microsoft.extensions.localization/index.json'.
  Too many open files in system
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnet.myget.org/F/templating/api/v3/flatcontainer/microsoft.extensions.localization/index.json'.
  Too many open files in system
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnet.myget.org/F/templating/api/v3/flatcontainer/microsoft.extensions.localization/index.json'.
  Too many open files in system
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnetfeed.blob.core.windows.net/dotnet-toolset/flatcontainer/microsoft.extensions.localization/index.json'.
  Too many open files in system
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnetfeed.blob.core.windows.net/dotnet-toolset/flatcontainer/microsoft.extensions.localization/index.json'.
  Too many open files in system
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnetfeed.blob.core.windows.net/dotnet-core/flatcontainer/microsoft.extensions.localization.abstractions/index.json'.
  Too many open files in system
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnetfeed.blob.core.windows.net/dotnet-core/flatcontainer/microsoft.extensions.localization.abstractions/index.json'.
  Too many open files in system
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnetfeed.blob.core.windows.net/dotnet-windowsdesktop/flatcontainer/microsoft.extensions.localization.abstractions/index.json'.
  Too many open files in system
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnetfeed.blob.core.windows.net/dotnet-windowsdesktop/flatcontainer/microsoft.extensions.localization.abstractions/index.json'.
  Too many open files in system
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnetfeed.blob.core.windows.net/aspnet-aspnetcore/flatcontainer/microsoft.extensions.localization.abstractions/index.json'.
  Too many open files in system
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnetfeed.blob.core.windows.net/aspnet-aspnetcore/flatcontainer/microsoft.extensions.localization.abstractions/index.json'.
  Too many open files in system
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnetfeed.blob.core.windows.net/aspnet-aspnetcore-tooling/flatcontainer/microsoft.extensions.localization.abstractions/index.json'.
  Too many open files in system
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnetfeed.blob.core.windows.net/aspnet-aspnetcore-tooling/flatcontainer/microsoft.extensions.localization.abstractions/index.json'.
  Too many open files in system
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnetfeed.blob.core.windows.net/aspnet-entityframeworkcore/flatcontainer/microsoft.extensions.localization.abstractions/index.json'.
  Too many open files in system
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnetfeed.blob.core.windows.net/aspnet-entityframeworkcore/flatcontainer/microsoft.extensions.localization.abstractions/index.json'.
  Too many open files in system
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnetfeed.blob.core.windows.net/aspnet-extensions/flatcontainer/microsoft.extensions.localization.abstractions/index.json'.
  Too many open files in system
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnetfeed.blob.core.windows.net/aspnet-extensions/flatcontainer/microsoft.extensions.localization.abstractions/index.json'.
  Too many open files in system
  Retrying 'FindPackagesByIdAsync' for source 'https://api.nuget.org/v3-flatcontainer/microsoft.extensions.localization.abstractions/index.json'.
  Too many open files in system
  Retrying 'FindPackagesByIdAsync' for source 'https://api.nuget.org/v3-flatcontainer/microsoft.extensions.localization.abstractions/index.json'.
  Too many open files in system
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnetfeed.blob.core.windows.net/dotnet-core/flatcontainer/microsoft.extensions.logging.debug/index.json'.
  Too many open files in system
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnetfeed.blob.core.windows.net/dotnet-core/flatcontainer/microsoft.extensions.logging.debug/index.json'.
  Too many open files in system
  Retrying 'FindPackagesByIdAsync' for source 'https://api.nuget.org/v3-flatcontainer/microsoft.aspnetcore/index.json'.
  Too many open files
/home/vsts/work/1/s/artifacts/bin/redist/Release/dotnet-with-previous-runtimes/sdk/3.0.100-rc2-014208/NuGet.targets(123,5): error : Failed to retrieve information about 'Microsoft.Extensions.Localization' from remote source 'https://dotnetmyget.blob.core.windows.net/artifacts/nuget-build/nuget/v3/flatcontainer/microsoft.extensions.localization/index.json'. [/home/vsts/work/1/s/artifacts/tmp/Release/dotnetSdkTests/It_publishes_---43300696/web.csproj]
/home/vsts/work/1/s/artifacts/bin/redist/Release/dotnet-with-previous-runtimes/sdk/3.0.100-rc2-014208/NuGet.targets(123,5): error :   Too many open files in system [/home/vsts/work/1/s/artifacts/tmp/Release/dotnetSdkTests/It_publishes_---43300696/web.csproj]
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnetfeed.blob.core.windows.net/aspnet-entityframeworkcore/flatcontainer/microsoft.extensions.logging.debug/index.json'.
  Bad value for ai_flags
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnetfeed.blob.core.windows.net/aspnet-aspnetcore-tooling/flatcontainer/microsoft.extensions.logging.debug/index.json'.
  Bad value for ai_flags
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnetfeed.blob.core.windows.net/dotnet-windowsdesktop/flatcontainer/microsoft.extensions.logging.debug/index.json'.
  Bad value for ai_flags
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnetfeed.blob.core.windows.net/dotnet-toolset/flatcontainer/microsoft.extensions.options.configurationextensions/index.json'.
  Name or service not known
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnetfeed.blob.core.windows.net/dotnet-toolset/flatcontainer/microsoft.extensions.options.configurationextensions/index.json'.
  Name or service not known
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnetmyget.blob.core.windows.net/artifacts/msbuild/nuget/v3/flatcontainer/microsoft.extensions.localization/index.json'.
  Bad value for ai_flags
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnetmyget.blob.core.windows.net/artifacts/msbuild/nuget/v3/flatcontainer/microsoft.extensions.localization/index.json'.
  Bad value for ai_flags
  Retrying 'FindPackagesByIdAsync' for source 'https://api.nuget.org/v3-flatcontainer/microsoft.extensions.localization/index.json'.
  Bad value for ai_flags
  Retrying 'FindPackagesByIdAsync' for source 'https://api.nuget.org/v3-flatcontainer/microsoft.extensions.localization/index.json'.
  Bad value for ai_flags
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnetfeed.blob.core.windows.net/aspnet-extensions/flatcontainer/microsoft.extensions.localization/index.json'.
  Bad value for ai_flags
  Retrying 'FindPackagesByIdAsync' for source 'https://dotnetfeed.blob.core.windows.net/aspnet-extensions/flatcontainer/microsoft.extensions.localization/index.json'.
  Bad value for ai_flags

Here is a zip with the test run logs: Microsoft.NET.Publish.Tests.zip

[rrelyea]

@zkat - can you please take this issue on as this weeks hotseat...and do initial repro and or triage.

[zkat]

@dsplaisted What's your ulimit -n set to? Usually, this is issue with Docker images having oppressively low fd limits configured.

[nguerrera]

Seeing this again. Here's one https://dev.azure.com/dnceng/public/_build/results?buildId=384679&view=logs&j=fd8df696-9db3-57b4-fb8e-b67f5ba463c3

@MattGal This is using machine provisioned via

            name: NetCorePublic-Pool
            queue: BuildPool.Ubuntu.1604.Amd64.Open

Do you know what ulimit -n is for those? Can it be increased?

[MattGal]

@nguerrera taking a look now, but if we're talking about docker images the question is more for the docker image not the build agent.

[nguerrera]

This is just running our tests on the build agent where it hit.

[MattGal]

Indeed, I see that (and building inside a docker container might be an eventual good idea). Was this recently changed over due to the lowered disk size on hosted ubuntu?

Ulimit on these machines is:

time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        8192
coredump(blocks)     unlimited
memory(kbytes)       unlimited
locked memory(kbytes) 64
process              63990
nofiles              1024
vmemory(kbytes)      unlimited
locks                unlimited
rtprio               0

and I'm guessing the 1024 is your problem.

[nguerrera]

Was this recently changed over due to the lowered disk size on hosted ubuntu?

Yes. Happy to change again if there's a better config.

[MattGal]

I'll make an issue to move this to 8192 or something. For the time being, I believe if you literally can just call ulimit -n 8192 before starting the test process you should succeed (assuming you only need 8K files open at once)

[nkolev92]

Thank you for filling this issue.

At this point we don't see us taking any action on this issue.
If anyone else is hitting this issue, please increase the ulimit.

[MattGal]

Thank you for filling this issue.

At this point we don't see us taking any action on this issue.
If anyone else is hitting this issue, please increase the ulimit.

(for completeness, the value going back to default was a regression, we fixed this on our build machines)