Home: error NU1101: Unable to find package - Azure DevOps (Private Nuget)

[gerhardmareede]

Environment

VS version (if appropriate): 2017 or 2019

OS version : Hosted Ubuntu 1604

Environment: Azure Devops Build Pipeline

Detailed repro steps so we can see the same problem

1. Create a new ASP.NET Core Web Application
2. Under Nuget Package Manager add a "Package Source" that point to my private source.
3. Install the nuget package from your source.
4. Add docker support to the project. (Rick click project, Add, Docker Support). Edit the Dockerfile.

`FROM mcr.microsoft.com/dotnet/core/aspnet:2.2-stretch-slim AS base
WORKDIR /app

FROM mcr.microsoft.com/dotnet/core/sdk:2.2-stretch AS build
WORKDIR /src
COPY . .
COPY *.csproj ./
COPY nuget.config /
RUN dotnet restore

COPY . .
WORKDIR "/src"
RUN dotnet build "TestPipeline.csproj" -c Release -o /app

FROM build AS publish
RUN dotnet publish "TestPipeline.csproj" -c Release -o /app

FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "TestPipeline.dll"]`

1. Upload to Azure git Repo.
2. Create a Build pipeline with the following steps.
   6.1 Nuget Installer
   6.2 Nuget restore (This is successfull and restore packages. I added a Connect Feed for my Nuget package)
   6.3 Commandline task that does docker build -t testpipeline ./TestPipeline
3. Queue a build

Nuget restore details

============================================================================== Task : NuGet Description : Restore, pack, or push NuGet packages, or run a NuGet command. Supports NuGet.org and authenticated feeds like Package Management and MyGet. Uses NuGet.exe and works with .NET Framework apps. For .NET Core and .NET Standard apps, use the .NET Core task. Version : 2.151.0 Author : Microsoft Corporation Help : [More Information](https://go.microsoft.com/fwlink/?LinkID=613747) ==============================================================================

`NuGet Config files used:
/home/vsts/work/1/Nuget/tempNuGet_264.config

Feeds used:
https://company.pkgs.visualstudio.com/_packaging/fcab3aa3-9d0a-4896-b61d-919cff8348e3/nuget/v3/index.json

Installed:
60 package(s) to /home/vsts/work/1/s/TestPipeline/TestPipeline.csproj

[section]Finishing: NuGet restore`

Error log

`Step 9/18 : RUN dotnet restore #"TestPipeline/TestPipeline.csproj"
---> Running in 0843de55d156
/src/TestPipeline.csproj : error NU1101: Unable to find package Company.Connect.Shared.Logging. No packages exist with this id in source(s): nuget.org
Restore failed in 1.5 sec for /src/TestPipeline.csproj.
The command '/bin/sh -c dotnet restore #"TestPipeline/TestPipeline.csproj"' returned a non-zero code: 1

[error]Bash exited with code '1'.

[section]Finishing: Build Docker Image`

[zivkan]

From your error log, I see:

No packages exist with this id in source(s): nuget.org

which means that the nuget.config file that was copied into your docker image did not contain your private feed. When you use Azure DevOps to do a NuGet restore and tell it to use your private feed, it creates a temporary nuget.config file and uses that, as you can see in the other log:

NuGet Config files used:
/home/vsts/work/1/Nuget/tempNuGet_264.config

At the beginning of your repro steps, you said you added your private feed as a package source in Visual Studio, however, Visual Studio's NuGet sources list edits your user profile's nuget.config, which means those settings don't apply to other users on the same computer or any other computer. It does apply to all solutions on your machine when run as your user account. You should manually create a nuget.config in your repo, either at the root or in the same directory as your .sln file and add your private source there if you want it to apply to all machines that clone your repo. However, if the nuget.config that is copied into the docker image doesn't contain authentication information for the private feed (the temporary nuget.config file that Azure DevOps created), restoring within the image will likely fail with an unauthorized error.

There is already work planned to improve NuGet hierarchical config management in Visual Studio, but for now you need to do it manually. That won't solve the authentication issues inside docker contains/images though.

[gerhardmareede]

Thanks, I will try manually creating the nuget.config. We are using a Linux container, but I found a solution for Windows containers. If I add a Visual Studio Build task before my Docker build task (Not doing a dotnet restore as part of the docker build task), it works

`steps:

• task: VSBuild@1
  displayName: 'Build solution TestPipeline.sln'
  inputs:
  solution: TestPipeline.sln
  msbuildArchitecture: x64`

[gerhardmareede]

I have tried various ways of using a nuget.config file. I have a custom nuget step that populate the authorization credentials.

sources update -Name "ConnectFeed" -Username "vsts" -Password "$(System.AccessToken)" -StorePasswordInClearText -ConfigFile ./TestPipeline/nuget.config

Then step runs fine

Task         : NuGet
Description  : Restore, pack, or push NuGet packages, or run a NuGet command. Supports NuGet.org and authenticated feeds like Package Management and MyGet. Uses NuGet.exe and works with .NET Framework apps. For .NET Core and .NET Standard apps, use the .NET Core task.
Version      : 2.151.0
Author       : Microsoft Corporation
Help         : 

SYSTEMVSSCONNECTION exists true
SYSTEMVSSCONNECTION exists true
Detected NuGet version 5.0.2.5988 / 5.0.2+971e4d1beed0771b1a469a8cd9267eb3d1a887be
SYSTEMVSSCONNECTION exists true
[command]/usr/bin/mono /opt/hostedtoolcache/NuGet/5.0.2/x64/nuget.exe sources update -Name ConnectFeed -Username vsts -Password $(System.AccessToken) -StorePasswordInClearText -ConfigFile ./TestPipeline/nuget.config -NonInteractive
Package source "ConnectFeed" was successfully updated.
Finishing: Authenticate with ConnectFeed NuGet

But then on the RUN dotnet restore I still get the same error

/usr/share/dotnet/sdk/2.2.204/NuGet.targets(119,5): error : Unable to load the service index for source https://company.pkgs.visualstudio.com/_packaging/ConnectFeed/nuget/v3/index.json. [/src/TestPipeline.csproj] /usr/share/dotnet/sdk/2.2.204/NuGet.targets(119,5): error : Response status code does not indicate success: 401 (Unauthorized). [/src/TestPipeline.csproj] 0 Warning(s) 1 Error(s)

[gerhardmareede]

I have also tried this nuget.config file and adding a username and password to the environment variables

<?xml version="1.0" encoding="utf-8"?> <configuration> <solution> <add key="disableSourceControlIntegration" value="true" /> </solution> <packageSources> <clear /> <add key="Nuget" value="https://api.nuget.org/v3/index.json" protocolVersion="3" /> <add key="ConnectFeed" value="https://company.pkgs.visualstudio.com/_packaging/ConnectFeed/nuget/v3/index.json" protocolVersion="3" /> </packageSources> <packageSourceCredentials> <ConnectFeed> <add key="Username" value="%USER_VARIABLE%" /> <add key="ClearTextPassword" value="%PAT%" /> </ConnectFeed> </packageSourceCredentials> </configuration>

[zivkan]

I'm not an expert on Azure DevOps, and I have no experience at all with docker. As a workaround I suggest building directly on the agent machine, running dotnet publish, then copying the binaries in the publish directory to your docker image, rather than building inside a container/image. On one hand I can understand that people try to use docker as a consistent build environment, in the same way that the original intent of containers was to have consistent environments for production services. On the other hand, docker introduces new classes of issues, such as the one you're experiencing. As you can tell, I'm not as enthusiastic about containers as many other people.

I'll try to find someone with experience with docker, or an azure devops person, to chime in on the thread.

[zivkan]

@shubham90 does Azure DevOps have guidance on how to use private feeds from docker containers?

[shubham90]

@gerhardmaree-compeon If I understand your problem correctly, you are trying to auth to a private Azure DevOps artifact both in the nuget restore step and the docker image creation step. And it's failing to do auth on the docker step.

So, is it possible for you to pass the azure artifacts feed endpoint and the PAT as a --build-arg to the docker command? You can then use the feed endpoint and password(PAT) to do auth in the docker image directly. Something similar to this . You will also need the artifacts credential provider which will do the auth for you if you set the VSS_NUGET_EXTERNAL_FEED_ENDPOINTS env var correctly. In the gist I provided, see line 7 and 19.

[shubham90]

@zivkan The best solution today is to use the artifacts credential provider to do auth from docker containers. We are working to integrate the cred provider with docker images so that the experience is seamless and this extra step of adding it manually is not required.

[gerhardmareede]

Thanks, I tried what you suggested as follows

CredentialProvider.VSS.exe -Uri https://company.pkgs.visualstudio.com/_packaging/ConnectFeed/nuget/v3/index.json

Then I get the login credential that I use in the docker file (Only to test, then I will remove the hardcoding)

`FROM mcr.microsoft.com/dotnet/core/sdk:2.2-stretch AS build

RUN wget -qO- https://raw.githubusercontent.com/Microsoft/artifacts-credprovider/master/helpers/installcredprovider.sh | bash

WORKDIR /workdir
COPY ./ .

ENV NUGET_CREDENTIALPROVIDER_SESSIONTOKENCACHE_ENABLED true

ENV VSS_NUGET_EXTERNAL_FEED_ENDPOINTS {\"endpointCredentials\": [{\"endpoint\":\"https://company.pkgs.visualstudio.com/_packaging/ConnectFeed/nuget/v3/index.json\", \"username\":\"username\", \"password\":\"token value\"}]}

RUN dotnet restore

RUN dotnet build "TestPipeline.csproj"`

Then I run the following in commandline docker build -t testpipeline . and get the following error.

Step 7/8 : RUN dotnet restore ---> Running in b28d91dd07e0 /usr/share/dotnet/sdk/2.2.204/NuGet.targets(119,5): error : Unable to load the service index for source https://company.pkgs.visualstudio.com/_packaging/ConnectFeed/nuget/v3/index.json. [/workdir/TestPipeline.csproj] /usr/share/dotnet/sdk/2.2.204/NuGet.targets(119,5): error : Resource temporarily unavailable [/workdir/TestPipeline.csproj] The command '/bin/sh -c dotnet restore' returned a non-zero code: 1

[gerhardmareede]

Thanks for the help I managed to get it working. I created an environment variable with the PAT. Then calling docker as follows

docker build --build-arg TEST=$(PAT) -t testpipeline ./TestPipeline

And then in the Dockerfile.

`
ARG TEST

ENV VSS_NUGET_EXTERNAL_FEED_ENDPOINTS {\"endpointCredentials\": [{\"endpoint\":\"https://company.pkgs.visualstudio.com/_packaging/ConnectFeed/nuget/v3/index.json\", \"username\":\"username\", \"password\":\"${TEST}\"}]}`

[gerhardmareede]

The only question I still have is it possible to set an expiry on the credentials created with CredentialProvider.VSS.exe because I do not want to generate these for every build

[gerhardmareede]

This solved it

`steps:

• bash: echo This is a script that could use $SYSTEM_ACCESSTOKEN
  env:
  SYSTEM_ACCESSTOKEN: $(System.AccessToken)
• powershell: Write-Host "This is a script that could use $env:SYSTEM_ACCESSTOKEN"
  env:
  SYSTEM_ACCESSTOKEN: $(System.AccessToken)`

[shubham90]

Great! yeah the best way to get a PAT is to use System.AccessToken as you said. Glad that it's working for you again.

[nick5454]

This solved it

`steps:

• bash: echo This is a script that could use $SYSTEM_ACCESSTOKEN
  env:
  SYSTEM_ACCESSTOKEN: $(System.AccessToken)
• powershell: Write-Host "This is a script that could use $env:SYSTEM_ACCESSTOKEN"
  env:
  SYSTEM_ACCESSTOKEN: $(System.AccessToken)`

This doesn't solve usings the SystemAccessToken in the Nuget.config file. Anyone get this working?