Dotnet-docker: Dotnet restore fail with 403 when using azure devops Nuget feed

[jeffpriz]

dotnet restore fails within docker image when calling a custom feed hosted in Azure DevOps. The call returns 403 when run inside docker, but runs successfully when execute the dotnet restore with the same nuget.config and same project on the Agent itself.

Steps to reproduce the issue

1. multi-stage dockerfile build for dotnet core .sln

FROM microsoft/dotnet:2.2-aspnetcore-runtime AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443

FROM microsoft/dotnet:2.2-sdk AS build
WORKDIR /src
COPY ["./.", "./"]
RUN dotnet restore --configfile automation-nuget.config
COPY . .
WORKDIR "/src/service.admin/api/host/"
RUN dotnet build "host.netcore.admin.csproj" -c Release -o /app

FROM build AS publish
RUN dotnet publish "host.netcore.admin.csproj" -c Release -o /app

FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "host.netcore.admin.dll"]

2. NuGet artifacts hosted in AzureDevOps, with Nuget.config source added:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <config>
        <add key="repositoryPath" value="Packages" />
    </config>
    <packageSources>
        <add key="CustomComponents" value="https://pkgs.dev.azure.com/<ourAzureDevOpsAccount>/_packaging/<our feed name>/nuget/v3/index.json" />
        <add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
    </packageSources>
    <packageSourceCredentials>
        <CustomComponents>    
            <add key="Username" value="---service account user id here---" />      
            <add key="ClearTextPassword" value="--- userPAT here ---" />
        </CustomComponents>
  </packageSourceCredentials>
</configuration>

1. dotnet restore with --conifgFile to our NugetConfig

Expected behavior

dotnet restore should successfully restore nuget packages from our Azure Devops Artifacts feed

Actual behavior

Step 7/17 : COPY ["./.", "./"]
---> 3546bdeada29
Step 8/17 : RUN dotnet restore --configfile automation-nuget.config
---> Running in 5ea3a2a2f556
C:\Program Files\dotnet\sdk\2.1.804\NuGet.targets(123,5): error : Unable to load the service index for source https://pkgs.dev.azure.com//_packaging//nuget/v3/index.json. [C:\src\api.admin.sln]
C:\Program Files\dotnet\sdk\2.1.804\NuGet.targets(123,5): error : Response status code does not indicate success: 403 (The requested operation is not allowed.). [C:\src\api.admin.sln]
The command 'cmd /S /C dotnet restore --configfile automation-nuget.config' returned a non-zero code: 1

[error]C:\Program Files\Dockerdocker.exe failed with return code: 1

Additional information (e.g. issue happens only occasionally)

Running in an Azure Devops hosted agent, specification "windows-2019"
Running a dotnet restore using same project, and same nuget config for same nuget feed on same hosted agent succeeds when not running from docker but just running on the build agent

Output of docker version

Client: Docker Engine - Enterprise
Version: 19.03.5
API version: 1.40
Go version: go1.12.12
Git commit: 2ee0c57608
Built: 11/13/2019 08:00:16
OS/Arch: windows/amd64
Experimental: false

Server: Docker Engine - Enterprise
Engine:
Version: 19.03.5
API version: 1.40 (minimum version 1.24)
Go version: go1.12.12
Git commit: 2ee0c57608
Built: 11/13/2019 07:58:51
OS/Arch: windows/amd64
Experimental: false

Output of docker info

[command]"C:\Program Files\Dockerdocker.exe" info
Client:
Debug Mode: false
Plugins:
cluster: Manage Docker clusters (Docker Inc., v1.2.0)

Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 5
Server Version: 19.03.5
Storage Driver: windowsfilter
Windows:
Logging Driver: json-file
Plugins:
Volume: local
Network: ics internal l2bridge l2tunnel nat null overlay private transparent
Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434)
Operating System: Windows Server 2019 Datacenter Version 1809 (OS Build 17763.973)
OSType: windows
Architecture: x86_64
CPUs: 2
Total Memory: 7GiB
Name: fv-az674
ID: PBOW:LIWJ:L6FI:R7TZ:CCPI:3BO4:AQGR:HMAP:CVGG:QXD5:WCE6:6LSK
Docker Root Dir: C:\ProgramDatadocker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

[MichaelSimons]

FWIW, I took a look and nothing jumps out at me as being wrong. We document this very pattern in our Managing NuGet Credentials in Docker Scenarios snippet.

[jeffpriz]

FWIW, I took a look and nothing jumps out at me as being wrong. We document this very pattern in our Managing NuGet Credentials in Docker Scenarios snippet.

Yes, I've read that document... Which is why I am confused as to why this fails. I don't know where the trouble lies.. is this an issue with docker/dotnet restore.. is it an issue on the side of Azure DevOps and the nuget feed? Seems like these should all play together nicely, but right now all I have to go on is a 403 error.. I thought I might have to fight through some 401 or something, but 403? From docker but not the hosted agent itself? I'm at a loss

[mthalman]

I'm curious why you're not using Azure Artifact Credential Provider since you're targeting an Azure Artifact feed. That pattern is documented here. I'm not sure whether that would change things but certainly worth a shot.

[jeffpriz]

I'm curious why you're not using Azure Artifact Credential Provider since you're targeting an Azure Artifact feed. That pattern is documented here. I'm not sure whether that would change things but certainly worth a shot.

Yeah.. I was trying to just replicate things for the process we had in place already for nuget.config (and I saw no reason why I felt that should not work)... But I'll go try to give the Credential Provider stuff a try and see if that changes anything.

[jeffpriz]

@mthalman any sample on using that in a dockerfile that works in a windows docker image? the sample that is in the documentation runs curl | bash... which fails in my build on the bash... if I attempt to use the sample in the microsoft/artifacts-credprovider git repo, they use wget | bash which fails on wget..
I guess I'm not sure how to call a powershell version of that in my dockerfile
Invoke-Expression $((Invoke-WebRequest -Uri "https://raw.githubusercontent.com/microsoft/artifacts-credprovider/master/helpers/installcredprovider.ps1").Content)

throws a docker error because Invoke-Expression is not valid..

[jeffpriz]

Interestingly, I tried running this in an Ubuntu image (instead of windows).

I was able to utilize the Azure Artifact Credential Provider and get it installed as per the documentation you mentioned above. However I still get a 403 on my dotnet restore step

Downloading from https://github.com/Microsoft/artifacts-credprovider/releases/latest/download/Microsoft.NuGet.CredentialProvider.tar.gz
INFO: credential provider netcore plugin extracted to /root/.nuget/
Removing intermediate container a32dc35e0242
 ---> 6629d87ec96e
Step 8/22 : ENV NUGET_CREDENTIALPROVIDER_SESSIONTOKENCACHE_ENABLED true
 ---> Running in 3ae044f79487
Removing intermediate container 3ae044f79487
 ---> 5a41f2e2c25f
Step 9/22 : ENV DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER=0
 ---> Running in 7ec1815cfb6f
Removing intermediate container 7ec1815cfb6f
 ---> b1b1b5a3c4a8
Step 10/22 : ENV VSS_NUGET_EXTERNAL_FEED_ENDPOINTS "{\"endpointCredentials\": [{\"endpoint\":\"https://pkgs.dev.azure.com/<ourADOProject>/_packaging/<ournugetfeed>/nuget/v3/index.json\", \"username\":\"<ourusername>\", \"password\":\"${FEED_ACCESSTOKEN}\"}]}"
 ---> Running in 01e395c3ac2d
Removing intermediate container 01e395c3ac2d
 ---> 0db511a91994
Step 11/22 : WORKDIR /src
 ---> Running in 2d13d1ebb1ea
Removing intermediate container 2d13d1ebb1ea
 ---> b24aadbcbc82
Step 12/22 : COPY ["./.", "./"]
 ---> 697815d47f0c
Step 13/22 : RUN dotnet restore --configfile Nuget.Config "api.admin.sln"
 ---> Running in 4fd548ebba2d
/usr/share/dotnet/sdk/2.1.804/NuGet.targets(123,5): error : Unable to load the service index for source https://pkgs.dev.azure.com/<ourADOSub>/_packaging/<our nuget feed>/nuget/v3/index.json. [/src/api.admin.sln]
/usr/share/dotnet/sdk/2.1.804/NuGet.targets(123,5): error :   Response status code does not indicate success: 403 (). [/src/api.admin.sln]
The command '/bin/sh -c dotnet restore --configfile Nuget.Config "api.admin.sln"' returned a non-zero code: 1

[jeffpriz]

One other thing is that If i do nothing with any of the credential items, I do get a 401 (as I would expect).

I guess I don't understand yet the 403 when in docker

[jeffpriz]

well. I tried setting up a new Nuget feed our organization's Azure DevOps instance and i set up a dummy .net core project to try this out cleanly.. and I still got a 403...

So THEN I went to my personal Azure DevOps instance and set up a Nuget Feed and a nuget package and a simple sample project to consume it and built it in Docker using the same process (basically the same exact Nuget Config and same exact dockerfile, just swapped project names and nuget feeds) ... and it works fine there.. no troubles..

SO that leads me to believe there is something blocking things in our organization's Azure DevOps instance somewhere along the way.. I guess I'll go chat with our NetSec guys and the team running the Azure Subscription to see if they have a clue as to what is up

[Devnsyde]

My issue seems similar except I am not getting a 403.. See #1722

[MichaelSimons]

@jeffpriz - Were you able to track down if this was caused by something within your organization?

[MichaelSimons]

Inactive - closing. Feel free to respond it you find evidence this is an issue with the .NET Docker images.

[jeffpriz]

just a follow up to this for anybody in the future.

Going back and forth with the NetSec team in the organization was met with a lot of "it's not our fault, it's something inside Azure DevOps"... I still had no way to show that it was in fact not something inside Azure DevOps causing this, as everything was running in Azure, and i had no access to Azure's behind-the-scenes logs to show the source of the 403.

SO we filed a support ticket with Microsoft to help to get to the bottom of this.. which at first was met with a lot of "Have you tried this?" kinds of things (even pointing me to this github repo, which I responded to with a link to this issue :) ) . However recently we were able to get to the level of support within Microsoft that in fact was able to help track down the behind-the-scenes logging tracing and get to a root cause.. which did indeed turn out to be an issue with our Two-Factor auth system not recognizing the credentials when coming in from the Docker container. We have now taken this info back to the NetSec guys to hopefully get to the final bottom, but my suspicion has been for a while that the way they have things configured to recognize a "trusted" computer is failing for this docker build. At least I feel like we are getting to make some progress on this after many months.

[GaTechThomas]

Any progress on understanding the issue?

[dpetrizze]

Likewise, we are seeing precisely this problem, a 403 error response. I'm confident that authentication itself is working, if I intentionally change the PAT, I get the 401's as expected. With the correct PAT therefore, we know it is successfully authenticating yet returning 403 during dotnet restore.

One additional observation, I was able to duplicate the 403 problem locally in WSL2 attempting a dotnet restore. Then I used the --interactive option, it prompted for a https://login.microsoftonline.com/common/oauth2/deviceauth devicelogin operation. I put the code in the browser and was then authenticated in WSL2 and then able to successfully restore the nuget feed packages.

This leads me to believe it cannot authenticate my docker build environment (--interactive in a docker build doesn't prompt) even when the PAT is correct, hence the 403.

Have not been able to work past this issue. Any progress or update on this issue?