Azure-pipelines-agent: Azure DevOps Agent unable to download artifact

[sdegenhardt]

### Agent Version and Platform
Version of your agent? 2.169.0
OS of the machine running the agent? Microsoft Windows Server 2019 (10.0.177763)

Azure DevOps Type and Version
dev.azure.com

What's not working?
Download artifacts from GitHub fails starting May 22nd. It worked before, the configuration wasn't changed

Agent and Worker's Diagnostic Logs
2020-05-22T18:56:10.5631926Z Starting artifacts download...
2020-05-22T18:56:10.8414080Z ##[error]Downloading artifacts failed: System.InvalidOperationException: NotFound at Microsoft.VisualStudio.Services.Agent.Worker.Release.Artifacts.GitHubHttpClient.GetUserRepo(String accessToken, String repositoryName) at Microsoft.VisualStudio.Services.Agent.Worker.Release.Artifacts.GitHubArtifact.GetArtifactDetails(IExecutionContext context, AgentArtifactDefinition agentArtifactDefinition) at Microsoft.VisualStudio.Services.Agent.Worker.Release.ReleaseJobExtension.<>c__DisplayClass46_0.b__1()
at Microsoft.VisualStudio.Services.Agent.Worker.Release.RetryExecutor.Execute(Action action)
at Microsoft.VisualStudio.Services.Agent.Worker.Release.ReleaseJobExtension.ConvertToArtifactDefinition(AgentArtifactDefinition agentArtifactDefinition, IExecutionContext executionContext, IArtifactExtension extension)
at Microsoft.VisualStudio.Services.Agent.Worker.Release.ReleaseJobExtension.DownloadArtifacts(IExecutionContext executionContext, IList`1 agentArtifactDefinitions, String artifactsWorkingFolder)
at Microsoft.VisualStudio.Services.Agent.Worker.Release.ReleaseJobExtension.DownloadArtifactsAndCommitsAsync(IExecutionContext executionContext, Object data)
2020-05-22T18:56:10.8588299Z ##[error]NotFound
2020-05-22T18:56:10.8604804Z ##[section]Finishing: Download Artifacts

[jeffpatton1971]

I'm seeing this as well with a new repository within our github org. I've gone through the security settings of the repo and they line up with other repos' that work just fine. To be honest it appears to actually checkout the repo, but something crashes at the end, so I'm unclear what the actual issue is.

I have updated this message with more details, I originally was seeing this with repositories created in github very recently so I have attempted to use older repositories and am seeing this behaviour across all existing repositories new and old. Additionally i was unsure if this was an agent issue, so I have included the output from both our windows pool and the linux pool.

2020-11-17T16:05:17.1349741Z ##[section]Starting: Download Artifacts
2020-11-17T16:05:17.1510910Z Creating artifacts directory: C:\rs-pkgs\scu-bld-agnt-01-agent-01\_work\r329\a
2020-11-17T16:05:17.1904814Z Created artifacts directory: C:\rs-pkgs\scu-bld-agnt-01-agent-01\_work\r329\a
2020-11-17T16:05:17.1905567Z Downloading artifact
2020-11-17T16:05:17.1931347Z Starting artifacts download...
2020-11-17T16:05:17.5512820Z Downloading linked artifact customer of type GitHub...
2020-11-17T16:05:17.5542082Z Ensuring artifact folder C:\rs-pkgs\scu-bld-agnt-01-agent-01\_work\r329\a\customer exists and is clean.
2020-11-17T16:05:17.5567778Z Received GitHub artifact Details
2020-11-17T16:05:17.5733597Z Syncing repository: f7969199-a739-4394-ae52-c0c8c484218b (GitHub)
2020-11-17T16:05:17.5767040Z Prepending Path environment variable with directory containing 'git.exe'.
2020-11-17T16:05:17.5800778Z ##[command]git version
2020-11-17T16:05:18.4676807Z git version 2.25.0.windows.1
2020-11-17T16:05:18.4847956Z ##[command]git init "C:\rs-pkgs\scu-bld-agnt-01-agent-01\_work\r329\a\customer"
2020-11-17T16:05:18.6038105Z Initialized empty Git repository in C:/rs-pkgs/scu-bld-agnt-01-agent-01/_work/r329/a/customer/.git/
2020-11-17T16:05:18.6062006Z ##[command]git remote add origin https://github.com/org/customer.git
2020-11-17T16:05:18.6762140Z ##[command]git config gc.auto 0
2020-11-17T16:05:18.7372590Z ##[command]git config --get-all http.https://github.com/org/customer.git.extraheader
2020-11-17T16:05:18.7893396Z ##[command]git config --get-all http.proxy
2020-11-17T16:05:18.8439764Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --tags --prune --progress --no-recurse-submodules origin
2020-11-17T16:05:19.4682035Z remote: Enumerating objects: 4, done.        
2020-11-17T16:05:19.4685282Z remote: Counting objects:  25% (1/4)        
2020-11-17T16:05:19.4687481Z remote: Counting objects:  50% (2/4)        
2020-11-17T16:05:19.4690358Z remote: Counting objects:  75% (3/4)        
2020-11-17T16:05:19.4693923Z remote: Counting objects: 100% (4/4)        
2020-11-17T16:05:19.4697513Z remote: Counting objects: 100% (4/4), done.        
2020-11-17T16:05:19.4698460Z remote: Compressing objects:  33% (1/3)        
2020-11-17T16:05:19.4699423Z remote: Compressing objects:  66% (2/3)        
2020-11-17T16:05:19.4700289Z remote: Compressing objects: 100% (3/3)        
2020-11-17T16:05:19.4701578Z remote: Compressing objects: 100% (3/3), done.        
2020-11-17T16:05:19.4702494Z remote: Total 4 (delta 0), reused 4 (delta 0), pack-reused 0        
2020-11-17T16:05:19.4931028Z From https://github.com/org/customer
2020-11-17T16:05:19.4933009Z  * [new branch]      master     -> origin/master
2020-11-17T16:05:19.5180480Z ##[command]git checkout --progress --force 1f91e36129daae68bff1287183d2d97df94f7d54
2020-11-17T16:05:19.5964385Z Note: switching to '1f91e36129daae68bff1287183d2d97df94f7d54'.
2020-11-17T16:05:19.5970421Z 
2020-11-17T16:05:19.5972155Z You are in 'detached HEAD' state. You can look around, make experimental
2020-11-17T16:05:19.5973958Z changes and commit them, and you can discard any commits you make in this
2020-11-17T16:05:19.5974979Z state without impacting any branches by switching back to a branch.
2020-11-17T16:05:19.5975551Z 
2020-11-17T16:05:19.5976375Z If you want to create a new branch to retain commits you create, you may
2020-11-17T16:05:19.5977141Z do so (now or later) by using -c with the switch command. Example:
2020-11-17T16:05:19.5977762Z 
2020-11-17T16:05:19.5978269Z   git switch -c <new-branch-name>
2020-11-17T16:05:19.5978710Z 
2020-11-17T16:05:19.5979272Z Or undo this operation with:
2020-11-17T16:05:19.5979828Z 
2020-11-17T16:05:19.5980251Z   git switch -
2020-11-17T16:05:19.5980632Z 
2020-11-17T16:05:19.5981363Z Turn off this advice by setting config variable advice.detachedHead to false
2020-11-17T16:05:19.5981908Z 
2020-11-17T16:05:19.5982796Z HEAD is now at 1f91e36 Adding the current Azure sample pif for testing
2020-11-17T16:05:19.6041690Z Downloaded linked artifact customer
2020-11-17T16:05:19.6045637Z Starting artifacts download...
2020-11-17T16:05:19.8973944Z ##[error]Downloading artifacts failed: System.InvalidOperationException: NotFound
   at Microsoft.VisualStudio.Services.Agent.Worker.Release.Artifacts.GitHubHttpClient.GetUserRepo(String accessToken, String repositoryName)
   at Microsoft.VisualStudio.Services.Agent.Worker.Release.Artifacts.GitHubArtifact.GetArtifactDetails(IExecutionContext context, AgentArtifactDefinition agentArtifactDefinition)
   at Microsoft.VisualStudio.Services.Agent.Worker.Release.ReleaseJobExtension.<>c__DisplayClass46_0.<ConvertToArtifactDefinition>b__1()
   at Microsoft.VisualStudio.Services.Agent.Worker.Release.RetryExecutor.Execute(Action action)
   at Microsoft.VisualStudio.Services.Agent.Worker.Release.ReleaseJobExtension.ConvertToArtifactDefinition(AgentArtifactDefinition agentArtifactDefinition, IExecutionContext executionContext, IArtifactExtension extension)
   at Microsoft.VisualStudio.Services.Agent.Worker.Release.ReleaseJobExtension.DownloadArtifacts(IExecutionContext executionContext, IList`1 agentArtifactDefinitions, String artifactsWorkingFolder)
   at Microsoft.VisualStudio.Services.Agent.Worker.Release.ReleaseJobExtension.DownloadArtifactsAndCommitsAsync(IExecutionContext executionContext, Object data)
2020-11-17T16:05:19.9024048Z ##[error]NotFound
2020-11-17T16:05:19.9030581Z ##[section]Finishing: Download Artifacts

2020-11-17T16:16:49.4928940Z ##[section]Starting: Download Artifacts
2020-11-17T16:16:49.5015430Z Creating artifacts directory: /rs-pkgs/scu-lin-pool-02/scu-lin-pool-02-agent-01/_work/r2/a
2020-11-17T16:16:49.5029415Z Created artifacts directory: /rs-pkgs/scu-lin-pool-02/scu-lin-pool-02-agent-01/_work/r2/a
2020-11-17T16:16:49.5029786Z Downloading artifact
2020-11-17T16:16:49.5047116Z Starting artifacts download...
2020-11-17T16:16:49.8656733Z Downloading linked artifact customer of type GitHub...
2020-11-17T16:16:49.8678642Z Ensuring artifact folder /rs-pkgs/scu-lin-pool-02/scu-lin-pool-02-agent-01/_work/r2/a/customer exists and is clean.
2020-11-17T16:16:49.8695254Z Received GitHub artifact Details
2020-11-17T16:16:49.8826740Z Syncing repository: f7969199-a739-4394-ae52-c0c8c484218b (GitHub)
2020-11-17T16:16:49.8919207Z ##[command]git version
2020-11-17T16:16:49.9262373Z git version 2.17.1
2020-11-17T16:16:49.9339405Z ##[command]git init "/rs-pkgs/scu-lin-pool-02/scu-lin-pool-02-agent-01/_work/r2/a/customer"
2020-11-17T16:16:49.9417604Z Initialized empty Git repository in /rs-pkgs/scu-lin-pool-02/scu-lin-pool-02-agent-01/_work/r2/a/customer/.git/
2020-11-17T16:16:49.9454261Z ##[command]git remote add origin https://github.com/org/customer.git
2020-11-17T16:16:49.9516822Z ##[command]git config gc.auto 0
2020-11-17T16:16:49.9554636Z ##[command]git config --get-all http.https://github.com/org/customer.git.extraheader
2020-11-17T16:16:49.9585859Z ##[command]git config --get-all http.proxy
2020-11-17T16:16:49.9634927Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --tags --prune --progress --no-recurse-submodules origin
2020-11-17T16:16:50.6003397Z remote: Enumerating objects: 4, done.        
2020-11-17T16:16:50.6003830Z remote: Counting objects:  25% (1/4)        
2020-11-17T16:16:50.6012605Z remote: Counting objects:  50% (2/4)        
2020-11-17T16:16:50.6012978Z remote: Counting objects:  75% (3/4)        
2020-11-17T16:16:50.6013463Z remote: Counting objects: 100% (4/4)        
2020-11-17T16:16:50.6013803Z remote: Counting objects: 100% (4/4), done.        
2020-11-17T16:16:50.6014100Z remote: Compressing objects:  33% (1/3)        
2020-11-17T16:16:50.6014728Z remote: Compressing objects:  66% (2/3)        
2020-11-17T16:16:50.6015328Z remote: Compressing objects: 100% (3/3)        
2020-11-17T16:16:50.6015761Z remote: Compressing objects: 100% (3/3), done.        
2020-11-17T16:16:50.7456024Z remote: Total 4 (delta 0), reused 4 (delta 0), pack-reused 0        
2020-11-17T16:16:50.7555275Z From https://github.com/org/customer
2020-11-17T16:16:50.7555599Z  * [new branch]      master     -> origin/master
2020-11-17T16:16:50.7697514Z ##[command]git checkout --progress --force 1f91e36129daae68bff1287183d2d97df94f7d54
2020-11-17T16:16:50.7742605Z Note: checking out '1f91e36129daae68bff1287183d2d97df94f7d54'.
2020-11-17T16:16:50.7742894Z 
2020-11-17T16:16:50.7743126Z You are in 'detached HEAD' state. You can look around, make experimental
2020-11-17T16:16:50.7743462Z changes and commit them, and you can discard any commits you make in this
2020-11-17T16:16:50.7743845Z state without impacting any branches by performing another checkout.
2020-11-17T16:16:50.7744071Z 
2020-11-17T16:16:50.7744295Z If you want to create a new branch to retain commits you create, you may
2020-11-17T16:16:50.7744685Z do so (now or later) by using -b with the checkout command again. Example:
2020-11-17T16:16:50.7744930Z 
2020-11-17T16:16:50.7745454Z   git checkout -b <new-branch-name>
2020-11-17T16:16:50.7745838Z 
2020-11-17T16:16:50.7746040Z HEAD is now at 1f91e36 Adding the current Azure sample pif for testing
2020-11-17T16:16:50.7762906Z Downloaded linked artifact customer
2020-11-17T16:16:50.7764353Z Starting artifacts download...
2020-11-17T16:16:51.1015446Z ##[error]Downloading artifacts failed: System.InvalidOperationException: NotFound
   at Microsoft.VisualStudio.Services.Agent.Worker.Release.Artifacts.GitHubHttpClient.GetUserRepo(String accessToken, String repositoryName)
   at Microsoft.VisualStudio.Services.Agent.Worker.Release.Artifacts.GitHubArtifact.GetArtifactDetails(IExecutionContext context, AgentArtifactDefinition agentArtifactDefinition)
   at Microsoft.VisualStudio.Services.Agent.Worker.Release.ReleaseJobExtension.<>c__DisplayClass46_0.<ConvertToArtifactDefinition>b__1()
   at Microsoft.VisualStudio.Services.Agent.Worker.Release.RetryExecutor.Execute(Action action)
   at Microsoft.VisualStudio.Services.Agent.Worker.Release.ReleaseJobExtension.ConvertToArtifactDefinition(AgentArtifactDefinition agentArtifactDefinition, IExecutionContext executionContext, IArtifactExtension extension)
   at Microsoft.VisualStudio.Services.Agent.Worker.Release.ReleaseJobExtension.DownloadArtifacts(IExecutionContext executionContext, IList`1 agentArtifactDefinitions, String artifactsWorkingFolder)
   at Microsoft.VisualStudio.Services.Agent.Worker.Release.ReleaseJobExtension.DownloadArtifactsAndCommitsAsync(IExecutionContext executionContext, Object data)
2020-11-17T16:16:51.1059304Z ##[error]NotFound
2020-11-17T16:16:51.1069908Z ##[section]Finishing: Download Artifacts

[jeffpatton1971]

As I look at this more I am noticing that it only appears to happen with multiple github repositories. If I just add one github repo to the artifacts it is happy, doesn't matter which one (so no issue with the repo itself or its' configuration) as soon as I add the second is when things go sideways.

[jeffpatton1971]

I've created a new release with one github artifact and it worked, then add a second github artifact and it failed, these are on self-hosted pools. Running this against the microsoft hosted pools I don't hit this issue, which makes me wonder what the difference between the self-hosted and the microsoft-hosted pools are. As we need to run our own build servers, i wonder if there is potentially some dependency missing that we can add.

[jeffpatton1971]

Initially what I thought was that the artifacts were being downloaded but now as I take a closer look at those logs, it appears taht only the first artifact gets downloaded and the second fails with the NotFound message.

[anatolybolshakov]

Hi @jeffpatton1971 I didn't manage to reproduce it with two artifacts from different github repos and last version of self-hosted agent - could you please share your yaml file or provide some minimum reproducible sample?

[kartsm]

I am affected by this as well; we had a release pipeline that started breaking after we added a second GitHub artifact. We were using self-hosted agents running version 2.177.1.

@jeffpatton1971 My teammate proposed this workaround, and it worked for us:

• store GitHub credentials (username and personal access token) in a variable group
• add that variable group to the release pipeline
• add a task to the release pipeline to clone the repository into the same location that the release pipeline artifact would have appeared i.e. $(System.DefaultWorkingDirectory)/$(artifact_name).

[andy-mishechkin]

Hello, @jeffpatton1971
Is it still actual issue ? If yes - may you provide your yaml file or some minimum reproducible sample?
If it's already been fixed - can we close this ticket ?

Thank you

[andy-mishechkin]

Closed, because no any activity in last 3 days.
@jeffpatton1971, please feel free to reopen, if you'll have any updates