Azure-pipelines-agent: Publishing Artifacts throws Forbidden sporadically with latest version (2.181.1)

[Jetski5822]

Agent Version and Platform

2.181.1, Windows Server 2016

Azure DevOps Type and Version

dev.azure.com with on-premises agents

If dev.azure.com, what is your organization name? https://dev.azure.com/tesco-colleague

What's not working?

After updating to the latest Azure Agent version, Publishing Artifacts seems to throw Forbidden exceptions regularly.

It starts to upload and then just fails, and other times it works fine. At the moment id say its 20% successful.

==============================================================================
Task         : Publish Pipeline Artifacts
Description  : Publish (upload) a file or directory as a named artifact for the current run
Version      : 1.2.3
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/publish-pipeline-artifact
==============================================================================
Artifact name input: assets
Uploading pipeline artifact from E:\agents\TPC003548_A2\_work\150\a\build_hram for build #32833
ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session 65fa8555-43fe-45c9-a97e-a8418f36a325
DedupManifestArtifactClient will correlate http requests with X-TFS-Session 65fa8555-43fe-45c9-a97e-a8418f36a325
13 files processed.
Processed 13 files from E:\agents\TPC003548_A2\_work\150\a\build_hram successfully.
Uploading 13 files from directory E:\agents\TPC003548_A2\_work\150\a\build_hram.
Uploaded 599,073 out of 48,807,646 bytes.
ArtifactHttpRetryMessageHandler.SendAsync: https://vsblobprodsu6weu.vsblob.visualstudio.com/A5fe323aa-e1d9-4317-a532-5784c5ec8eef/_apis/dedup/chunks attempt 1/6 failed with StatusCode Forbidden, IsRetryableResponse False
ArtifactHttpRetryMessageHandler.SendAsync: https://vsblobprodsu6weu.vsblob.visualstudio.com/A5fe323aa-e1d9-4317-a532-5784c5ec8eef/_apis/dedup/chunks attempt 1/6 failed with StatusCode Forbidden, IsRetryableResponse False
Upload failure. Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException: Forbidden
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.HandleResponseAsync(HttpResponseMessage response, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpMethod method, IEnumerable`1 additionalHeaders, Guid locationId, Object routeValues, ApiResourceVersion version, HttpContent content, IEnumerable`1 queryParameters, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Content.Common.AsyncHttpRetryHelper`1.InvokeAsync(CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Content.Common.ExceptionExtensions.ReThrow(Exception ex)
   at Microsoft.VisualStudio.Services.Content.Common.AsyncHttpRetryHelper`1.InvokeAsync(CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreHttpClient.PutChunksAsync(Dictionary`2 allChunks, KeepUntilBlobReference keepUntil, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreClient.UploadSession.<>c__DisplayClass29_1.<<UploadChildrenAsync>b__2>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreClient.UploadSession.UploadChildrenAsync(DedupNode node, KeepUntilReceipt[] existingReceipts, DedupNodeChildrenNeedAction childrenAction, Lazy`1 file, String path, Nullable`1 offset, String indent, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreClient.UploadSession.<>c__DisplayClass28_2.<<UploadNodeAsyncInternal>b__2>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreClient.UploadSession.UploadNodeAsyncInternal(NodeDedupIdentifier nodeId, DedupNode node, Lazy`1 file, String path, Nullable`1 offset, String indent, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreClient.UploadSession.UploadNodeAsync(DedupNode node, String path, Lazy`1 file, Nullable`1 offset, String indent, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreClient.UploadSession.UploadChildrenAsync(DedupNode node, KeepUntilReceipt[] existingReceipts, DedupNodeChildrenNeedAction childrenAction, Lazy`1 file, String path, Nullable`1 offset, String indent, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreClient.UploadSession.<>c__DisplayClass28_2.<<UploadNodeAsyncInternal>b__2>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreClient.UploadSession.UploadNodeAsyncInternal(NodeDedupIdentifier nodeId, DedupNode node, Lazy`1 file, String path, Nullable`1 offset, String indent, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreClient.UploadSession.UploadNodeAsync(DedupNode node, String path, Lazy`1 file, Nullable`1 offset, String indent, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupManifestArtifactClient.<>c__DisplayClass20_2.<<PublishAsync>b__0>d.MoveNext()
DedupManifestArtifactClient will correlate http requests with X-TFS-Session 65fa8555-43fe-45c9-a97e-a8418f36a325

[Jetski5822]

I have a worker log file too... could I email it to someone instead of putting it here...?

[alexander-smolyakov]

I have a worker log file too... could I email it to someone instead of putting it here...?

Hi @Jetski5822, sure, here's my email - [email protected]

[Jetski5822]

Note: Submitted Worker file. Let me know if you didnt get it. - N

[Jetski5822]

So; I have been monitoring this for a couple days now and this is what I've noticed.

publishbuildartifacts = Works fine, and every time
publishpipelineartifact = Fails sporadically with forbidden.

`

Starting: Publish Artifacts

Task : Publish build artifacts
Description : Publish build artifacts to Azure Pipelines or a Windows file share
Version : 1.158.3
Author : Microsoft Corporation

Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/publish-build-artifacts

Async Command Start: Upload Artifact
Uploading 13 files
Uploading 'assets/build/Tesco.Clocking.APIAdaptor.Web.7z' (33%)
Uploading 'assets/build/Tesco.Clocking.APIAdaptor.Web.7z' (66%)
Uploading 'assets/build/Tesco.Clocking.APIAdaptor.Web.7z' (100%)
Uploading 'assets/build/Tesco.TAM.Web.7z' (25%)
Uploading 'assets/build/Tesco.TAM.Web.7z' (50%)
Uploading 'assets/build/Tesco.HRAM.Web.7z' (25%)
Uploading 'assets/build/Tesco.HRAM.Web.7z' (50%)
Uploading 'assets/build/Tesco.HRAM.Web.7z' (75%)
Uploading 'assets/build/Tesco.HRAM.Web.7z' (100%)
Uploading 'assets/build/Tesco.TAM.WorkerService.7z' (50%)
Uploading 'assets/build/Tesco.TAM.WorkerService.7z' (100%)
File upload succeed.
Upload 'E:\agents\TPC003548_A2_work\150\a\build_hram' to file container: '#/8317454/assets'
Associated artifact 20003 with build 33010
Async Command End: Upload Artifact
Finishing: Publish Artifacts

`

[Jetski5822]

So; I have been monitoring this for a couple days now and this is what I've noticed.

publishbuildartifacts = Works fine, and every time
publishpipelineartifact = Fails sporadically with forbidden.

Starting: Publish Artifacts
==============================================================================
Task         : Publish build artifacts
Description  : Publish build artifacts to Azure Pipelines or a Windows file share
Version      : 1.158.3
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/publish-build-artifacts
==============================================================================
Async Command Start: Upload Artifact
Uploading 13 files
Uploading 'assets/build/Tesco.Clocking.APIAdaptor.Web.7z' (33%)
Uploading 'assets/build/Tesco.Clocking.APIAdaptor.Web.7z' (66%)
Uploading 'assets/build/Tesco.Clocking.APIAdaptor.Web.7z' (100%)
Uploading 'assets/build/Tesco.TAM.Web.7z' (25%)
Uploading 'assets/build/Tesco.TAM.Web.7z' (50%)
Uploading 'assets/build/Tesco.HRAM.Web.7z' (25%)
Uploading 'assets/build/Tesco.HRAM.Web.7z' (50%)
Uploading 'assets/build/Tesco.HRAM.Web.7z' (75%)
Uploading 'assets/build/Tesco.HRAM.Web.7z' (100%)
Uploading 'assets/build/Tesco.TAM.WorkerService.7z' (50%)
Uploading 'assets/build/Tesco.TAM.WorkerService.7z' (100%)
File upload succeed.
Upload 'E:\agents\TPC003548_A2\_work\150\a\build_hram' to file container: '#/8317454/assets'
Associated artifact 20003 with build 33010
Async Command End: Upload Artifact
Finishing: Publish Artifacts

[alexander-smolyakov]

@Jetski5822, could you please tell us which version of the agent you used before the update?

[Jetski5822]

@alexander-smolyakov Looks like we went

2.177.1 (2021-01-24) = Working
2.181.0 (2021-01-25) = Sporadically not working
2.181.1 (2021-02-08) = Sporadically not working (same as above)

[mjroghelia]

cc @fadnavistanmay In case you have any insights on this one.

[Jetski5822]

Just to give you an idea of the weird behaviour;

Publish Pipeline Artifacts:

Attempt1:

Uploading pipeline artifact from E:\agents\TPC003548_A2\_work\150\a\build_hram for build #33212 ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session d65e5b6b-f3ae-4cf8-b8b7-7f0e19155fc3 DedupManifestArtifactClient will correlate http requests with X-TFS-Session d65e5b6b-f3ae-4cf8-b8b7-7f0e19155fc3 13 files processed. Processed 13 files from E:\agents\TPC003548_A2\_work\150\a\build_hram successfully. ArtifactHttpRetryMessageHandler.SendAsync: https://vsblobprodsu6weu.vsblob.visualstudio.com/A5fe323aa-e1d9-4317-a532-5784c5ec8eef/_apis/dedup/chunks attempt 1/6 failed with StatusCode Forbidden, IsRetryableResponse False Uploading 13 files from directory E:\agents\TPC003548_A2\_work\150\a\build_hram. Uploaded 3,981,326 out of 48,817,418 bytes. Upload failure. Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException: Forbidden

Attemp2:

Uploading pipeline artifact from E:\agents\TPC003548_A2\_work\150\a\build_hram for build #33212 ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session 777a02ae-164d-49a6-a226-315419608d1a DedupManifestArtifactClient will correlate http requests with X-TFS-Session 777a02ae-164d-49a6-a226-315419608d1a 13 files processed. Processed 13 files from E:\agents\TPC003548_A2\_work\150\a\build_hram successfully. ArtifactHttpRetryMessageHandler.SendAsync: https://vsblobprodsu6weu.vsblob.visualstudio.com/A5fe323aa-e1d9-4317-a532-5784c5ec8eef/_apis/dedup/chunks attempt 1/6 failed with StatusCode Forbidden, IsRetryableResponse False ArtifactHttpRetryMessageHandler.SendAsync: https://vsblobprodsu6weu.vsblob.visualstudio.com/A5fe323aa-e1d9-4317-a532-5784c5ec8eef/_apis/dedup/chunks attempt 1/6 failed with StatusCode Forbidden, IsRetryableResponse False ArtifactHttpRetryMessageHandler.SendAsync: https://vsblobprodsu6weu.vsblob.visualstudio.com/A5fe323aa-e1d9-4317-a532-5784c5ec8eef/_apis/dedup/chunks attempt 1/6 failed with StatusCode Forbidden, IsRetryableResponse False Upload failure. Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException: Forbidden

Publish build artifacts:

Async Command Start: Upload Artifact Uploading 11 files Uploading 'assets/build/Tesco.HRAM.Web.zip' (12%) Uploading 'assets/build/Tesco.HRAM.Web.zip' (25%) Uploading 'assets/build/Tesco.HRAM.Unipay.HostedService.zip' (33%) Uploading 'assets/build/Tesco.HRAM.Unipay.HostedService.zip' (66%) Uploading 'assets/build/Tesco.HRAM.Unipay.HostedService.zip' (100%) Uploading 'assets/build/Tesco.HRAM.Clocking.Web.zip' (50%) Uploading 'assets/build/Tesco.HRAM.Clocking.Web.zip' (100%) Uploading 'assets/build/Tesco.HRAM.Web.zip' (37%) Uploading 'assets/build/Tesco.HRAM.Web.zip' (50%) Uploading 'assets/build/Tesco.HRAM.Web.zip' (62%) Uploading 'assets/build/Tesco.HRAM.Web.zip' (75%) Uploading 'assets/build/Tesco.HRAM.Web.zip' (87%) Uploading 'assets/build/Tesco.HRAM.Web.zip' (100%) Uploading 'assets/supporting/devops.zip' (14%) Uploading 'assets/supporting/devops.zip' (28%) Uploading 'assets/supporting/devops.zip' (42%) Uploading 'assets/supporting/devops.zip' (57%) Uploading 'assets/supporting/devops.zip' (71%) Uploading 'assets/supporting/devops.zip' (85%) Uploading 'assets/build/tools.zip' (50%) Uploading 'assets/build/tools.zip' (100%) File upload succeed. Upload 'E:\agents\TPC003548_A4\_work\129\a\build_hram' to file container: '#/8343454/assets' Associated artifact 20115 with build 33222 Async Command End: Upload Artifact Finishing: Publish Artifacts

Ill update the agents to 2.181.2 and see if the error remains.

[Jetski5822]

Okay - I have finally found someone where I work on the network team, and its being blocked by the corporate proxy.

The error Forbidden is weird for this however.

Closing - Thanks team.

[fadnavistanmay]

Yes, in most of the cases, the proxy is the culprit!

[Jetski5822]

For anyone who (is) interested in the issue...

The proxy scans the files on the way out, and if it cant read or identify them it blocks them. When moving from Build to Pipeline artifacts, Pipelines uploads in a different manner to that of Build Artifacts. The proxy cant read it, and so block the stream.

The URL wasnt blocked; just the stream.

So - long story short, if you have this weird issue - Talk to the team managing your proxy, ask them to check on the URL https://vsblobprodsu6weu.vsblob.visualstudio.com and what segment is being blocked.