Azure-pipelines-tasks: Download Pipeline Artifacts - System.OutOfMemoryException

[jonathaneckman]

Question, Bug, or Feature?
Type: Bug

Enter Task Name: Download Pipeline Artifacts

Environment

• Server - Azure Pipelines

  • If using Azure Pipelines, provide the account name, team project name, build definition name/build number:
    
    
    
    • I'd prefer to email you this info.
    
    

• Agent - Hosted or Private:

  • Private Windows VM running agent version 2.155.1

Issue Description

About 60% of our Download Pipeline Artifacts tasks are failing with the exception: Exception of type 'System.OutOfMemoryException. The private agent has 7 GB RAM and is less than 50% utilized during a pipeline run.

Task logs

Starting: Download Artifact |  
-- | --
  | ============================================================================== |  
  | Task         : Download Pipeline Artifact |  
  | Description  : Download Pipeline Artifact |  
  | Version      : 1.1.3 |  
  | Author       : Microsoft Corporation |  
  | Help         : Download named artifact from a pipeline to a local path. |  
  | ============================================================================== |  
  | Download from the specified build: #937 |  
  | Download artifact to: C:\servername\_work\12/ |  
  | Downloading 1 pipeline artifacts... |  
  | Information, DedupManifestArtifactClient will correlate http requests with X-TFS-Session 38163401-19ce-417c-9bcd-e7f4f73b167a |  
  | Information, Start downloading artifact - drop |  
  | Information, Minimatch patterns: [**] |  
  | Information, ArtifactHttpRetryMessageHandler.SendAsync: https://vsblobprodeus21.vsblob.visualstudio.com/A24fdab85-8310-4bf7-8853-2d9fabb509d5/_apis/dedup/nodes/9D767611D7CDAE106570D6B48B077E65EAD61E6C3F96CB909DD06911736711B802 attempt 1/6 failed with StatusCode RedirectMethod, IsRetryableResponse False |  
  | Information, Filtered 700 files from the Minimatch filters supplied. |  
  | Information, Could not initialize dataport. |  
  | Information, Downloaded 0.0 MB out of 1,111.6 MB (0%). |  
  | Information, Downloaded 260.0 MB out of 1,111.6 MB (23%). |  
  | Warning, [https://uh5vsblobprodeus2111.blob.core.windows.net/db24fdab8583104bf788532d9fabb509d5/4EC0816BB54CB442564996189CEA65F8C5FC3A79A05E4F60C108DECBD57CCA0701?sv=2017-04-17&sr=b&sig=SpoY6Shb8bJTM9C8PZwdXOZqulPN2NUa9jYAeY62pl0%3D&spr=https&se=2019-08-03T20%3A18%3A27Z&sp=r&rscl=x-e2eid-6c2f0c7b-f4ab4b64-936e1c18-d902e5e7-session-38163401-19ce417c-9bcde7f4-f73b167a] Try 1/5, non-retryable exception caught. Throwing. Details: |  
  | No LastRequestResponse on exception OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. |  
  | at Microsoft.VisualStudio.Services.BlobStore.Common.ByteArrayPool.CreateNew(Int32 bufferSize) |  
  | at Microsoft.VisualStudio.Services.BlobStore.Common.ByteArrayPool.<>c__DisplayClass2_0.<.ctor>b__0() |  
  | at Microsoft.VisualStudio.Services.BlobStore.Common.Pool`1.Get() |  
  | at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreHttpClient.ReadResponseAsync(Boolean knownToBeCompressed, Uri redirect, HttpResponseMessage responseMessage, CancellationToken cancellationToken) |  
  | at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreHttpClient.<>c__DisplayClass59_0.<<HandleRedirectAsync>b__0>d.MoveNext() |  
  | --- End of stack trace from previous location where exception was thrown --- |  
  | at Microsoft.VisualStudio.Services.Content.Common.AsyncHttpRetryHelper`1.InvokeAsync(CancellationToken cancellationToken) |  
  | Information, Downloaded 910.4 MB out of 1,111.6 MB (82%). |  
  | ##[error]One or more errors occurred. (Exception of type 'System.OutOfMemoryException' was thrown.) |  
  | Finishing: Download Artifact

[jahsu-MSFT]

Can you please provide info of your VM? Does it run on 32bit or 64bit environment?

[jonathaneckman]

1. I am using this Azure Marketplace image:

Publisher: MicrosoftVisualStudio
Offer: VisualStudio
Sku: VS-2017-Comm-Latest-Preview-WS2016

[jonathaneckman]

This bug is killing us. Is there a known good Windows private agent configuration we can use to rebuild our agents to get around this?

[fadnavistanmay]

For the private agent question - @TingluoHuang would be a better person to answer. We have released a fix in our client libraries which could mitigate the memory issue, it will be release in the agent 2.156.0. Thanks.

[jonathaneckman]

I deployed a 2nd agent using a VS 2019 VM image from Azure marketplace but am seeing the same issue. When 2.156 comes out of prerelease I'll update and test again.

[fadnavistanmay]

Hi @jonathaneckman - 156 should be release this week. I'll update the the thread once it's out. Thanks.

[jonathaneckman]

This seems to be resolved. I learned it was caused by using a 32 bit agent on a 64 bit VM.