Azure-pipelines-tasks: CacheBeta task fails with "Method Not Allowed" error during restore

[Cosmius]

Required Information

Type: Bug
Task Name: CacheBeta

Environment

• Azure Pipelines, multiple builds, provided below
• Hosted Agent multiple platforms:

It is a public project, you can access them from the URL.

Issue Description

VssServiceResponseException when restoring cache and the job failed

Build URL: https://dev.azure.com/cosmiafu/gugugu/_build/results?buildId=36
Job: Windows
Probable step causing the problem: Cache stack binary

Method Not Allowed when restoring cache and the job failed

Build URL: https://dev.azure.com/cosmiafu/gugugu/_build/results?buildId=36
Job: macOS
Probable step causing the problem: Cache stack binary

[danapoklepovich]

Also noticing the Method Not Allowed, was working ~24 hrs ago.

[tjhowse]

I'm also experiencing ##[error]Method Not Allowed on the CacheBeta@0 task when attempting to restore a cache. I tried to fix it by changing the cache key but I was unsuccessful.

[tjhowse]

I am no longer experiencing this error, however I haven't had a successful build so I cannot confirm the cache is working as it was before.

[danapoklepovich]

seems to be working again for me with a successful build! 🥇

[cocreature]

We sadly still see the Method Not Allowed error on our CI, example: https://dev.azure.com/digitalasset/daml/_build/results?buildId=12863&view=logs

[tharidu]

We are experiencing "Exception of type 'Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException' was thrown." error thrown from CacheBeta task as well.

[johanbenschop]

We're seeing the second issue since this morning too. The task version reports as 0.1.0 and the current agent version is 2.154.3.

Error message ##[error]Exception of type 'Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException' was thrown.

We enabled continue on error to prevent this from breaking our builds.

[willsmythe]

@Cosmius - FYI ... I moved the throttling problem to a separate issue (#11060) to make it easier to track.

[willsmythe]

Quick update: this issue is under investigation. There is currently no known workaround. Best option right now is to temporarily disable (or remove) the task. We are working to identify the best fix and will report back ASAP.

[baywet]

Getting the same error, here is the log with diagnostics enabled if it can help.

##[debug]Evaluating condition for step: 'Cache Npm' |  
-- | --
  | ##[debug]Evaluating: SucceededNode() |  
  | ##[debug]Evaluating SucceededNode: |  
  | ##[debug]=> True |  
  | ##[debug]Result: True |  
  | Starting: Cache Npm |  
  | ============================================================================== |  
  | Task         : Cache (Beta) |  
  | Description  : Cache files between runs |  
  | Version      : 0.1.0 |  
  | Author       : Microsoft Corporation |  
  | Help         : https://aka.ms/pipeline-caching-docs |  
  | ============================================================================== |  
  | ##[debug]Dedup parallelism: 192 |  
  | Information, Using the following fingerprint argument (Windows_NT) as a string value. |  
  | Information, Using the following fingerprint argument (D:\a\1\s/package-lock.json) as a file. |  
  | Information, Generated fingerprint: E97BC7208F7CAE6D627D148FA65CC8A7FBFA78FBBE43D0F154056CA219B3C0DB00 |  
  | Information, Getting a pipeline cache artifact with the following fingerprint: E97BC7208F7CAE6D627D148FA65CC8A7FBFA78FBBE43D0F154056CA219B3C0DB00 |  
  | ##[error]Exception of type 'Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException' was thrown. |  
  | ##[debug]Processed: ##vso[task.logissue type=error;]Exception of type 'Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException' was thrown. |  
  | ##[debug]Processed: ##vso[task.complete result=Failed;] |  
  | ##[debug]   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.HandleResponseAsync(HttpResponseMessage response, CancellationToken cancellationToken) |  
  | at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken) |  
  | at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken) |  
  | at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpMethod method, IEnumerable`1 additionalHeaders, Guid locationId, Object routeValues, ApiResourceVersion version, HttpContent content, IEnumerable`1 queryParameters, Object userState, CancellationToken cancellationToken) |  
  | at Microsoft.VisualStudio.Services.PipelineCache.WebApi.PipelineCacheClient.GetPipelineCacheArtifactAsync(GetPipelineCacheArtifactOptions getOptions, CancellationToken cancellationToken) |  
  | at Microsoft.VisualStudio.Services.PipelineCache.WebApi.PipelineCacheClient.<>c__DisplayClass9_0.<<GetPipelineCacheArtifactAsync>b__1>d.MoveNext() |  
  | --- End of stack trace from previous location where exception was thrown --- |  
  | at Microsoft.VisualStudio.Services.BlobStore.Common.Telemetry.BlobStoreClientTelemetry.MeasureActionAsync[TResult](BlobStoreTelemetryRecord record, Func`1 actionAsync) |  
  | at Microsoft.VisualStudio.Services.PipelineCache.WebApi.PipelineCacheClient.GetPipelineCacheArtifactAsync(GetPipelineCacheArtifactOptions getOptions, CancellationToken cancellationToken, PipelineCacheTelemetryRecord cacheRecordOptional) |  
  | at Agent.Plugins.PipelineCache.PipelineCacheServer.DownloadAsync(AgentTaskPluginExecutionContext context, IEnumerable`1 key, String path, String salt, String cacheHitVariable, CancellationToken cancellationToken) |  
  | at Agent.Plugins.PipelineCache.RestorePipelineCacheV0.ProcessCommandInternalAsync(AgentTaskPluginExecutionContext context, String path, String keyStr, String salt, CancellationToken token) |  
  | at Agent.Plugins.PipelineCache.PipelineCacheTaskPluginBase.RunAsync(AgentTaskPluginExecutionContext context, CancellationToken token) |  
  | at Agent.PluginHost.Program.Main(String[] args)

[garyverhaegen-da]

We have implemented a workaround that allows our builds to keep running while the cache is broken, whilst at the same time keeping our build able to exploit the cache as soon as it gets fixed.

Here is how it works.

1. Any job that uses caching is marked as continueOnError: true (example).
2. Immediately following the cache step, we add a new bash step that checks if the cache step has failed; if it has, we reset the state of the current job to succeeded and create a new variable that flags caching has failed (example).
3. All of the subsequent cache-related operations (unpacking, packing) are conditional on the cacheFailed variable.
4. At the very end of the job, we emit an output variable that says the job reached that point, i.e. was actually successful (example). This is required because the "post-job" cache action will run and fail, and, because that one is post-job, we can't "catch" it with an additional script the way we did in step 2.
5. We added an additional job that dependsOn all the jobs that use the cache (example) and checks for that magic "job actually succeeded" output variable produced in step 4. This is required because we still want the entire job to _fail_ if one of the jobs that had a cache failed for another reason (i.e. this partially undoes the "continueOnError" flag).

[johnterickson]

We've found an issue in the REST version negotiation logic that appears between the AT and DT stages of deployment. DT is now proceeding so, if our understanding is right, this should be working again within the hour.

[willsmythe]

This problem should be fixed now. If not, let us know. Sorry again for the inconvenience.

[tjhowse]

Confirmed to be all working here. Thanks!