Azure-pipelines-tasks: Pipeline Caching - executable bit is unset after restore

[arjunguha]

Question, Bug, or Feature?
Type: Bug

Enter Task Name: CacheBeta

Environment

• Server - Azure Pipelines - https://dev.azure.com/arjunguha/umass-plasma/_build/results?buildId=48
• Agent - Hosted, Linux

Issue Description

I am building a Rust project using Azure Pipelines and using CacheBeta to cache the ~/.cargo directory, which contains executable files. When the cache is restored, the executable files do not appear to have the executable permission set.

For example, you can see a log here, where the ~/.cargo directory is restored, but the files in ~/.cargo/bin are not executable:

##[section]Starting: Cache Cargo and Yarn packages
==============================================================================
Task         : Cache (Beta)
Description  : Cache files between runs
Version      : 0.1.0
Author       : Microsoft Corporation
Help         : https://aka.ms/pipeline-caching-docs
==============================================================================
Resolving key `version4 | Linux`...
Resolved to `version4|Linux`.
Information, Getting a pipeline cache artifact with one of the following fingerprints:
Information, Fingerprint: `version4|Linux`
Information, There is a cache hit: `version4|Linux`
Information, Used scope: 1;81b2d833-c987-40d5-ae7c-71f4a772623a;refs/pull/18/merge;plasma-umass/decontainerization
Information, Missed on the following scopes: 1;81b2d833-c987-40d5-ae7c-71f4a772623a;refs/heads/fix-xi;plasma-umass/decontainerization, 1;81b2d833-c987-40d5-ae7c-71f4a772623a;refs/heads/master;plasma-umass/decontainerization
Entry found at fingerprint: `version4|Linux`
Information, DedupManifestArtifactClient will correlate http requests with X-TFS-Session 387a8845-b742-44f4-88be-3b7500a55b84
Information, ArtifactHttpRetryMessageHandler.SendAsync: https://vsblobprodcus3.vsblob.visualstudio.com/A6e26e063-0937-4cda-b3d1-f1ed1b5322e2/_apis/dedup/nodes/0B91EEE78E95BFCA45AC5AFCBC257CC4734646AD8634355A4C1C44FB07DBFCE802 attempt 1/6 failed with StatusCode RedirectMethod, IsRetryableResponse False
Information, Could not initialize dataport.
Information, Downloaded 0.0 MB out of 192.8 MB (0%).
Information, Downloaded 39.7 MB out of 192.8 MB (21%).
Information, Downloaded 192.8 MB out of 192.8 MB (100%).
Information, 
Download statistics:
Total Content: 192.8 MB
Physical Content Downloaded: 88.9 MB
Compression Saved: 103.9 MB
Local Caching Saved: 0.0 MB
Chunks Downloaded: 6,813
Nodes Downloaded: 0

Information, Download completed.
Cache restored.
##[section]Finishing: Cache Cargo and Yarn packages

Next step:

##[section]Starting: CmdLine
==============================================================================
Task         : Command line
Description  : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
Version      : 2.151.2
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
==============================================================================
Generating script.
========================== Starting Command Output ===========================
[command]/bin/bash --noprofile --norc /home/vsts/work/_temp/bdb3033b-5dd7-4163-9a56-05489048dc8f.sh
+ set -e
+ cd rust/compiler
+ ls -l /home/vsts/.cargo/bin
total 141088
-rw-rw-rw-+ 1 vsts docker  7014160 Aug 24 13:33 bindgen
-rw-rw-rw-+ 1 vsts docker 10734264 Aug 24 13:33 cargo
-rw-rw-rw-+ 1 vsts docker 10734264 Aug 24 13:33 cargo-clippy
-rw-rw-rw-+ 1 vsts docker 10734264 Aug 24 13:33 cargo-fmt
-rw-rw-rw-+ 1 vsts docker 10734264 Aug 24 13:33 cargo-miri
-rw-rw-rw-+ 1 vsts docker  8627352 Aug 24 13:33 cbindgen
-rw-rw-rw-+ 1 vsts docker 10734264 Aug 24 13:33 clippy-driver
-rw-rw-rw-+ 1 vsts docker 10734264 Aug 24 13:33 rls
-rw-rw-rw-+ 1 vsts docker 10734264 Aug 24 13:33 rustc
-rw-rw-rw-+ 1 vsts docker 10734264 Aug 24 13:33 rustdoc
-rw-rw-rw-+ 1 vsts docker 10734264 Aug 24 13:33 rustfmt
-rw-rw-rw-+ 1 vsts docker 10734264 Aug 24 13:33 rust-gdb
-rw-rw-rw-+ 1 vsts docker 10734264 Aug 24 13:33 rust-lldb
-rw-rw-rw-+ 1 vsts docker 10734264 Aug 24 13:33 rustup

I can address this problem by running chmod a+x ~/.cargo/bin/*. This seems to not invalidate the cache.

Task logs

logs_46.zip

[johnterickson]

Fix is in deployment train. CC @fadnavistanmay

[jni-]

@arjunguha I don't know if it helps, but setting AZP_CACHING_TAR creates a tar instead, which preserves permissions (and symlinks too, when caching a node_modules with a .bin, this is also important).

You can enable it this way :

variables:
  AZP_CACHING_TAR: true

There is a PR somewhere on this github with the details too, not sure how official this is yet, but it works for us on hosted agents with CacheBeta@1.

[johnterickson]

Yes, that is rolling out. We've not doc'd it yet because we have https://github.com/microsoft/azure-pipelines-agent/pull/2400.

[johnterickson]

Covered by #10925