Azure-pipelines-tasks: Pipeline caching cannot be used on pre-existing directories

[jonhoo]

Question, Bug, or Feature?
Type: Bug

Enter Task Name: Cache@2

Environment

• Server - Azure Pipelines @ jonhoo/jonhoo, definition 15
• Agent - Hosted

Issue Description

I want to cache a compiled file stored in a system-managed directory. For example:

steps:
  - task: Cache@2
    inputs:
      key: cargo-deny | $(Agent.OS)
      path: /usr/share/rust/.cargo/bin
      cacheHitVar: CACHE_RESTORED
  - script: cargo install cargo-deny
    displayName: install cargo-deny
    condition: ne(variables.CACHE_RESTORED, 'true')

cargo install installs the requested program as /usr/share/rust/.cargo/bin/cargo-deny. The directory /usr/share/rust/.cargo/bin is managed by the system, with the permissions:

drwxrwxrwx+ 2 root root 4096 Jan 20 20:11 bin

It is also not empty by default, but instead includes some pre-installed (root-owned) files. Storing the cache works just fine with the task definition above, but restoring the cache does not. It errors (log) with:

Information, Downloaded 0.0 MB out of 40.5 MB (0%).
tar: .: Cannot utime: Operation not permitted
tar: .: Cannot change mode to rwxr-xr-x: Operation not permitted
tar: Exiting with failure status due to previous errors

This is presumably because the restore operation attempts to overwrite the entire bin directory, which it does not have permissions to do.

I attempted to set path to just the one file I would like to cache (bin/cargo-deny), but this fails when trying to store the cache (log) with

##[error]Please specify path to a directory, File path is not allowed. /usr/share/rust/.cargo/bin/cargo-deny is a file.

My current workaround is to prefix the cache task with

 - script: sudo rm -r /usr/share/rust/.cargo/bin

But that does not feel like it should be necessary?

[jonhoo]

You can see the workaround I've had to implement here:
https://github.com/jonhoo/flurry/blob/5f8d37f5db19789db2b2eafc00d9f140f242939d/azure-pipelines.yml#L11-L23

[johnterickson]

@jonhoo I handle this by changing the path that cargo install installs to: https://github.com/johnterickson/cargo2junit/blob/master/azure-pipelines.yml

That way, everything I need is all in the target folder. You can also see how I leverage cargo install-update to keep cargo-installed tools up-to-date.

Does that help?

[johnterickson]

My YAML is a bit of a convoluted mess because I was also experimenting with sccache 😅

[fadnavistanmay]

Looks like John's recommendation unblocked you. Closing the issue.

[jonhoo]

@fadnavistanmay Why is this being closed? This is still an issue that just happens to have a bad and undocumented workaround.

[evandrocoan]

Related to: https://github.com/actions/cache/issues/133 - Tracking: Permission Errors when restoring and creating cache.

Instead of removing the directory, you coudl change its permissions to the current user as on: (https://github.com/actions/cache/issues/133#issuecomment-599102035)

# Necessary for now for the cargo cache:
# https://github.com/actions/cache/issues/133#issuecomment-599102035
- name: Fix ~/.cargo directory permissions
  run: sudo chown -R $(whoami):$(id -ng) ~/.cargo/