Cache: Caching problem - unpacking tar

[mikart143]

Hello !
I found error as shown in #133
I tried to fix this by using windows as system.
Cache was made but the new error occured.

Run actions/cache@v1
Cache Size: ~102 MB (107455822 B)
C:\windows\System32\tar.exe -xz -f d:\a_temp\5bff5da6-b2e7-439a-9f1e-b8eb52fb391e\cache.tgz -C C:\Usersrunneradmin.cargoregistry
./index/github.com-1ecc6299db9ec823/.git/objects/pack/pack-a1b8e2d02676504008fea86ca360367fcee5bd43.idx: Can't unlink already-existing object
./index/github.com-1ecc6299db9ec823/.git/objects/pack/pack-a1b8e2d02676504008fea86ca360367fcee5bd43.pack: Can't unlink already-existing object
tar.exe: Error exit delayed from previous errors.
[warning]Tar failed with error: The process 'C:\windows\System32\tar.exe' failed with exit code 1. Ensure BSD tar is installed and on the PATH.

I am trying to cache cargo. I am using code from Action examples

[dhadka]

Hi @mikart143, could you please provide your full workflow file? The error Can't unlink already-existing object seems to indicate those files in the cargo registry already exist and the action does not have permission to delete them. Is cargo creating / populating the registry before the cache restore step is performed?

[mikart143]

It occure on security_audit job

name: Rust

on: 
  push:
   branches:
       - master

jobs:
  security_audit:
    runs-on: windows-latest
    steps:
      - uses: actions/checkout@master
      - name: Cache cargo registry
        uses: actions/cache@v1
        with:
          path: ~/.cargo/registry
          key: ${{ runner.os }}-cargo-registry-${{ hashFiles('**/Cargo.lock') }}
      - name: Cache cargo index
        uses: actions/cache@v1
        with:
          path: ~/.cargo/git
          key: ${{ runner.os }}-cargo-index-${{ hashFiles('**/Cargo.lock') }}
      - name: Cache cargo build
        uses: actions/cache@v1
        with:
          path: target
          key: ${{ runner.os }}-cargo-build-target-${{ hashFiles('**/Cargo.lock') }}
      - uses: actions/checkout@v1
      - uses: actions-rs/audit-check@v1
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
      # - uses: EmbarkStudios/cargo-deny-action@v0

  test:
    needs: [security_audit]
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest, macOS-latest]
        rust: [stable]

    steps:
    - uses: hecrj/setup-rust-action@v1
      with:
        rust-version: ${{ matrix.rust }}
    - uses: actions/checkout@master
    - name: Run tests
      run: cargo test --lib --verbose 
  docs:
    needs: [security_audit, test]
    runs-on: ubuntu-latest

    steps: 
    - uses: actions/checkout@v2
    - name: Generating Docs
      run: |
        cargo doc --no-deps
        mv target/doc docs
        echo '<meta http-equiv="refresh" content="0; url=syncer/index.html">' > docs/index.html
  build:
    needs: [security_audit, test]
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2
    - name: Build
      run: cargo build --verbose
    - name: Run tests
      run: cargo test --verbose

[mzabaluev]

This workflow run demonstrates the problem. Compared to the tar of the same repository index on Linux and Mac, the archive created on Windows is more than 1.5 times larger and unpacking fails with the "Can't unlink already-existing object" messages.

My guess is that tar on Windows does not handle hardlinks or symlinks properly?

[mzabaluev]

One thing that may help debugging this is if you dump the tar file contents with tar -tzv prior to unpacking.

[mzabaluev]

It's caused by https://github.com/actions/virtual-environments/issues/895 and here's proof.

On Linux the same pre-populated repository checkout exists too, but tar manages to overwrite it without an issue.

[mzabaluev]

It's caused by actions/virtual-environments#895

To clarify, this explains the particular failure as it was reported. This does not explain why tar.exe in Windows fails to overwrite files that were baked into the virtual environment image, when unpacking from an archive that it itself packaged these files into, while having no problems unpacking an archive that was created by GNU tar on Linux.

[mzabaluev]

This does not explain why tar.exe in Windows fails to overwrite files that were baked into the virtual environment image, when unpacking from an archive that it itself packaged these files into, while having no problems unpacking an archive that was created by GNU tar on Linux.

This may be a red herring since there is no intersection in the git pack files between the two checkouts, as tested here:

$ tar -tzv --xattrs --acls -f index-linux.tgz | grep pack
drwxrwxrwx  runner/docker      0 2020-05-25 14:52 ./github.com-1ecc6299db9ec823/.git/objects/pack/
-rwxrwxrwx  runner/docker 3974748 2020-05-18 12:11 ./github.com-1ecc6299db9ec823/.git/objects/pack/pack-e63d82649edaa0ff776e5243a52d12d6f7bf5f0d.idx
-r--r--r--  runner/docker 4679324 2020-05-25 14:52 ./github.com-1ecc6299db9ec823/.git/objects/pack/pack-f3820eff603c81a0e0019112c7acc8a883d383a4.pack
-rwxrwxrwx  runner/docker 43632056 2020-05-18 12:11 ./github.com-1ecc6299db9ec823/.git/objects/pack/pack-e63d82649edaa0ff776e5243a52d12d6f7bf5f0d.pack
-r--r--r--  runner/docker   365156 2020-05-25 14:52 ./github.com-1ecc6299db9ec823/.git/objects/pack/pack-f3820eff603c81a0e0019112c7acc8a883d383a4.idx

$ tar -tzv --xattrs --acls -f index-windows.tgz | grep pack
drwxrwxrwx  0/0               0 2020-05-25 14:52 ./github.com-1ecc6299db9ec823/.git/objects/pack/
-r--r--r--  0/0          410768 2020-05-25 14:52 ./github.com-1ecc6299db9ec823/.git/objects/pack/pack-41eccb34a2a6c9172486aee7624e0889c2612d80.idx
-r--r--r--  0/0         5085807 2020-05-25 14:52 ./github.com-1ecc6299db9ec823/.git/objects/pack/pack-41eccb34a2a6c9172486aee7624e0889c2612d80.pack
-r--r--r--  0/0         3930508 2020-05-17 12:26 ./github.com-1ecc6299db9ec823/.git/objects/pack/pack-4e4ff12a1cfe7b45aa90385fcebdbf6eb08bc15d.idx
-r--r--r--  0/0        43301476 2020-05-17 12:26 ./github.com-1ecc6299db9ec823/.git/objects/pack/pack-4e4ff12a1cfe7b45aa90385fcebdbf6eb08bc15d.pack

[mzabaluev]

Sure enough, repro'd with the Windows content repacked on Linux.

As long as files that come from the virtual environment image are being replaced, tar fails.

[mzabaluev]

I see no difference in ACLs between the pre-populated files:

Path   : Microsoft.PowerShell.Core\FileSystem::C:\Users\runneradmin\.cargo\registry\index\github.com-1ecc6299db9ec823\.
         git\objects\pack
Owner  : BUILTIN\Administrators
Group  : fv-az103\None
Access : BUILTIN\Administrators Allow  FullControl
         BUILTIN\Administrators Allow  FullControl
         NT AUTHORITY\SYSTEM Allow  FullControl
         CREATOR OWNER Allow  268435456
         BUILTIN\Users Allow  ReadAndExecute, Synchronize
         BUILTIN\Users Allow  AppendData
         BUILTIN\Users Allow  CreateFiles
Audit  : 
Sddl   : O:BAG:S-1-5-21-3063699593-698096204-2912911206-513D:(A;;FA;;;BA)(A;OICIID;FA;;;BA)(A;OICIID;FA;;;SY)(A;OICIIOI
         D;GA;;;CO)(A;OICIID;0x1200a9;;;BU)(A;CIID;LC;;;BU)(A;CIID;DC;;;BU)



Path   : Microsoft.PowerShell.Core\FileSystem::C:\Users\runneradmin\.cargo\registry\index\github.com-1ecc6299db9ec823\.
         git\objects\pack\pack-4e4ff12a1cfe7b45aa90385fcebdbf6eb08bc15d.idx
Owner  : BUILTIN\Administrators
Group  : fv-az103\None
Access : BUILTIN\Administrators Allow  FullControl
         NT AUTHORITY\SYSTEM Allow  FullControl
         BUILTIN\Users Allow  ReadAndExecute, Synchronize
Audit  : 
Sddl   : O:BAG:S-1-5-21-3063699593-698096204-2912911206-513D:(A;ID;FA;;;BA)(A;ID;FA;;;SY)(A;ID;0x1200a9;;;BU)



Path   : Microsoft.PowerShell.Core\FileSystem::C:\Users\runneradmin\.cargo\registry\index\github.com-1ecc6299db9ec823\.
         git\objects\pack\pack-4e4ff12a1cfe7b45aa90385fcebdbf6eb08bc15d.pack
Owner  : BUILTIN\Administrators
Group  : fv-az103\None
Access : BUILTIN\Administrators Allow  FullControl
         NT AUTHORITY\SYSTEM Allow  FullControl
         BUILTIN\Users Allow  ReadAndExecute, Synchronize
Audit  : 
Sddl   : O:BAG:S-1-5-21-3063699593-698096204-2912911206-513D:(A;ID;FA;;;BA)(A;ID;FA;;;SY)(A;ID;0x1200a9;;;BU)

and files created by a script in a runner job:

    Directory: C:\Users\runneradmin\.cargo\registry

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---           5/25/2020  2:51 PM              0 foo


Path   : Microsoft.PowerShell.Core\FileSystem::C:\Users\runneradmin\.cargo\registry\foo
Owner  : BUILTIN\Administrators
Group  : fv-az103\None
Access : BUILTIN\Administrators Allow  FullControl
         NT AUTHORITY\SYSTEM Allow  FullControl
         BUILTIN\Users Allow  ReadAndExecute, Synchronize
Audit  : 
Sddl   : O:BAG:S-1-5-21-3063699593-698096204-2912911206-513D:(A;ID;FA;;;BA)(A;ID;FA;;;SY)(A;ID;0x1200a9;;;BU)




    Directory: C:\Users\runneradmin\.cargo\registry

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d----           5/25/2020  2:51 PM                bar


Path   : Microsoft.PowerShell.Core\FileSystem::C:\Users\runneradmin\.cargo\registry\bar
Owner  : BUILTIN\Administrators
Group  : fv-az103\None
Access : BUILTIN\Administrators Allow  FullControl
         BUILTIN\Administrators Allow  FullControl
         NT AUTHORITY\SYSTEM Allow  FullControl
         CREATOR OWNER Allow  268435456
         BUILTIN\Users Allow  ReadAndExecute, Synchronize
         BUILTIN\Users Allow  AppendData
         BUILTIN\Users Allow  CreateFiles
Audit  : 
Sddl   : O:BAG:S-1-5-21-3063699593-698096204-2912911206-513D:(A;;FA;;;BA)(A;OICIID;FA;;;BA)(A;OICIID;FA;;;SY)(A;OICIIOI
         D;GA;;;CO)(A;OICIID;0x1200a9;;;BU)(A;CIID;LC;;;BU)(A;CIID;DC;;;BU)

[mzabaluev]

This has nothing to do with ACLs or symlinks (AKA junctions), and everything to do with read-only files: https://github.com/actions/virtual-environments/issues/936

[mzabaluev]

A solution suggested in https://github.com/actions/virtual-environments/issues/936#issuecomment-634068778 is to use the tar executable bundled with Git, and use the --force-local option to interpret Windows path names as local file paths rather than remote tape drive servers :old_key:.

[lukka]

Any updates on this long standing issue?
Look like @mzabaluev has done the needed investigation and found a potential solution.

This affects any consumer of the GitHub action actions/cache and of the @actions/cache NPM package.

It affects for example run-vpkg.