Yarn: When using an offline cache, yarn upgrade adds the new package to the cache but does not remove the old one

[rmacklin]

Do you want to request a feature or report a bug?

Bug

What is the current behavior?

If you're using an in-repo offline cache (https://yarnpkg.com/blog/2016/11/24/offline-mirror), when you run yarn upgrade <package>, the new version of <package> will be added to the offline cache, but the old version won't be removed (even if it is not used by any other packages). It should be removed, so that manual deletion is not required to prevent the cache from growing to contain many unused packages over time.

If the current behavior is a bug, please provide the steps to reproduce.

Reproduction repository: https://github.com/rmacklin/yarn-issue-2109-repro

Steps to reproduce:

1. git clone [email protected]:rmacklin/yarn-issue-2109-repro.git
2. cd yarn-issue-2109-repro
3. node yarn-0.17.10.js upgrade left-pad
4. Observe:

This added a new, untracked tarball for the new version of left-pad in the
cache:

On branch master
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    modified:   package.json
    modified:   yarn.lock

Untracked files:
  (use "git add <file>..." to include in what will be committed)

    npm-packages-offline-cache/left-pad-1.1.3.tgz

no changes added to commit (use "git add" and/or "git commit -a")

However, the old npm-packages-offline-cache/left-pad-1.1.2.tgz tarball was _not_ removed even though it's no longer used. This means that it has to be manually deleted (an easy step to forget) in order to prevent the offline cache from growing to contain many unused packages over time.

What is the expected behavior?

The old, unused tarball should be removed, so that people don't have to remember to remove it from the repo. It would be fine if this behavior is behind an option or a .yarnrc setting.

Please mention your node.js, yarn and operating system version.

Node v6.2.0, yarn v0.17.10, Mac OSX 10.9.5

[bestander]

This was intentional for a monorepo when the offline cache is shared between multiple package.json/yarn.lock files.
I understand your concern, yarn should probably have a command or setting that would clean up orphan files from monorepo.
PR and discussion is welcome.

In the meantime cleanup can be done with a very basic script that parses yarn.lock file.

[talaviss]

How does Yarn supports offline cache when using monorepo with multiple yarn.lock files? Does is allow to upgrade the offline cache?

[bestander]

We have a .yarnrc file at the root of the monorepo with yarn-offline-mirror setting.
And then we just run yarn install in any of the subfolders and they all reuse the same offline mirror folder.

[dguo]

Thanks for the explanation, @bestander.

A built-in clean up ability would be great, since I think it's the last step towards making it trivial to keep package.json, node_modules, yarn.lock, and the offline cache totally in sync.

A .yarnrc setting to toggle it on makes sense to me because 1) the developer wouldn't need to remember to use a command option and 2) it could automatically apply to both the upgrade and remove commands.

Of course, that doesn't preclude making it a command option as well.

[bestander]

@dguo I see this may be useful for the community with single offline mirror per lockfile.
It should be fairly easy to implement as a standalone command and call it automatically after every change to the lockfile.
Do you want to try driving this feature from start to finish?
If yes, go ahead, send an RFC

[rmm5t]

In the meantime, is there a equivalent command to manually prune the offline cache until https://github.com/yarnpkg/rfcs/issues/49 is implemented?

The only thing I can think of is to manually delete the directory referenced by the yarn-offline-mirror setting and then re-run yarn install again to rebuild based on the up-to-date yarn.lock.

[bestander]

It can be a very simple grep of yarn.lock file with xargs that copies the
files to some other location temporarily and then removing the remaining
files.
Rerunning a fresh install won't help because yarn.lock refers to files and
not http urls, it will fail install.

On 15 February 2017 at 18:51, Ryan McGeary [e-mail] wrote:

In the meantime, is there a equivalent command to manually prune the
offline cache until yarnpkg/rfcs#49
https://github.com/yarnpkg/rfcs/pull/49 is implemented?

The only thing I can think of is to manually delete the directory
referenced by the yarn-offline-mirror setting and then re-run yarn install
again to rebuild based on the up-to-date yarn.lock.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/yarnpkg/yarn/issues/2109#issuecomment-280102354, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACBdWOyynTdtCKwBW93lglJDOtrriUo5ks5rc0kzgaJpZM4LCFft
.

[rmm5t]

@bestander Hmm 🤔

Rerunning a fresh install won't help because yarn.lock refers to files and not http urls, it will fail install.

Interesting, but it seems to work for me. Are the files getting pulled automatically from the global cache under ~/Library/Caches/Yarn and the yarn-offline-mirror directory is automatically re-built from that?

For example, I just wrote a quick bash function to automate this.

function yarn-prune {
  offline_dir=`yarn config get yarn-offline-mirror`
  if [[ "$offline_dir" == "undefined" ]]; then
    yarn prune $@
  elif [ -d "$offline_dir" ]; then
    read -r -p "Are you sure you want to delete and rebuild $offline_dir? [y/N] " response
    if [[ "$response" =~ ^([yY][eE][sS]|[yY])+$ ]]; then
      rm -r $offline_dir
      yarn install
    else
      return 1
    fi
  else
    echo "$offline_dir is not a directory."
    return 1
  fi
}

What am I misunderstanding?

It can be a very simple grep of yarn.lock file with xargs that copies the files to some other location temporarily and then removing the remaining files.

Aside, this is a worthwhile solution to investigate as well.

[bestander]

Interesting, but it seems to work for me. Are the files getting pulled automatically from the global cache under ~/Library/Caches/Yarn and the yarn-offline-mirror directory is automatically re-built from that?

Yeah, I think this worked out because you had caches intact.
From a clean slate Yarn would not know where to pull JSONStream-1.2.1.tgz#32aa5790e799481083b49b4b7fa94e23bae69bf9 from.

[rmacklin]

I left a comment on the PR (https://github.com/yarnpkg/yarn/pull/2836#issuecomment-285485971), but it seems like this issue is essentially done. Although as mentioned in https://github.com/yarnpkg/website/pull/409#discussion_r105274659, it's not released yet, so I suppose we could wait to close this issue until there's a release.

[bestander]

This feature is available https://yarnpkg.com/en/docs/prune-offline-mirror