yarn install changes yarn.lock file

[artlogic]

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
When installing certain packages the yarn.lock file is changed. It seems to be package specific. In this case cordova triggers the behavior.

If the current behavior is a bug, please provide the steps to reproduce.

In an empty directory, do the following:

yarn init
yarn add cordova
cp yarn.lock yarn.lock.initial
rm -r node_modules
yarn install
diff yarn.lock.initial yarn.lock

Notice that the yarn.lock file has changed.

What is the expected behavior?

Installing should not change the lock file if it already exists.

Please mention your node.js, yarn and operating system version.

node v6.11.3, yarn v1.0.1, Mac OS 10.11.6

[BYK]

I don't think this is a bug. I was able to reproduce the issue but the difference was

52,55d51
< ansi-regex@*:
<   version "3.0.0"
<   resolved "https://registry.yarnpkg.com/ansi-regex/-/ansi-regex-3.0.0.tgz#ed0317c322064f79466c02966bddb605ab37d998"
<
1352c1348
< imurmurhash@*, imurmurhash@^0.1.4:
---
> imurmurhash@^0.1.4:

This change is just an optimization of the lockfile and doesn't change the semantics. We are planning to add a warning when the lockfile needs an update when running yarn install but if you want to ensure your lock file stays the same and is accurate, you should be using the --frozen-lockfile option.

I have to admit that this is a bit unintuitive and we are trying to come up with a good solution to this. You can follow or contribute to the discussion on #4147.

If you agree that this is not a bug and #4147 is a good place to continue the discussion, pleae close the issue :)

[artlogic]

@BYK thanks for your thoughtful response. After reading #4147, I'm on the fence about closing this issue. I'm hoping a bit more discussion here can help.

4147 seems to be exclusively about the case where the yarn.lock and package.json are not in sync - in fact the only time optimization is mentioned is in this comment: https://github.com/yarnpkg/yarn/issues/4147#issuecomment-321894341 - and that comment touches on the issue that concerns me: yarn modifying the yarn.lock file when no changes have been made to package.json.

It seems to me that the yarn.lock file should only change if my dependencies are somehow changing. In fact, up to this point, I've been telling my devs that if they see a change in the yarn.lock file, it's time to run yarn install. With the recent upgrades several reported back that after they received my changes and they ran yarn install, their lock file changed (the optimization). This makes for a confusing situation - should they commit the change? Should I somehow pre-optimize the lock file (is that even possible)?

It looks like I may have to, in the short term, add --frozen-lockfile true to .yarnrc in all my repos - but I'm not a huge fan of this solution because it precludes the workflow mentioned in #4147 where you edit the package.json and then run yarn install.

Is the intention that any invocation of yarn might modify yarn.lock?

[artlogic]

@BYK I'm sorry to bother you about this because I know you are very busy - but I'm hoping to get a definitive answer on this. My workflow for distributing dependency changes to my team has devolved into this:

yarn add [blah]
rm -r node_modules yarn.lock
yarn install
rm -r node_modules
yarn install

This seems to update the lock file properly (see #4476), and then optimize the lock file so that my team mates aren't checking in lock file changes.

Is my team using yarn incorrectly? Should we be expecting the lock file to change with every install command?

[k0pernikus]

@artlogic IMHO, your team mates should be allowed to check in lockfile changes even if they are just optimizations. I'd agree with the notion that yarn install optimizing the lockfile is counter-intuitive, yet still something that each developer of your team could safely commit. I don't understand why you want to prohibit that.

That being said, your workflow might lead to unexpected side-effects, as you are pretty much giving up the locked-in versions. Deleting the lockfile means that yarn has to resolve all the dependencies version again, and it will try to fetch the most current version as pointed out in your package.json leading to potential breakage, esp. if more than one dependency breaks at the same time in a future version.

If you still want to enforce that install does not touch lockfile, then you should use the frozen-lockfile flag:

 yarn add {blah}

Check in lockfile, and then another dev would use:

   yarn install --frozen-lockfile

DON'T ADD --frozen-lockfile to true in .yarnrc as it prevents you from ever updating, see: #4570

If you want to upgrade your dependencies, run:

 yarn upgrade

If you want to upgrade only one dependency:

  yarn upgrade {blah}

You can also just install a specific version of one dependency:

  yarn install {blah}@1.5

This allows for more fine-tuning and less headaches in the long run.

[artlogic]

@k0pernikus - thanks for your detailed response. First, I should apologize for conflating two issues - #4476 is driving some of my workflow. The fact that yarn upgrade appears to be somewhat broken has nothing to do with this issue, of course.

As for workflow, I can't imagine mine is the only team where one developer is in charge of maintaining/testing new dependencies while the other developers work on the code base. The big issue I have with what has started happened is that I would install/test new dependencies, tell everyone to run yarn install and then get 4-5 questions about if the changed yarn.lock should be committed. It seems like I'd be in a worse place if I said "if the lockfile changes, commit it" because at that point I could end up for a commit lock that looks something like this:

• yarn.lock changed (dev4)
• yarn.lock changed (dev3)
• yarn.lock changed (dev2)
• yarn.lock changed (dev1)
• Updated dependencies (me)

However, it could be this is just the way yarn is intended to work, so for my team's projects we'll change all our repos to use frozen lockfile so that only add and upgrade cause the lockfile to change.

One additional scenario... it seems like this could be a potential annoyance for projects with lots of contributors because potentially each time they clone and install, the lockfile might change. I certainly wouldn't want these optimizations in pull requests. Is the intention that most FOSS projects using yarn transition to a frozen lockfile as well?

[jboler]

@artlogic Our team has --install.pure-lockfile true in the project's .yarnrc for exactly this reason.

[artlogic]

@jboler - I hadn't seen that setting before. I'm going to have to take a look, but it looks like I'll be suggesting that we add that to the .yarnrc for all our repos, and all the repos of any FOSS projects I have.

If someone from the yarn team can confirm that the intention is that yarn install can and will change the lock file on every run (despite no changes to dependencies) then I'll close this issue.

[BYK]

@artlogic sorry for the late response.

The thing is, it is __expected__ that yarn install to potentially change the lock file. I know this is not ideal and needs better communication though. I really want to have a better default but in the meantime, I think something like @jboler suggested with but maybe using --frozen-lockfile instead, would be the best solution until we figure out how yarn should behave under different scenarios like for workflows where people prefer simply editing package.json to update dependencies instead of using yarn add etc.

[vkrol]

@BYK I agreed with @artlogic . This behavior is very confusing:

The big issue I have with what has started happened is that I would install/test new dependencies, tell everyone to run yarn install and then get 4-5 questions about if the changed yarn.lock should be committed.

Our team consists of ~50 people who run yarn install every day :(

[BYK]

@vkrol you should be able to check in a .yarnrc file that has --install.frozen-lockfile true or --install.pure-lockfile true to avoid issues until Yarn changes its behavior. Would that help?

[vkrol]

@BYK pure-lockfile will help us, but not frozen-lockfile. If I understand correctly yarn install will fail with an error if frozen-lockfile is enabled. This behavior is absolutely unacceptable in our case.

[BYK]

@vkrol my preference is frozen-lockfile since it will fail only if your lock file needs changing and is not consistent. With pure-lockfile you may have a lockfile close to being useless or very inaccurate but you still wouldn't get a failure. Yarn would simply use the internal resolutions it calculated happily. If all your team members use the exact same version of Yarn, this should be fine. If not, it can be dangerous.

[vkrol]

@BYK maybe I do not understand the behavior of frozen-lockfile correctly. Does it fail when the lockfile is consistent with the package.json content, but some internal optimization needed?

[BYK]

@vkrol it should not fail when the file can be optimized further but it is consistent with package.json and is consistent in itself. If not, this is a bug. There is even a test for this: https://github.com/yarnpkg/yarn/blob/0415b07b3293ab125a77f3f66fe14034d6e5b376/__tests__/commands/install/lockfiles.js#L72-L84

[vkrol]

@BYK I did not know about it. I will try this option, thanks! I think that this fact should be documented somewhere.

[vkrol]

@BYK

it should not fail when the file can be optimized further but it is consistent with package.json and is consistent in itself.

I found some downside of the using of this flag. If yarn.lock can be optimized than the repeated invocations of this command yarn install --frozen-lockfile are not optimized via "integrity" check as without this flag. It this a bug? Do I need to create the separate issue?

[artlogic]

@BYK Thanks for your response. For my team's projects, we're now using frozen-lockfile and things are working (relatively) fine. This issue remains rather concerning for me, however. Specifically the unnecessary optimizations yarn is making to the lock file when running yarn install.

Here's the clearest example I can give of where this behavior causes problems:

1. I add a new dependency using yarn add [blah] and commit my updated package.json and yarn.lock.
2. Contributors to my package pull down changes and notice that yarn.lock has been updated, and thus run a yarn install.
3. In some cases these contributors end up with a modified yarn.lock after the install. What should they do in this case? Is it a race to see who can submit the pull request first?

To be clear, I have no problem with yarn install modifying yarn.lock if package.json has been modified. My problem is the optimization when package.json has not been changed. I see three possible solutions:

1. Direct any project that manages dependencies centrally to add --install.frozen-lockfile true to a .yarnrc file. This would likely be a LOT of projects - most NPM packages I'd guess.
2. Provide a switch to force yarn to optimize the lock file during the add phase, e.g. yarn add --optimize [blah]. Then direct package maintainers to always use that switch.
3. Disallow yarn from optimizing yarn.lock during the install phase unless the package.json appears to have been updated.

I am advocating for option 3, but I could see option 2 being a reasonable choice as well.

Thanks for your time!

[fender]

I kind of like the idea that yarn install does not update the yarn.lock file but instead emits a warning saying that "Your yarn.lock file is out of date, run yarn blah to update it now`.

[spencer-brown]

@BYK why ~doesn't~ does yarn apply optimizations to yarn.lock on yarn install rather than when add/upgrade are run?

edit: fix typo

[artlogic]

@spencer-brown: based on what I've seen, it actually does apply optimizations on install - that's what I find problematic

[mattblackdev]

Shouldn't the optimization to yarn.lock be run on yarn add? yarn.lock will be changing anyway. This is option number two in @artlogic's recent comment, right? Why doesn't the optimization occur then, so nobody else has to commit another yarn.lock change?

[spencer-brown]

@artlogic oops, typo :) - my comment should say "does"; edited.

[ryancastle]

We use composer a lot and composer install does not ever touch the lock file.

composer update or composer require (equivalent to yarn add) must be run for any changes or optimisations to be made to the lock file.

In many years, we have never had to worry about ambiguous behaviour of composer install.

I realise yarn fills a slightly different role, but composer has the equivalent issue of manual updates being made to composer.json and it getting out of sync with the lock file. Currently, composer install will completely ignore the composer.json file and issue a warning.

[ryancastle]

We use composer a lot and composer install does not ever touch the lock file.

composer update or composer require (equivalent to yarn add) must be run for any changes or optimisations to be made to the lock file.

In many years, we have never had to worry about ambiguous behaviour of composer install.

I realise yarn fills a slightly different role, but composer has the equivalent issue of manual updates being made to composer.json and it getting out of sync with the lock file. Currently, composer install will completely ignore the composer.json file and issue a warning.

[mikeyc7m]

hey kids, the lock file should never change on subsequent installs. it's just dumb. if you feel that the optimisation is worth touching it, just create a separate file instead that doesn't go into version control.

[jboler]

As with Bundler's Gemfile.lock, the whole point of yarn.lock is that every time yarn install is run, you can guarantee deterministic results across all environments. If yarn install changes the lock file you lose that guarantee. Other commands such as yarn add or yarn update should obviously change yarn.lock but yarn install should not. Our team has run into issues where we have slightly different package versions installed in different environments which is exactly what we don't want.

[ryancastle]

I'm inclined to think that yarn install --pure-lockfile should be the default, and there could be an --fix-my-dev-process-inconsistencies-for-me-magically option to get the current behaviour.

[k0pernikus]

@ryancastle --pure-lockfile won't tell you that an update is for your lockfile. It only doesn't generate one. This might still lead to unexpected behavior. --frozen-lockfile will fail if an update is required.

[artlogic]

I've been operating with --install.frozen-lockfile true in my .yarnrc for awhile now and it seems to work in a sane way. The trick is to create the initial yarn.lock for the repo without that flag in effect. Once it's created, installs can't change the yarn.lock, but updates can. Once I started using this workflow the number of accidental changes to yarn.lock dropped to zero.

[k0pernikus]

@artlogic Sidenote: I used to have frozen-lockfile true in my .yarnrc as well, now I only enforce it on the build server, as we ran into issues were intentional changes to the yarn.lock weren't made, e.g. when trying to sync a changed package.json to the yarn.lock, because a developer manually edited the package.json or used npm to add dependencies.

See: https://github.com/yarnpkg/yarn/issues/4570

[BYK]

@k0pernikus' use case is why we haven't made --frozen-lockfile the default with yarn install. I honestly don't know what a good compromise would be without hampering folks who edit package.json and then run yarn install to update the lockfile. I proposed a flag or a new command like yarn sync but the idea needs to be fleshed out and then judged by the community before being implemented.

Moreover, this would be a breaking change so it would require Yarn 2.x :)

[indeyets]

a flag or a new command like yarn sync

sounds good to me 👍 . and yarn install could show a notice "package.json and yarn.lock are out of sync. Run 'yarn sync' to update the lockfile"

[artlogic]

I honestly don't know what a good compromise would be without hampering folks who edit package.json and then run yarn install to update the lockfile.

I think at a certain point a decision needs to be made as what's the appropriate work flow. To me, enabling this almost seems like encouraging an anti-pattern.

I proposed a flag or a new command like yarn sync but the idea needs to be fleshed out and then judged by the community before being implemented.

To me, this seems like a reasonable compromise. If you insist upon hand editing your package.json, then running a command to get things back into sync makes sense.

Here's what I would like to see:

• yarn install without a yarn.lock creates the yarn.lock.
• yarn install with a yarn.lock doesn't modify it. If it finds there's a discrepancy between yarn.lock and package.json an error is returned (like @indeyets indicated).
• yarn sync (or maybe yarn install -s) gets yarn.lock back into sync with package.json.

I still have an underlying concern that yarn install modifies the yarn.lock even when package.json isn't out of sync with the lock file. You'll see above it performs some "optimizations". Would it be possible to at the very least disable this behavior without a breaking change?

[mattblackdev]

I still have an underlying concern that yarn install modifies the yarn.lock even when package.json isn't out of sync with the lock file. You'll see above it performs some "optimizations". Would it be possible to at the very least disable this behavior without a breaking change?

Same sentiment here. I thought this was the specific reason for this issue remaining open. As the "sync" is being discussed in #4147

It's why I searched for this issue in the first place.

[BYK]

Shall we merge this and #4147?

[artlogic]

@BYK I think nearly all of what's been discussed here can be merged with #4147. As I've said before, I think the only thing that still concerns me is the "optimizations" that yarn install performs on yarn.lock even when no changes have taken place to package.json.

My thought is that as a stop-gap on the way to yarn 2.0, these optimizations should be disabled. I don't think this would be a breaking change.

[krue-doug]

I 100% agree with @artlogic .

I was setting up CircleCI and my build kept failing and I traced it all the way to a caching issue with yarn.lock. Coming from Rails, I would expect lock file to be locked by on a install command.

This is what is being suggested for cache checking on CircleCI: https://circleci.com/docs/2.0/yarn/

#...
      - restore_cache:
          name: Restore Yarn Package Cache
          keys:
            - yarn-packages-{{ checksum "yarn.lock" }}
      - run:
          name: Install Dependencies
          command: yarn install
      - save_cache:
          name: Save Yarn Package Cache
          key: yarn-packages-{{ checksum "yarn.lock" }}
          paths:
            - ~/.cache/yarn
#...

This will never work because yarn.lock can change, when it shouldn't.

[dtgriscom]

FYI, on yarn 1.10.1 under macOS 10.13.6, I just tried the OP's bug stimulus, and found that yarn.lock wasn't changed. In other words, when I tried:

yarn init
yarn add cordova
cp yarn.lock yarn.lock.initial
rm -r node_modules
yarn install
diff yarn.lock.initial yarn.lock

So, when using yarn 1.10.1, the yarn install didn't change yarn.lock, as it had for the OP when using yarn 1.0.1. (I consider this a good thing, as I think would the OP.)

[artlogic]

@dtgriscom That's great that the problem isn't happening anymore for Cordova! I'm wondering if it's been fixed entirely or if yarn still might attempt to optimize the lock file on install?

[dtgriscom]

It would be nice to know, wouldn't it? You noted the problem was package-specific; it makes sense that it might be version-specific as well. (I never heard a justification for "we want to change something that shouldn't change"; other than "we're optimizing"...)

[szymach]

I wholeheartedly agree that an install operation needs to be deterministic by default. Updating a .lock file should require the developer to perform an action that he knows will produce a change in the file. For example, you could run something like yarn install --optimize that would allow for small optimization that do not break current state of packages.

Adding separate options that I have to look up in to documentation just so I can have the certainty my lock file is untouched without my permission is pretty confusing, especially if they can cause failure. As a developer I expect to have control over the tools I use and I feel like this behaviour takes that away from me.

Please reconsider flipping the default behaviour to not modify the lock file and operate solely on it when installing dependencies. An integrity check with a warning that package.json is out of sync with the lock file is really all people need.

[MartijnHols]

I've been telling people that PRs including changes to the lock file will not be merged as the lock file is only changed when adding new or updating dependencies. This is what the install documentation states:

yarn install

Install all the dependencies listed within package.json in the local node_modules folder.

The yarn.lock file is utilized as follows:

• If yarn.lock is present and is enough to satisfy all the dependencies listed in package.json, the exact versions recorded in yarn.lock are installed, and yarn.lock will be unchanged. Yarn will not check for newer versions.
• If yarn.lock is absent, or is not enough to satisfy all the dependencies listed in package.json (for example, if you manually add a dependency to package.json), Yarn looks for the newest versions available that satisfy the constraints in package.json. The results are written to yarn.lock.

If you want to ensure yarn.lock is not updated, use --frozen-lockfile.

Is the documentation wrong as there are these optimizations that can happen stated above or is this a bug to take into account for the time being?

ps. it still happens, and yarn --frozen-lockfile does not fail when it does.

[konekoya]

I'm current using --frozen-lockfile in my .yarnrc file, which effectively prevent yarn from modifying yarn.lock file when running yarn install on different machines. But this also prevent yarn add blahblah from modifying my yarn.lock. Is that the correct behavior or I'm missing something? I've read thru the entire thread and it seems to me that --frozen-lockfile is the recommended approach for now?

[k0pernikus]

@konekoya Yes, you are right that you currently MUST use

yarn install --frozen-lockfile

and cannot rely on the setting in the .yarnrc as it will prevent your yarn.lock from ever updating.

Depending on your content of your .npmrc and .yarnrc you may be able to use

yarn install --no-default-rc {dependency}

but I would not rely on that (since it would break as soon as you have other settings in your rc-files.)

See my comment and the relevant issue: #4570, esp. my conclusion.

[konekoya]

Thanks for clearing that up @k0pernikus. You just made my day :)

[qtiki]

@BYK why does yarn apply optimizations to yarn.lock on yarn install rather than when add/upgrade are run?

I think @spencer-brown 's comment here is key to "fixing" this problem. I just tested for myself and running yarn upgrade and then yarn I was told "Already up-to-date.". So I deleted my node_modules and ran yarn which then resulted in the "optimized" lockfile which the other devs in my project would have ended up if I had pushed the one I got from yarn upgrade.

Now if yarn upgrade (and yarn add) would run the same optimization as yarn install we wouldn't have this problem in the first place. There's been a lot of talk about not wanting to break any existing functionality and I don't see running the optimization after yarn upgrade and yarn add as breaking anything - just making it a little bit slower.

As a workaround I am thinking of instructing all of our devs to delete node_modules whenever they need to update the lockfile to force the "optimization" before pushing the lockfiles to others. Or maybe they can run yarn upgrade && yarn prune - maybe that would do the trick?

[qtiki]

As a workaround I am thinking of instructing all of our devs to delete node_modules whenever they need to update the lockfile to force the "optimization" before pushing the lockfiles to others. Or maybe they can run yarn upgrade && yarn prune - maybe that would do the trick

Update: You can't run prune manually, you just get this message: _"The prune command isn't necessary. yarn install will prune extraneous packages"._ And since you can't run install either because you are _"Already up-to-date"_ I guess we're going with deleting node_modules and then running yarn to do the pruning.

[peteorpeter]

For me, the single biggest selling point for yarn is predictable, reliable lock file behavior. Changing the lock file on yarn install gives me real anxiety.

[dtgriscom]

For me, the single biggest selling point for yarn is predictable, reliable lock file behavior. Changing the lock file on yarn install gives me real anxiety.

The lock file is presented as an exact specification of what will be installed, so that we don't have to worry about what will be installed. Use the same lock file, and you'll get the same install every time. But sometimes yarn will, on its own initiative, for its own reasons, and with no warning, change the lock file contents. Yes, you can argue that "the file changes don't change what will be installed", but why take such a clear guarantee and muddle it up? Why make me wonder if I should be re-committing the changed lock file? Why sully your main claim to fame?

[omarreiss]

This is really unbelievable. Others have already stated it well. A lock file should not change on install. This negatively impacts commit workflows. Every developer in our team knows under what circumstances the lockfile is allowed to change. When a lockfile changes without the dependencies being updated, that causes unnecessary confusion and extra checking. This is basic package manager UX that is just broken. Please fix.

[lobo-tuerto]

I just got surprised by this behaviour, when moving from Node.js 13 to 14 all my yarn.lock files changed when doing a yarn install and I was wondering why...

I think the appropriate behaviour would be to emit a warning if _optimizing_ the yarn.lock file in any way.