yarn.lock file changing due to non-deterministic generation.

[SPSpwetter]

Bug: Yarn.lock file changing ordering of dependencies and resolutions.

What is the current behavior?

The yarn.lock file has changes between different developers systems, with small differences. Mostly they seem to be ordering differences.

If the current behavior is a bug, please provide the steps to reproduce.

1. Check yarn.lock file into source repository.
2. Tell all developers we're moving to yarn, that they have to update everything.
3. Developers get modification notices from git that yarn.lock has changed and are unsure what to do because they're not front end developers.

See differences sample below. These are all transitive dependencies from the package.json root. Reviewing the differences, it seems that yarn isn't fully deterministic in ordering when generating the yarn.lock file, as the ultimate content is identical, but the pack ordering and the version ordering for a package shows differences.

What is the expected behavior?

Expectation: Developer can check in yarn.lock file. Other developers can run yarn and get an up-to-date node-modules folder. If a new yarn.lock file is produced, it is identical to the checked in version.

Please mention your node.js, yarn and operating system version.

node: 4.6.0 with npm 3.10.8
yarn: 0.15.1
OS: Mac OS Sierra, and Ubuntu 14.4

Sample Differences:

diff --git a/skycontrol/src/sc-web/yarn.lock b/skycontrol/src/sc-web/yarn.lock
index 7cf9362..a4dd27f 100644
--- a/skycontrol/src/sc-web/yarn.lock
+++ b/skycontrol/src/sc-web/yarn.lock
@@ -8,13 +8,6 @@ abbrev@1, [email protected]:
   version "1.0.9"
   resolved "http://darkstar-devops.lab.skyportsystems.com:4873/abbrev/-/abbrev-1.0.9.tgz#91b4792588a7738c25f35dd6f63752a2f8776135"

-accepts@~1.3.3:
-  version "1.3.3"
-  resolved "http://darkstar-devops.lab.skyportsystems.com:4873/accepts/-/accepts-1.3.3.tgz#c3ca7434938648c3e0d9c1e328dd68b622c284ca"
-  dependencies:
-    mime-types "~2.1.11"
-    negotiator "0.6.1"
-
 [email protected]:
   version "1.1.4"
   resolved "http://darkstar-devops.lab.skyportsystems.com:4873/accepts/-/accepts-1.1.4.tgz#d71c96f7d41d0feda2c38cd14e8a27c04158df4a"
@@ -22,6 +15,13 @@ [email protected]:
     mime-types "~2.0.4"
     negotiator "0.4.9"

+accepts@~1.3.3:
+  version "1.3.3"
+  resolved "http://darkstar-devops.lab.skyportsystems.com:4873/accepts/-/accepts-1.3.3.tgz#c3ca7434938648c3e0d9c1e328dd68b622c284ca"
+  dependencies:
+    mime-types "~2.1.11"
+    negotiator "0.6.1"
+
 acorn-globals@^1.0.4:
   version "1.0.9"
   resolved "http://darkstar-devops.lab.skyportsystems.com:4873/acorn-globals/-/acorn-globals-1.0.9.tgz#55bb5e98691507b74579d0513413217c380c54cf"
@@ -164,14 +164,14 @@ async-each@^1.0.0:
   version "1.0.1"
   resolved "http://darkstar-devops.lab.skyportsystems.com:4873/async-each/-/async-each-1.0.1.tgz#19d386a1d9edc6e7c1c85d388aedbcc56d33602d"

[email protected], async@^1.3.0, async@^1.4.0:
+  version "1.5.2"
+  resolved "http://darkstar-devops.lab.skyportsystems.com:4873/async/-/async-1.5.2.tgz#ec6a61ae56480c0c3cb241c95618e20892f9672a"
+
 async@^0.9.0, async@~0.9.0:
   version "0.9.2"
   resolved "http://darkstar-devops.lab.skyportsystems.com:4873/async/-/async-0.9.2.tgz#aea74d5e61c1f899613bf64bda66d4c78f2fd17d"

-async@^1.3.0, async@^1.4.0, [email protected]:
-  version "1.5.2"
-  resolved "http://darkstar-devops.lab.skyportsystems.com:4873/async/-/async-1.5.2.tgz#ec6a61ae56480c0c3cb241c95618e20892f9672a"
-
 async@^2.0.1:
   version "2.1.1"
   resolved "http://darkstar-devops.lab.skyportsystems.com:4873/async/-/async-2.1.1.tgz#e11b6d10043f2254efb61a21163d840ccddb8d28"
@@ -213,7 +213,7 @@ babel-code-frame@^6.11.0, babel-code-frame@^6.16.0:
     esutils "^2.0.2"
     js-tokens "^2.0.0"

[SPSpwetter]

Sample files.
yarn.lock.txt
yarn.lock.txt

[SPSpwetter]

This may be a MacOS vs. Linux thing, as the linux versions seem to be self consistent.

[markstos]

Are you sure all the devs involve were using the same version for the "before" and "after" versions of the file @SPSpwetter ?

[SPSpwetter]

Yes, it was a git pull in all cases.

[SPSpwetter]

Confirming that this appears in 0.16.0 as well.

[rjmunro]

Do the lockfiles still get modified if you use --pure-lockfile? AFAICS, if #570 was fixed, this might become much less of an issue - The lockfile would only get modified if there was a legitimate reason to modify the lockfile. It may still change order, but only when the file was changing for some other reason anyway.

[SPSpwetter]

Testing... no they don't get modified.

I'll have to think about that though. I can easily change the makefile the non-UI developers use to enforce that, but its kind of a "feature" that the lock file gets modified if the resolution is different somehow. If yarn changed to require --save-lockfile, I'd probably want a --warn-lockfile instead, unless that's what the behavior of --pure-lockfile does is not write a file, but warn if it wants to. Still, the versioning order should be deterministic across platforms.

[SPSpwetter]

Closing this because #1712 resolves the issue. While the lock file generation should be the same across platforms, if it doesn't get rewritten without and install change, I'm good.

[pior]

It seems that the original issue is not solved.

I just observed this after running yarn add raygun:

@@ -2778,14 +2780,14 @@ hyphenate-style-name@^1.0.1:
   version "1.0.2"
   resolved "https://registry.yarnpkg.com/hyphenate-style-name/-/hyphenate-style-name-1.0.2.tgz#31160a36930adaf1fc04c6074f7eb41465d4ec4b"

-iconv-lite@^0.4.5, iconv-lite@~0.4.13, [email protected]:
-  version "0.4.13"
-  resolved "https://registry.yarnpkg.com/iconv-lite/-/iconv-lite-0.4.13.tgz#1f88aba4ab0b1508e8312acc39345f36e992e2f2"
-
 [email protected]:
   version "0.2.11"
   resolved "https://registry.yarnpkg.com/iconv-lite/-/iconv-lite-0.2.11.tgz#1ce60a3a57864a292d1321ff4609ca4bb965adc8"

[email protected], iconv-lite@^0.4.5, iconv-lite@~0.4.13:
+  version "0.4.13"
+  resolved "https://registry.yarnpkg.com/iconv-lite/-/iconv-lite-0.4.13.tgz#1f88aba4ab0b1508e8312acc39345f36e992e2f2"
+

Original was commited on Mac OS, I'm on Linux.

I understand that a big pain point was fixed (reordering without action) but the ordering issue itself is major.
I'm reluctant to open a pull-request with 1226 lines changes in yarn.lock after only adding one package (+ 2 deps)

Yarn version: 0.18.1

[markstos]

@pior Were both the original and changed yarn.lock generated with yarn 0.18.1?

[SPSpwetter]

Hmmm... I just did some testing with 0.18.1, and got slightly only different results between linux and MacOS, but this seems to be much better than it used to be. @pior?

[markstos]

@SPSpwetter could you post what the slight difference was?

[SPSpwetter]

Ok, might just be different resolution:

< domelementtype@1, domelementtype@~1.1.1:
---
> domelementtype@1:
>   version "1.3.0"
>   resolved "http://darkstar-devops.lab.skyportsystems.com:4873/domelementtype/-/domelementtype-1.3.0.tgz#b17aed82e8ab59e52dd9c19b1756e0fc187204c2"
> 
> domelementtype@~1.1.1:
2223c2227
< [email protected], glob@^7.0.3, glob@^7.0.5:
---
> [email protected]:
2243a2248,2258
> glob@^7.0.3, glob@^7.0.5:
>   version "7.1.1"
>   resolved "http://darkstar-devops.lab.skyportsystems.com:4873/glob/-/glob-7.1.1.tgz#805211df04faaf1c63a3600306cdf5ade50b2ec8"
>   dependencies:
>     fs.realpath "^1.0.0"
>     inflight "^1.0.4"
>     inherits "2"
>     minimatch "^3.0.2"
>     once "^1.3.0"
>     path-is-absolute "^1.0.0"
> 

[SPSpwetter]

[markstos]

@SPSpwetter Shouldn't this issue be re-opened? From your screenshot and diff, it looks like the glob@^7.0.3 dependency is getting resolved differently on the two platforms. In one case it resolves to 7.0.5 and in the other case to 7.1.1.

Perhaps there is some relationship with the state of the local ./node_modules directory. Without a published spec for how yarn.lock is supposed to work, it's to help troubleshoot this much further.

[conf]

We also encountered this issue recently. Simple yarn add <package> produces big churn of yarn.lock:

♥ yarn add async_fn@^0.1.7
yarn add v0.18.1
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
warning [email protected]: The engine "browser" appears to be invalid.
[3/4] 🔗  Linking dependencies...
warning Unmet peer dependency "[email protected] || 0.20.x".
[4/4] 📃  Building fresh packages...
success Saved lockfile.
success Saved 1 new dependency.
└─ [email protected]
✨  Done in 6.98s.
♥ git status
## master...origin/master
 M package.json
 M yarn.lock
♥ git diff --stat
 package.json |    2 +-
 yarn.lock    | 1061 +++++++++++++++++++++++++++++++++++++++++++++++++-----------------------------------------------------------------------------------------------------------
 2 files changed, 335 insertions(+), 728 deletions(-)

We reproduced this issue both on yarn 0.17.10 and 0.18.1, Node version: 6.9.1. Our team uses both Linux and macOS.

[rjmunro]

@conf Depending on the package you add and how many dependencies it has, I'd expect a lot of changes in yarn.lock. This bug is about yarn.lock changing without changing package.json.

[conf]

@rjmunro So you're saying this is normal behavior because async_fn is a quite popular package and a lot of other packages depend on it?

[rjmunro]

@conf Possibly. I'm just a random user interested in yarn, not someone who makes it. I'm just saying how I feel it should work, based on my experience of yarn and similar tools in other languages.

Was the old yarn.lock generated by the same version of yarn as the new one?
Are any of the lines that were added and lines that were removed the same, but in a different order?

If both of those are true, then there is still an issue. I think @markstos & @SPSpwetter's issue about different resolutions on different platforms is not the same as this one. This is about the same resolution being stored differently in yarn.lock.

[conf]

@rjmunro Thank you for your input. I'm also not 100% sure, I'll try to do some tests with different packages, and yarn versions.

[SPSpwetter]

I'll confirm that the issue here is specific to getting different versions of yarn.lock _across platforms_ with the _same package.json, and using the same in-house yarn repository_.

[ik5]

The following issue happens to me as well, I'm using Linux, and other developers here are using Mac.
So I have dependency that the version requirement is different between machines. For example:

-classnames@^2.2.0, classnames@^2.2.4, classnames@^2.2.5:
+classnames@^2.1.2, classnames@^2.2.0, classnames@^2.2.4, classnames@^2.2.5:

All we did was yarn install to add new dependency that it is not that one specific.

[aaronbeall]

We're running into this issue immediately after migrating to yarn. In our case all devs are using OSX (possibly different minor versions) yet each dev gets different ordering of lock file dependencies every time we run yarn, and even more surprisingly some of us (I haven't checked with everyone) get slightly different order with subsequent runs of yarn (within seconds of each other) on a single machine! Kind of ironic, given that we switched from npm to yarn mainly on the selling point that it's more deterministic. 😕

Yes, presumably --pure-lock-file will partially fix this issue, but as soon as someone adds/upgrades/removes anything I expect we'll have the same "random" re-ordering of half our lock file.

[markstos]

If there were a published spec for how yarn.lock is supposed to work, it would easier for other developers to dive into the code with some confidence about what's a bug expected behavior related to yarn.lock.

It would be best if someone involved with the yarn.lock code wrote the spec, since other people can see what the code does, the authors would have the clearest idea what it was intended to do.

Right now it's not clear if the spec itself has a flawed design or if the spec is sound and there's an implementation bug that someone can dive in and fix.

[Vertux]

I run into the same problem:

~/live$ yarn install
yarn install v0.22.0
[1/4] Resolving packages...
[2/4] Fetching packages...
warning [email protected]: The engine "browser" appears to be invalid.
warning [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
warning "[email protected]" has incorrect peer dependency "[email protected]".
warning "[email protected]" has incorrect peer dependency "@kadira/storybook@^1.35.1".
[4/4] Building fresh packages...
warning Your current version of Yarn is out of date. The latest version is "0.23.2" while you're on "0.22.0".
info To upgrade, run the following command:
$ npm upgrade --global yarn
Done in 158.38s.

Ubuntu v16.04
yarn 0.22.0
node v4.8.2
npm 2.15.11

[dwasyluk]

This appears to still be not fixed... can we please reopen or get a pointer to a new bug that shows this is being addressed? The biggest problem is when non-deterministic lockfile regeneration leads to lockfile conflicts between developers. This has happened to our organization's dev team _frequently_ we've had so much so that we've had to craft a process for dealing with these "lockfile conflicts". That process being merging the package.json file in the case of a lockfile conflict, discarding the lockfile changes, and then regenerating the lockfile based on the merged package.json.

While this solution works and narrows the scope of the problem, it also overwrites other user's lockfile versions, so there is still an issue here that needs addressing imo.

[SPSpwetter]

Hmmm... Wondering if this is a regression? I'm using 0.18.1 and I only get yarn.lock file changes when package.json changes, even across platforms.

However, your process is correct, in that if your package.json file changes, you should discard the old one and generate a new one.

[markstos]

It's hard to know what's a regression when there is no published spec for use of the yarn.lock file.

[SPSpwetter]

Well, the docs make it pretty clear that you're supposed to checkin your yarn.lock file with your source, and know that running yarn again should give you the same set of node_modules and the same yarn.lock file.

Our workflow here is:

1. UI Dev use Yarn add or whatever to get new package.
2. UI Dev checkin updated package.json and yarn.lock.
3. CI System builds on Linux, yarn.lock doesn't change.

Testing with latest.

[SPSpwetter]

Hmmm... Ok, I'm getting a diff in the yarn.lock file, where I didn't used to:

PierceWetter3:scweb pierce$ git diff yarn.lock
diff --git a/skycontrol/src/sc-web/yarn.lock b/skycontrol/src/sc-web/yarn.lock
index 3e169c3..a386fd7 100644
--- a/skycontrol/src/sc-web/yarn.lock
+++ b/skycontrol/src/sc-web/yarn.lock
@@ -5529,14 +5529,14 @@ semver-truncate@^1.0.0:
   dependencies:
     semver "^5.3.0"

-"semver@2 || 3 || 4 || 5", semver@^4.0.3, semver@~4.3.3:
-  version "4.3.6"
-  resolved "http://ourinternalnpmrespository:4873/semver/-/semver-4.3.6.tgz#300bc6e0e86374f7ba61068b5b1ecd57fc6532da"
-
-semver@^5.3.0:
+"semver@2 || 3 || 4 || 5", semver@^5.3.0:
   version "5.3.0"
   resolved "http://ourinternalnpmrespository:4873/semver/-/semver-5.3.0.tgz#9b2ce5d3de02d17c6012ad326aa6b4d0cf54f94f"

+semver@^4.0.3, semver@~4.3.3:
+  version "4.3.6"
+  resolved "http://ourinternalnpmrespository:4873/semver/-/semver-4.3.6.tgz#300bc6e0e86374f7ba61068b5b1ecd57fc6532da"
+
 [email protected]:
   version "0.15.1"
   resolved "http://ourinternalnpmrespository:4873/send/-/send-0.15.1.tgz#8a02354c26e6f5cca700065f5f0cdeba90ec7b5f"

[SPSpwetter]

Reverting to 0.18.1:

PierceWetter3:scweb pierce$ git diff yarn.lock
PierceWetter3:scweb pierce$ 

So doesn't happen with old version. That means this is either a regression, or the yarn lock format changed?

[Lekensteyn]

No idea why yarn upgrade would always resolve in a different order, but in the end I used the following and got a deterministic ordering:

yarn upgrade
yarn upgrade --prefer-offline

Versions: yarn 0.24.6, node v4.7.10.0. package.json used for testing:

{
    "name": "buildbot-nestedexample",
    "engines": {
        "node": ">=0.10.0",
        "npm": ">=1.4.0"
    },
    "dependencies": {
        "guanlecoja": "~0.8.3",
        "gulp": "3.9.0"
    }
}

[bestander]

Thanks for reporting, we are tracking this issue in #3490