Yarn: 1.6.0: [email protected] rebuilt every time anything is installed

[lydell]

Do you want to request a feature or report a bug?
bug

What is the current behavior?
[email protected] is rebuilt if installing five.

If the current behavior is a bug, please provide the steps to reproduce.
See below.

What is the expected behavior?
Installing five does not cause a rebuild of node-rdkafka.

Please mention your node.js, yarn and operating system version.

• Node.js 8.9.4
• yarn 1.6.0
• Ubuntu 17.10

Description
This is a follow-up issue to #932. yarn 1.6.0 seem like a big step in the right direction! Thanks! I have one really odd case where the "native modules always rebuilt" problem still persists, though.

In short, if I install [email protected] (that exact version) and then five (which depends on nothing and nothing depends on), node-rdkafka is rebuilt unnecessarily.

The strange thing is that if I install any later version of node-rdkafka, such as 2.3.2, everything works as expected (no rebuild)!

Steps to reproduce:

~/stuff  
❯ yarn cache clean -f
yarn cache v1.6.0
success Cleared cache.
Done in 0.03s.

~/stuff  
❯ mkdir tmp

~/stuff  
❯ cd tmp

~/stuff/tmp  
❯ echo '{"private":true}' > package.json

~/stuff/tmp  
❯ yarn add [email protected]
yarn add v1.6.0
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
success Saved 3 new dependencies.
info Direct dependencies
└─ [email protected]
info All dependencies
├─ [email protected]
├─ [email protected]
└─ [email protected]
Done in 40.62s.

~/stuff/tmp 40s  
❯ yarn add five
yarn add v1.6.0
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
success Saved 1 new dependency.
info Direct dependencies
└─ [email protected]
info All dependencies
└─ [email protected]
Done in 39.06s.

~/stuff/tmp 39s  
❯ echo Kafka was rebuilt!

I've tried this on two different computers, one of which I had never installed node-rdkafka on before.

I apologize if this turns out not to be a problem with yarn after all. But considering the number of people interested in #932 I think it can be good to have an answer in this issue tracker that people might find. Thanks!

[rally25rs]

This happens for me on OSX as well 😿
I'll try to figure out why...

---

_Update 1_

My previous fix for #932 ensured that differences in build artifacts tracked in the node_modules/.yarn-integrity file wouldn't trigger a rebuild.

In this case, it seems like a bunch of build artifacts for node-rdkafka don't get added to that integrity file.

Many files do get added, but a bunch named build/Release/obj.target/librdkafka/deps/librdkafka/src/lz4* don't get added.

---

_Update 2_

OK It looks like the build process for that package _deletes_ the files I mentioned above. They exist and are shipped with the package .tgz (probably incorrectly. why would you ship .o files that need to be built?)
Then the C/C++ build process deletes those files (they must not be rebuilt for whatever reason.)
We track _changed_ and _new_ files in .yarn-integrity but we don't track _removed_ files.

When yarn finds files that are in the package's .tgz that are missing from node_modules it decides it needs to reinstall / rebuild it.

[felixrabe]

I've been reading all the way from #932 via other related issues to this one, and I'm really looking forward to the speedups related to compiling native modules, especially when used together with electron. Hopefully https://github.com/yarnpkg/yarn/pull/5314#issuecomment-374226071 just works for me already :) (EDIT: it does!) Will try that and if I don't come back with specific issues, consider this comment a big Thank You for working on this! 🎉

PS: Not related to Yarn, but to my Electron-plus-native-modules issue: Together with Yarn's experimental caching, I ended up wrapping electron-rebuild in a custom script and running that in scripts.postinstall. The wrapper keeps ./node_modules as-is so its native packages still work with native Node processes, whereas it creates the directory ./electron-rebuild/<electronVersion>/node_modules with only the native modules (modules containing binding.gyp) for module.paths.unshift()-ing for Electron processes. As intended, native modules in that directory only get built if they have changed.

PS 2: Wrapper (in minimal example project) published at https://github.com/felixrabe/e-2018-065-yarn-better-sqlite3-electron.

[rally25rs]

So I guess this "fix" for this case will have to be tracking deleted files in the yarn integrity artifacts list. If anyone is willing to work up a PR, we would appreciate it.

[milottit]

node-sass is always recompiling after any yarn add package command. And it is really long. Also the package sharp postinstall is not getting called and I have to do it by hand every time.

Here you find my package.json for node-sass

{
    "name": "webpack",
    "version": "1.0.0",
    "main": "dist",
    "license": "MIT",
    "scripts": {
        "clean": "shx rm -r dist",
        "prebuild": "yarn clean",
        "postbuild": "yarn all-test",
        "build": "rm -R dist && tsc",
        "all-test": "yarn test && yarn e2e",
        "test": "jest --config jest-ut.config.js",
        "e2e": "yarn e2e-plain-webpack && yarn e2e-angular",
        "e2e-plain-webpack": "jest --config jest-e2e-plainwebpack.config.js",
        "e2e-angular": "jest --config jest-e2e-angular.config.js",
        "set-follow-symlinks-watcher": "sed -i 's/followSymlinks: false/followSymlinks: true/g' node_modules/watchpack/lib/DirectoryWatcher.js",
        "push": "npm version patch && git push && git push --tags",
        "publish": "tsc && npm publish --access public"
    },
    "devDependencies": {
        "@angular-devkit/build-angular": "^0.13.3",
        "@angular/core": "^7.2.7",
        "@types/globby": "^8.0.0",
        "@types/jest": "^24.0.9",
        "@types/node": "^11.9.5",
        "@types/parse5": "^5.0.0",
        "@types/webpack": "^4.4.25",
        "clean-webpack-plugin": "^1.0.1",
        "jest": "^24.1.0",
        "jest-expect-message": "^1.0.2",
        "micromatch": "^3.1.10",
        "p-each-series": "^1.0.0",
        "parse5": "^5.1.0",
        "shx": "^0.3.2",
        "ts-jest": "^24.0.0",
        "ts-node": "^8.0.2",
        "ts-util-is": "^1.1.3",
        "typescript": "^3.3.3333",
        "webpack": "^4.29.5",
        "webpack-cli": "^3.2.3"
    },
    "dependencies": {
        "@angular-devkit/core": "^7.3.3",
        "@ud-angular-builders/custom-webpack-7": "^7.5.1",
        "@upradata/browser-util": "^1.0.11",
        "@upradata/node-util": "^2.0.2"
    },
    "majestic": {
        "jestConfig": "jest-ut.config.js"
    }
}

[rimmartin]

openssl needs to be rebuilt after every yarn install
with

    "openssl" : "openssl/openssl#OpenSSL_1_1_1g",

btw I'm finding c++ projects can be fairly organized with yarn and package json's. Independent of IDE's

[jedwards1211]

this can get especially frustrating when working with global packages like now, which must be rebuilt anytime you add or remove any other global.

Also I got so fed up with fsevents getting rebuilt every time I installed or updated local deps in my packages that I replaced @babel/cli, which transitively depends on it, with my own script. It really wears on you that much.

[jedwards1211]

@rally25rs I don't guess it's possible to turn off integrity checks for cases like this? Personally I would rather yarn never ever touch a package directory within node_modules after installation, unless a different version of the package needs to be installed.

It's kind of a classic case of how solutions usually create other problems and everything is a tradeoff...