Yarn: Rebuilding native modules

[rgbkrk]

Do you want to request a _feature_ or report a _bug_?

Feature!

What is the current behavior?

https://yarnpkg.com/en/docs/migrating-from-npm does not list a way to rebuild.

What is the expected behavior?

We (in nteract) end up needing to rebuild one native module, zeromq. Normally we run this:

npm rebuild zeromq --runtime=electron --target=1.4.5 --disturl=https://atom.io/download/atom-shell --build-from-source

Our use case is specifically with Electron; I'm guessing there are others that use npm rebuild.

Please mention your node.js, yarn and operating system version.

$ node --version
v6.7.0

$ yarn --version
0.15.1

OS X 10.11.6

/cc @jdetle @captainsafia @lgeiger

[rgbkrk]

:heart: the experience here, thank you for shaving off so many edges to make developer experience wonderful!

[rgbkrk]

Looks like we can always run the rebuild directly (even scripting it like we do to match the Electron version in our package.json). However, running yarn again after will write over the native modules that were built for Electron, resulting in an ABI mismatch.

[gaelduplessix]

I would also like to have an equivalent to npm rebuild to be able to just extract a .tgz of my dependencies and rebuild the platform-specific stuff without having to re-fetch the packages. The main usage for this is building our app in Docker containers during CI. We currently can't use yarn for this use case.

[ConorIA]

electron-builder can't support yarn without a rebuild command that respects the architecture set in the environment. Does yarn install use only the real arch of the machine, or does it build for the arch specified in the environment?

[develar]

Clarification to @ConorIA question: electron-builder set npm_config_arch https://github.com/electron-userland/electron-builder/blob/master/src/util/util.ts#L29 The problem is that npm_config_arch=ia32 npm installl will install and build dependencies for ia32 (if no installed modules or not yet build), but subsequent npm_config_arch=x64 npm installl will not do rebuild dependencies for specified arch. Explicit npm rebuild is required.

[develar]

rebuild will be implemented on electron-builder side. For any project structure (not only 2-package.json). npm rebuild doesn't support rebuild only production deps (https://github.com/npm/npm/issues/5952). As electron-builder should continue to support npm (warning that yarn is recommended will be added soon) and due to the fact that rebuild is specific for Electron, it is not implemented as PR here (in any case, implementation is very simple and no need to complicate yarn).

[develar]

@rgbkrk It seems you can just use yarn add zmq-prebuilt --force --runtime=electron --target=1.4.3 --disturl=https://atom.io/download/atom-shell --build-from-source

--force here to force rebuild.

So, I guess, this issue can be closed (of course, no doubt, I don't recommend in any case use it for electron project – it is way to nowhere — some tool (e.g. electron-builder or electron-rebuild (if for some reasons you cannot use electron-builder ;))) should be used).

[rgbkrk]

Awesome, thanks @develar!

[develar]

Due to https://github.com/yarnpkg/yarn/issues/1749 yarn will be still not supported by electron-builder (probably I will send PR to fix it).

[develar]

Issue on electron-builder side — https://github.com/electron-userland/electron-builder/issues/861

[busches]

As @Blaccexican and @develar mentioned, the docs now mention that yarn install --force is the equivalent to npm rebuild. Can this be closed @rgbkrk?

[develar]

@busches Strictly speaking, it is not equivalent. yarn does it more slowly and touch more than need. Maybe I am wrong, but it is what I saw several weeks ago. electron-builder currently uses another approach in case of yarn to rebuild. But in general case — yes, I suggest to close issue since it is more electron issue then yarn (i.e. you should not in any case use it to rebuild native deps for electron).

[gaelduplessix]

npm rebuild can possibly work offline if the build tasks don't download anything. yarn install --force can't. As @develar said, it's also much slower. So we still can't use yarn to rebuild deps in a Docker for example

[rgbkrk]

Since my needs are addressed in Electron, the issue for _me_ is closed. However, it still stands as @gaelduplessix points out that you can't rebuild using yarn (without doing the full thing).

[LaurensRietveld]

I agree with @gaelduplessix : rebuilding deps in a docker-build scenario is a use case that's probably quite common. Having the real npm rebuild command here would be great.

[davidgoli]

yarn install is deprecated by yarn add, and yarn add --force does not respect the versions specified in package.json or yarn.lock. yarn add --force is not equivalent to npm rebuild.

[arcanis]

Note: yarn install is not deprecated by yarn add (however, yarn install <pkg-name> is deprecated by yarn add). Furthermore, the offline mirror feature makes it possible to run the install without needing any network capabilities.

[SimenB]

Well, certain post-installs might hit the network, like https://github.com/Medium/phantomjs/blob/master/install.js

[jedwards1211]

I think I'm seeing an issue in CI where the phantomjs-prebuilt package is cached and up-to-date but the binary is not built and installed, so it would be nice to be able to run the concurrent rebuilt manually from yarn instead of npm's serial rebuild.

[aj0strow]

It would be sweet to upgrade node versions, then run $ yarn rebuild or simply $ yarn and have native extensions rebuild.

[jedwards1211]

@aj0strow yeah no kidding :)

[williamboman]

$ yarn and have native extensions rebuild.

How would yarn know which Node runtime version a dependency tree has been installed & built with?

[aj0strow]

@williamboman put it at the top of yarn.lock.

# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v2    <-- bump version

$NODE=5.6.0    # or whatever version / syntax

# ... dependencies ...

[williamboman]

How would it know when a dependency requires a rebuild? I don't think a diff on the major version number of the Node runtime necessarily requires a rebuild. Also, would that require every developer installing dependencies to run the exact same Node version?

[aj0strow]

@williamboman on a team of just me. You're right it doesn't belong in yarn.lock.

How often do you update node? I do maybe 2-3 times a year. Considering how infrequent, auto detecting likely has diminishing returns. yarn rebuild that rebuilds everything is probably the way to go.

If detecting is important, it would be ideal to introspect the binaries instead of writing a node version file (have to explain not to check it in to git). I'm not familiar with native extensions (gyp?) but yarn seems to know when they are absent, so maybe there's a way to detect if out of sync with node version.

[mgol]

@williamboman

How would it know when a dependency requires a rebuild?

This is a solved problem; npm & node-gyp already know that, no need to reinvent the wheel. On the C side there's NODE_MODULE_VERSION and on the JS side there's process.versions.modules.

@aj0strow

How often do you update node? I do maybe 2-3 times a year. Considering how infrequent, auto detecting likely has diminishing returns.

I update Node wherever a new minor/patch release comes out. Sometimes twice in a week. Different people have different needs, rebuilds should happen only when NODE_MODULE_VERSION is bumped which is easily detectable.

[vkrol]

We have some discussion with @BYK about it in Discord:

Expand me

vkrol

Hello. I found some annoying problem with the node-sass (and probably with other binary dependencies) and updating Node.js to new major version.
Steps:
1. Install Node.js 7.x
2. Add the node-sass dependency to the project via Yarn
3. Update Node.js to 8.x (new NODE_MODULE_VERSION)
4. Run "yarn install" (nothing happens)
5. Run "yarn run node-sass"
Error:

Error: Missing binding C:\...\node_modules\node-sass\vendor\win32-x64-57\binding.node
Node Sass could not find a binding for your current environment: Windows 64-bit with Node.js 8.x


Found bindings for the following environments:
  - Windows 64-bit with Node.js 7.x

This usually happens because your environment has changed since running `npm install`.
Run `npm rebuild node-sass --force` to build the binding for your current environment.

Is there any official solution of this problem?

byk

when you upgrade node versions, yarn doesn't know about native built artifacts. this was being discussed somewhere but I don't remember the outcome
for now you can delete ./node_modules/.yarn-integrity to force a reinstall
or just try yarn -f
so it would recompile those bindings

vkrol

i was thinking that this can help somehow https://media.discordapp.net/attachments/226793650552569856/360395418254835713/image.png?width=400&height=272

byk

I think arcanis added that field recently and it sure can help. That said we need to store the last node version those artifacts were built with to be able to retrigger a build
it is tricky but may be a nice addition
or may be just yarn install --rebuild-artifacts
we have a similar behavior for yarn global add which forces a rebuild of all artifacts
which is quite annoying :smile:
this can fix those problems both

vkrol

That said we need to store the last node version those artifacts were built with to be able to retrigger a build
I think that this is the very good idea.
Ideally "yarn install" should reinstall binary dependencies without additional user actions.

byk

I agree
Would you mind creating an issue for that with some context from this convo so we don't forget?
I can't promise to work on it but I can promise to make the ticket handsome enough to lure potential contributors

[jwchang0206]

Would yarn --force --build-from-source work?

[andreyrd]

The problem with yarn --force is that you then have to rebuild all packages. My use case is that I just want to rebuild node-sass when I switch node versions. yarn install node-sass --force prompts me to use yarn add node-sass which modifies my package.json.

[ctcpip]

@andreyrd see #5271

[rynop]

Is the workaround here til #5271 is implemented, to run npm rebuild?