Node-serialport: electron-builder cross platform rebuild

[akaJes]

• __SerialPort Version__: 4.0.7
• __NodeJS Version__: 6.10.2

• __Operating System__ and __Hardware Platform__:ubuntu 4.17 x64

• Have you checked the right __version__ of the __api docs?__:

• Are you having trouble installing and you checked the __Installation Special Cases__ docs?
• Are you using __Electron__ and have you checked the _Electron Docs_?:no issues with native npm

Summary of Problem

generated installers for mac and win has ELF(from main ubuntu x64) executable - i suppose ia32 application has also x64 elf

Steps and Code to Reproduce the Issue

just try to wrap any sample from electron with electron-builder or electron-packager with
added require('serialport') to source code and get something like this #1103

electron-builder has option npm rebuild but it can't regenerate gyp config for desired platform/arch and rebuild it

is there another way to request from git prebuilt release packages for desired platform/arch?

[akaJes]

./node_modules/.bin/electron -v
v1.6.10
echo "console.log(process.version)">v.js && ./node_modules/.bin/electron v.js
v7.4.0

[schrockwell]

I just came across this issue, too, and have narrowed down the cause. I'm also using electron-builder to bundle up Electron app distributables. That being said, this issue actually has nothing to do with Electron, but is some weird bug with npm rebuild:

# This works:
npm rebuild --build-from-source

# This does nothing:
npm rebuild serialport --build-from-source

The particular problem is that electron-builder executes the latter, and for some reason that particular incantation of npm rebuild with the explicit serialport argument is a no-op.

Here's the full output with verbose logging:

$ npm rebuild serialport --build-from-source -ddd
npm info it worked if it ends with ok
npm verb cli [ '/usr/local/Cellar/node/6.2.1/bin/node',
npm verb cli   '/usr/local/bin/npm',
npm verb cli   'rebuild',
npm verb cli   'serialport',
npm verb cli   '--build-from-source',
npm verb cli   '-ddd' ]
npm info using [email protected]
npm info using [email protected]
npm info readInstalled object
npm verb exit [ 0, true ]
npm info ok

This is on the latest version of serialport, 5.0.0-beta3 on master. I have confirmed the same behavior on Windows and macOS.

@akaJes: As a quick hack, you can modify yarn.js in the electron-builder module and comment out the lines that append the native module names to the rebuild command. Like this:

// execArgs.push.apply(execArgs, _toConsumableArray(nativeDeps.map(function (it) {
//     return it.name;
// })));

[akaJes]

i can invoke a prebuild script to get required platform/arch and target via electron-builder API for each case

npm rebuild serialport --update-binary --target_platform=win32 --target_arch=x64 --verbose --target=6.9.1

but on github for electron no exist prebuilt releases https://github.com/EmergingTechnologyAdvisors/node-serialport/releases
electron abi versions https://github.com/lgeiger/electron-abi

it must be serialport-v4.0.7-node-v53-win32-x64.tar.gz at least for windows with ABI 53
how to build these for other ABI and platform/arch ? or where to get these?

ABI versions from node-pre-gyp https://github.com/mapbox/node-pre-gyp/blob/master/lib/util/abi_crosswalk.json

[akaJes]

i solved it
https://www.npmjs.com/package/serialport-v4
this is the same but with precompiled for all latest version win/linux node/electron (i have no mac)
builder script https://gist.github.com/akaJes/331371ba5ac1df18b22c0531b03dc847

who has mac and can build https://github.com/akaJes/node-serialport/wiki/Platform-build

[reconbot]

We have limited support for electron in our build processes, it gets very messy very quickly. Because of that it's hard to provide binaries for every situation. Happy to add any further electron information to the docs if you would write it up. https://github.com/EmergingTechnologyAdvisors/node-serialport#electron

[akaJes]

I understand your situation, but with my tool, described in previous post, u can create the prebuilts for 32/64 bits for electron/node for list of various versions and upload these to github by One Click
Check out my branch v4

[develar]

Please do not use node-pre-gyp. Please use prebuild. https://github.com/electron-userland/electron-builder/issues/1812#issuecomment-314673270

[develar]

Migration to prebuild is very easy, you can see https://github.com/atom/node-keytar/pull/67/files

It will make a lot of Electron users happy.

[reconbot]

Our entire binary publishing is build on pre-gyp and pre-gyp-github, which will need to be changed. However ugly you find it there are a lot of packages out there that use it.

I'm not against considering it however it should noted that prebuild historically didn't support all the platforms we supported. It's been a while.

[develar]

@reconbot do you have plans to migrate to prebuilt? Electron-builder now doesn't force build from sources and fallback to build works only in case of prebuild. But node-pre-gyp doesn't fallbacks. Yet another report about it today:(

[akaJes]

With my github token file and one click on ubuntu and windows i get this set https://github.com/akaJes/node-serialport/releases/tag/4.0.9 and forget for this issue :)
Then i renamed a package name and have built all electron install packages for all OS using ubuntu also with one click
Ony one trouble - i have no Mac :(

[reconbot]

@develar I'm not opposed to it, but it will totally delay our 5.0.0 release which fixes a ton of bugs and cpu issues.

[hanfengcan]

rebuild use
node-pre-gyp rebuild --target=x.x.x --arch=x64 --target_arch=x64 --dist-url=https://atom.io/download/electron
by youself

target is the electron version

[develar]

@hanfengcan electron-builder does it for you. But since 19.x we don't pass force build from sources anymore. prebuild correctly fallbacks to sources if no suitable bin. node-pre-gyp doesn't. That's problem. It is not a solution to manually invoke this command.

[oldrich-s]

@develar Could you say what IS the solution? Thanks ;)

[develar]

@czb Until this issue is not fixed — the only workaround: "buildDependenciesFromSource": true in the build property of package.json (i.e. you can build electron app only on target platform).

[oldrich-s]

@develar Thanks, works perfectly :+1:

[akaJes]

@reconbot
I have build all possible latest version for v4 (with one click on each OS) https://github.com/akaJes/node-serialport/releases

working package https://www.npmjs.com/package/serialport-v4
electron build script(builds for all OS from your's OS) https://gist.github.com/akaJes/331371ba5ac1df18b22c0531b03dc847
working project https://github.com/akaJes/marlin-config

And i'm waiting for v5 release! :)

[reconbot]

Maybe we can get electron builder working on travis? I think keeping a fork
up to date would be harder than getting your build system into core ;-)

---

Francis Gulotta
[email protected]

On Wed, Jul 26, 2017 at 12:35 AM, akaJes [e-mail] wrote:

@reconbot https://github.com/reconbot
I have build all possible latest version for v4 (with one click on each
platform) https://github.com/akaJes/node-serialport/releases

working package https://www.npmjs.com/package/serialport-v4
electron build script(builds for all OS from your's OS)
https://gist.github.com/akaJes/331371ba5ac1df18b22c0531b03dc847
working project https://github.com/akaJes/marlin-config

And i'm waiting for v5 release! :)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/EmergingTechnologyAdvisors/node-serialport/issues/1180#issuecomment-317973569,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABlbmK5fNGYVPvHZNgrAN8rGHor45BYks5sRuwugaJpZM4Nq2Ys
.

[develar]

@akaJes Not yet clear for what do you need special build script for electron-builder. Or prepare PR to migrate to prebuild, or set "buildDependenciesFromSource": true as current workaround of node-pre-gyp bug.

[akaJes]

@develar how i can set "buildDependenciesFromSource": true ?
when i build from my ubuntu the installers for windows/mac?
i tried it and get x64 ELF in all installers, because my ubuntu doesn't want to compile windows binaries :)

[akaJes]

@reconbot it is good idea, but how to do it only single time when the release publication and can travis to upload a compiled binaries to the github?
this package https://www.npmjs.com/package/pre-gyp-pack can build and upload to the github, but how to make it friendly with travis and run it only when it need?

[reconbot]

We already run it once per platform by detecting the tag off the release.
We build natively for each platform.

On Wed, Jul 26, 2017, 10:19 PM akaJes [e-mail] wrote:

@reconbot https://github.com/reconbot it is good idea, but how to do it
only single time when the release publication and can travis to upload a
compiled binaries to the github?
this package https://www.npmjs.com/package/pre-gyp-pack can build and
upload to the github, but how to make it friendly with travis and run it
only when it need?

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/EmergingTechnologyAdvisors/node-serialport/issues/1180#issuecomment-318261286,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABlbvlM8gLlMZcO2Py1Vwm4KGSVw8yZks5sSB3egaJpZM4Nq2Ys
.

[akaJes]

i found these

- if [[ $PUBLISH_BINARY == true ]]; then node-pre-gyp package; fi;
- if [[ $PUBLISH_BINARY == true ]]; then node-pre-gyp-github publish --release; fi;

this script builds and publishes only current version
and i see that travis doesn't like windows :(

i'm using only npm run package it calls pre-gyp-pack from this config https://github.com/akaJes/node-serialport/blob/v4/package.json
and gets all desired versions at once then publishes these to the github using my personal token

[develar]

@akaJes until node-serialport not migrated to use prebuild instead of node-pre-gyp — you can build only on target platform. Not cross platform.

[akaJes]

@develar don't worry, because i can build the cross platform installers without any troubles ;)

[akaJes]

the sample of builder for my repo and npm
https://github.com/EmergingTechnologyAdvisors/node-serialport/compare/master...akaJes:v5?expand=1
the result is https://github.com/akaJes/node-serialport/releases/tag/5.0.0 (macos will be later)

then for each OS:
put the token file at home folder .github.json

{ "OAuth":"xxxxxxxxxxxxxxKEYxxxxxxxxxxxxxx" }

and type npm run package for building and publishing

[reconbot]

Thanks to examples from @akaJes I released [email protected] with prebuild. Please give it a try.

[reconbot]

To clarify this should work out of the box with electron-builder https://github.com/EmergingTechnologyAdvisors/node-serialport/releases/tag/v5.1.0-beta5 brings the number of precompiled binaries up to 60.

[reconbot]

@develar the size of the windows binaries are tiny in comparison to the node-pre-gyp-github uploads. Basic testing says they work fine, but do you have any idea as to why they'd be so much smaller?

[akaJes]

Thank you @reconbot and everybody involved in this project and its maintenance and development
This awesome for IoT projects and in my too :) https://www.instructables.com/id/Marlin-config/

[develar]

the size of the windows binaries are tiny in comparison to the node-pre-gyp-github uploads.

Probably debug information / runtime (I see in your PR that --msvs_version=2013 is removed).

[reconbot]

@akaJes glad it helps you in your projects!

Can anyone confirm that [email protected] works for electron without issue?

[akaJes]

works only when it builds for the same platform
for crossplatforming needs some changes in the prebuild-install module and needs additional script like this https://gist.github.com/akaJes/331371ba5ac1df18b22c0531b03dc847
with this code

config.beforeBuild=(args)=>new Promise((done,fail)=>{
  var cmd = 'npm run install '
  var env = Object.assign({},process.env,{ npm_config_platform: args.platform.nodeName })
  exec(cmd,{
    env: env,
    cwd: path.join(__dirname,'node_modules','serialport'),
    stdio: 'inherit'
  },(err,stdout,stderr)=>err?fail(stderr):(console.log(stdout),done(stdout)))
});

and if(opts.platform == process.platform) at https://github.com/mafintosh/prebuild-install/blob/master/download.js#L123

[develar]

@akaJes I cannot believe. It works for me in case of keytar and should also works for [email protected]. If it doesn't work — please file issue to electron-builder (and sorry, but please do not promote your incorrect script).

[develar]

@farfromrefug Could you please confirm that it [email protected] correctly works for you?

[akaJes]

@develar I see that in your post the key word is 'should also'. did you read the subject of this issue?

[develar]

@akaJes I see that all required electron binaries are published, so, cross platform rebuild should work. You can enable debug logging (env DEBUG=electron-builder) to inspect output of npm rebuild (it is called automatically on build, no additional script is required). I will check serialport@6 manually.

[akaJes]

everything stops right here https://github.com/mafintosh/prebuild-install/pull/41/files when you assemble installer for one platform from another platform

[akaJes]

I found strange npm behaviour: (i have in package.json "serialport": "5.1.0-beta5", )
but when i call npm rebuild serialport (the same does electron-builder) - does nothing
only when i call npm rebuild [email protected] it rebuilds as npm rebuild - for all modules

[develar]

@akaJes Ouch, WTF. https://github.com/npm/npm/blob/latest/lib/rebuild.js#L59 if (!semver.satisfies(data.version, v, true)) continue Thanks, to investigate.

[akaJes]

@develar I changed this line https://github.com/electron-userland/electron-builder/blob/165e11c6f7d6ea4be589a39823caa50c71b78928/packages/electron-builder/src/util/yarn.ts#L171 in yarn.js to

            execArgs.push.apply(execArgs, _toConsumableArray(nativeDeps.map(function (it) {
                return it.name+"@"+it.version;
            })));

and well done without any additional scripts

[develar]

@akaJes Thanks! Your fix released — 19.20.1 (prerelease version for now, tagged as next).

[develar]

So cool that now I can say "Just use serialport 6.x" for Electron users. @reconbot Thanks a lot. The dream is one step closer.

[perkyguy]

Thanks @reconbot!

So cool that now I can say "Just use serialport 6.x" for Electron users. @reconbot Thanks a lot. The dream is one step closer.

@develar was able to say exactly this and made my life so much easier!

[akaJes]

@develar i read the books and followed the instructions

[reconbot]

Since we're now using prebuild with v6 I think we're ok here.