Esy: NPM Publish: Cross-platform artifact management

[bryphe]

Goal

For esy, our goal is to scale out esy such that it works on Linux, Mac, and Windows - I can run npm install -g esy from any of these environments and start using the esy workflow.

Current Publish Strategy

Our current strategy today is to include both the Linux and Mac binaries in the npm package, which is problematic for Windows support. Our build is done on a Mac, and we copy over artifacts to Linux- and Mac- specific prefixes, and then use a postinstall script to copy the right binaries to the right place.

This strategy won't scale as-is - it works today because we can run the Linux build from a Mac Docker container, so we can get the Linux and Mac binaries in the same place, but this won't scale to Windows (you can't run a Windows Docker container from Linux/Mac, or a Mac Docker container from Windows).

The postinstall script also won't work on Windows today, because it's a bash script that knows about Linux/Mac binaries, but not Windows.

Proposal

My recommendation is to _download per-platform binaries in postinstall_. This is in-line with how several other prebuilt binary packages work (ie, electron, bsb-native).

When publishing a release, we'd have all the CI strategies push up a tar.gz of their built _releases folder as a release artifact - we'd have:

• esy-release-linux.tar.gz
• esy-release-darwin.tar.gz
• esy-release-win32.tar.gz
• esy-release-win64.tar.gz

Potential Benefits

• Makefile: We can remove our per-platform copying targets, simplifying our Make process. Each platform can build its one esyCommand.exe, esyi.exe, etc - and these artifacts can be pushed up from our CI machines.
• postinstall: We wouldn't need to do the per-platform copying of binaries we have today. We would run a download/extract script to get the build artifacts.
• Smaller NPM package per-platform - instead of needing to bundle _all possible binaries_ - we can just grab the right ones on postinstall.

Alternatives

If we wanted to stick with the strategy we have today, and not have any network traffic on postinstall, we could create an 'uber' package that downloads all the artifacts across platforms and bundles them, refactoring our postinstall script to a node script that is aware of all platforms.

[andreypopp]

I agree that the current release workflow isn't optimal for the reasons you listed + the fact that release binaries are now build on my machine which isn't really a good thing.

I think I still prefer to have a universal "fat" npm package which includes binaries for all supported platforms. Ideally such release should contain source code and all needed to bootstrap esy from source code on a "reasonable" host but that's out of topic for now.

Action points as I see them:

1. Make all CIs (Travis, AppVeyor) build releases and upload them to GitHub for tagged commits.
2. Rework manual release process to construct a release by download CI built binaries into _release directory and wrap them with a "shell" npm package.
3. Replace postinstall.sh with postinstall.js for the "shell" package which "activates" the correct set of binaries on the client host.

[bryphe]

Sounds good @andreypopp ! 1, 2, 3 actually aren't blocked by Windows work - we could start by getting this infrastructure set up for just Linux / OSX - and then AppVeyor will be able to plug into it.

[jordwalke]

The one problem with downloading binaries in postinstall, is that it's difficult to get working in corporate environments. In corporate environments, you'd use yarn offline packages or npm install --no-scripts which would not run the post install. Then you'd check those tarballs in and run the npm postinstall process at build time on an isolated network. So if it retrieves the binaries at postinstall time, this won't work.

[bryphe]

Makes sense @jordwalke ! This looks like the implementation @andreypopp has today - we'll use the CI machines to push the artifacts up to a release, and then our make release script will download the appropriate artifacts, bundle them into a 'fat' NPM package w/ binaries for all platforms - then we publish that to NPM.

[andreypopp]

I confirm we still publish "fat" release on npm with binaries for both linux and darwin in there. So there's no network access during postinstall step.