Node: Official Debian/Ubuntu packages

[Daniel15]

Over at https://github.com/nodejs/docker-node/issues/262 I asked why the Docker image installs Node.js from a tarball rather than Debian packages, and the response was:

the parent Node.js project does not release DEB packages, it only provides tarballs on release. The popular NodeSource DEB repository is not official, so this image should not depend on it.

Please consider adding official Debian/Ubuntu packages to your builds. Ubuntu is probably the most popular Linux distribution so having an 'official' upstream .deb package (even if you just use Debian's or NodeSource's current packaging scripts) would be great.

Debian does have Node.js packages, but I think they only package 4.x, not 6.x or 7.x.

[MylesBorins]

/cc @nodejs/build

I for one am very interested in improving the current story for Debian / Ubuntu. @daniel15 are you or the other individual in the other thread involved in the Debian or Ubuntu project? would simply shipping an official Deb be sufficient, or would be also need to maintain a ppa? is a Deb / ppa sufficient or are you also interested in improving what is available on apt?

[Daniel15]

are you or the other individual in the other thread involved in the Debian or Ubuntu project?

I'm not, but there's a active Debian JavaScript packaging group that maintains all the current Debian Node.js packages (the packages that are included as part of Debian). Their mailing list is at https://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-javascript-devel ([email protected]). The issue is that it seems to move pretty slowly at times (eg. the latest Node.js version in Debian Testing is v4.6.1, and a newer release v6.9.1 seems to be stuck in "experimental" right now)

would simply shipping an official Deb be sufficient, or would be also need to maintain a ppa?

Custom repo/PPA would be great, but even a .deb would be good (people could wget the deb then use dpkg -i to install it). NodeSource have a custom repo, and it's documented on the Node.js website: https://nodejs.org/en/download/package-manager/#debian-and-ubuntu-based-linux-distributions. This repo works well, but my understanding is that it's not an "official" Node.js repo.

Perhaps the existing NodeSource repo could be absorbed into the Node.js Foundation and be served from the Node.js site rather than its current home of deb.nodesource.com. NodeSource have their packaging scripts available here: https://github.com/nodesource/distributions

[jbergstroem]

I think we need to work more closely with node.js packagers; be it debian, fedora or freebsd. I'm not sure if the responsibility of packaging and/or keeping it in shape should be on us though. They all handle security and bundled libraries differently -- even if we might be in a good position to make a call about node.js itself, we don't know as much about the packaging ecosystem it will live in.

[chrislea]

Hi all (and @jbergstroem in particular for this). I've done most of the work on the NodeSource repositories and have a pretty good understanding of how the official Linux distros deal with handling packages.

The TLDR is that the zen of packaging for both RedHat and Debian style systems (which includes Ubuntu, Fedora, CentOS, Mint, etc) isn't terribly compatible with how Node.js itself is developed, and what actual Node.js users expect. The short of it is that there are two fundamental notions for official packages that Node runs into head on:

1. Once a distribution ships a version, that version should never change for the life of the distribution. The idea is that if you get something working on a distro with whatever it shipped with, you should get a guarantee that nothing will change out from under you. If there are security issues, those should get backported to the version that shipped, instead of shipping out a new version.
2. Anything that can be broken out as a library (shared .so or otherwise), must be broken out. This means that V8, cares, libicu, etc must be linked in dynamically, and it _also_ means that npm must be packaged as a separate library.

Neither of these work terribly well for Node.js as it's moving very quickly. Things like V8 and libuv version changes happen all the time. This is the primary reason we started hosting our own respositories: our packages build everything in statically, and we push out new versions in cadence with the official releases, since we think that's what an end user wanting to use Node.js would expect when they install a nodejs package.

A final thing to note is that our repositories build everything in the way that third party repo users tend to expect, which means that we actually build Node for every distribution and architecture target every time. If you install our rpm Node package for 32bit Fedora Core 24, then that binary was actually built in a clean-room, 32bit FC24 chroot from scratch. Again, this is the expectation for third party repos set by things like Launchpad or EPEL. If the Node Foundation is interested in making a "blessed" .deb and/or .rpm package, it may be preferable to simply smoosh the bits from the binary tarballs you already create into said .deb or .rpm package format, which would then be totally distribution agnostic. It's also a lot easier to do that than it is to build for every distro / architecture target.

I know zero about *BSD packaging, so I'm afraid I can't be of any help there.

Feel free to toss in any other technical questions you may have around this stuff and I'll do my best to answer them.

[chrislea]

Sorry folks, but looking over my last response, there's another thing I think I should point out that may not be obvious here. When you talk about having an "official" package or packages, it's possible you're actually talking about two related, but distinct things.

1. Building blessed packages.
2. Building and maintaining blessed repositories.

Please understand that just building a deb or rpm package may not satisfy the needs of end Linux users unless there is _also_ a repository that they can use with standard tools like yum and apt. And maintaining that repository represents additional and ongoing work. Finally, you would likely want to maintain separate repositories for every currently supported version of Node (so, that's currently four supported versions of Node x two packaging formats assuming you also make rpm packages, so today that's eight repositories to keep track of).

[Daniel15]

Thanks for your reply, @chrislea!

I've done most of the work on the NodeSource repositories

The NodeSource repositories work pretty well! Since a lot of the hardwork has already been done, I wonder if it would be possible to make this repo "official" (in that it'd be hosted by the Node.js foundation rather than NodeSource) rather than doing something from scratch?

it also means that npm must be packaged as a separate library.

This is totally fine, and can be beneficial if someone doesn't use npm (for example, if they use an alternative package management client such as Yarn) 😄 The only reason I have npm is due to a few bugs in Yarn, but I plan on uninstalling it once Yarn covers all my use cases since I'll no longer have a use for it.

If you install our rpm Node package for 32bit Fedora Core 24, then that binary was actually built in a clean-room, 32bit FC24 chroot from scratch

Do you use something like Docker/lxc for this, or is it a regular plain chroot?

Anything that can be broken out as a library (shared .so or otherwise), must be broken out.

This is also pretty useful if you have other apps that use V8.

it may be preferable to simply smoosh the bits from the binary tarballs you already create into said .deb or .rpm package format

That's definitely an option, it'd work fine.

[jbergstroem]

@chrislea thanks for elaborating; much appreciated!

@Daniel15 said:
The NodeSource repositories work pretty well! Since a lot of the hardwork has already been done, I wonder if it would be possible to make this repo "official" (in that it'd be hosted by the Node.js foundation rather than NodeSource) rather than doing something from scratch?

Even if we wanted to create 'official' packages, it doesn't mean that distributions or packagers would adopt them as such. They have their own Q/A, security, audit and packaging routines.

@chrislea said:
Anything that can be broken out as a library (shared .so or otherwise), must be broken out.

I don't think this holds quite true for V8. It _used_ to be common practice but broke somewhere between 0.12 and iojs moving ABI quickly. It's also increasingly hard to decouple V8 in general. As for c-ares, libuv and especially openssl I can understand packagers motives as to why they'd want to link against them. Most distros seems to do this (bsd included) with success. Also, most distributions have a different threshold to 'breaking out'. Far left would be things like Gentoo (everything sans v8) whereas some distributions only link against zlib and openssl.

Openssl is also a good example of where we will see increased packaging pain since distributions are moving forward with a co-existing libressl. Again a good example of why our position will be different to other distributions.

[Daniel15]

In this case I don't mean "official" in terms of getting the packages into
the Debian repo, I just mean in terms of having a first-party repo rather
than having a third party (NodeSource) hosting it :)

Sent from my phone.

On Nov 7, 2016 11:49 AM, "Johan Bergström" [email protected] wrote:

@chrislea https://github.com/chrislea thanks for elaborating; much
appreciated!

@Daniel15 https://github.com/Daniel15 said:
The NodeSource repositories work pretty well! Since a lot of the hardwork
has already been done, I wonder if it would be possible to make this repo
"official" (in that it'd be hosted by the Node.js foundation rather than
NodeSource) rather than doing something from scratch?

Even if we wanted to create 'official' packages, it doesn't mean that
distributions or packagers would adopt them as such. They have their own
Q/A, security, audit and packaging routines.

@chrislea https://github.com/chrislea said:
Anything that can be broken out as a library (shared .so or otherwise),
must be broken out.

I don't think this holds quite true for V8. It _used_ to be common
practice but broke somewhere between 0.12 and iojs moving ABI quickly. It's
also increasingly hard to decouple V8 in general. As for c-ares, libuv and
especially openssl I can understand packagers motives as to why they'd want
to link against them. Most distros seems to do this (bsd included) with
success. Also, most distributions have a different threshold to 'breaking
out'. Far left would be things like Gentoo (everything sans v8) whereas
some distributions only link against zlib and openssl.

Openssl is also a good example of where we will see increased packaging
pain since distributions are moving forward with a co-existing libressl.
Again a good example of why our position will be different to other
distributions.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/nodejs/node/issues/9496#issuecomment-258942302, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAFnHULbmPlSKu-hBrWhgXsFdOazaXymks5q74DPgaJpZM4KqwZz
.

[chrislea]

@Daniel15 If the goal is just to have a repository that the Node Foundation has blessed as official then I'm not sure why they wouldn't just say "the NodeSource repos are the official ones" and be done with it, since that's effectively what's already happening in the world. The issue I think people may have there again is that the binary that gets installed will be technically different than the one contained in the official binary tarball for a given OS / architecture.

The issue of who hosts it is actually a bit trickier as we are increasingly tied into the AWS ecosystem for the build / host cycle of all of this. For example, when we build new packages, we spin up massive 36 core boxes to do it on since we have so many things we need to build, and then from there they have to get pushed into the repositories themselves, which are fronted by CloudFront, and then we spin the huge machines down. It takes quite a while to build everything when there is for example a security release that requires building all supported versions fresh. The ARM builds in particular take forever.

As for the whole linking-vs-not thing, we link everything statically and include npm in the main package because that's what the official tarballs do, and that's certainly what normal users expect. The standard way that Debian and RedHat handle this doesn't work for people in this case.

For the Debian style builds we use the sbuild tool suite to do the chroot builds. For the RedHat style builds we use mock for the same purpose.

[jbergstroem]

@chrislea said:
As for the whole linking-vs-not thing, we link everything statically and include npm in the main package because that's what the official tarballs do, and that's certainly what normal users expect. The standard way that Debian and RedHat handle this doesn't work for people in this case.

I think what users expect might vary depending on where they get node.js from. For instance, you wouldn't expect debian to ship a nginx with a static zlib, openssl and glibc. That's why different linux distributions takes different paths. I agree that the 'default' case of Node.js likely is "batteries included" -- a shared build could accomplish this just as well. I guess we're slowly moving away from the original question but it's good to have this chat here for future eyeballs.

Do we have any actionable items? Is the result of this issue to reach out to packagers? I've had something like that on my list for a while but I just lack the horsepower.

[sam-github]

For example, when we build new packages, we spin up massive 36 core boxes to do it on since we have so many things we need to build,

@chrislea Who is "we"? Nodesource, or Debian?

nodesource's PPAs are quite nice, but don't contain all the architectures that can be downloaded from nodejs.org. It would be nice to have the full platform support, and for that the foundation build machines should be available.

It doesn't seem likely that the distros can be convinced to ship node patches faster, which is why having the PPAs is helpful.

Do they statically or dynamically link OpenSSL? I assume they do a node-default build, against the vendored openssl (and other) dependencies?

[Trott]

Should this remain open?

[gibfahn]

Should this remain open?

I think this is still a valid question/feature request. IDK if it's being discussed elsewhere, but unless it's a duplicate we should keep it open.

[aduh95]

What about using Snapcraft (available on all major Linux systems, installed by default on Ubuntu since 16.04)?

It would let the node team free to update its dependencies and the package will be available to users without needing to add a repo to their package manager.

EDIT: As of January 2018, NodeSource has started working on providing Node.js nightly snaps.

[refack]

Due to the volunteer nature of this project, and hence the limited resources of the BuildWG the current consensus is that the NodeSource deb (and rpm, and snap) repos are "blessed" and should be as trusted as any 1st party distribution.

[rvagg]

@aduh95 I maintain the node snap, it's got all the main release lines as well as nightlies. I use it and a bunch of other people do too, it's pretty sweet for a dev machine.

sudo snap install node --channel=10 --classic

sudo snap refresh node --channel=11

[aduh95]

@rvagg Oh I didn't know you have stable release as well! That's awesome, thank you for that.

BTW, maybe you want to update the README section of the snap folder which misled me:

https://github.com/nodesource/distributions/blob/19762a48e8dabd8e2e5362c671ceb2bcfff56b95/snap/README.md#L4

[rvagg]

README was updated long ago, you're looking at an old commit. See https://github.com/nodesource/distributions#snap

Also, today we got an npx alias approved, so you can run this from the snap along with npm and yarn too!

[aduh95]

Actually i was talking about the README of the snap subfolder https://github.com/nodesource/distributions/tree/master/snap, indeed the infornation of the main README is more relevant,

[rvagg]

Ah, thanks for that @aduh95, updating via https://github.com/nodesource/distributions/pull/753