Habitat: [RFC] Package Splits

[eeyun]

An issue came up in #general recently via community member @ShalokShalom that we thought might be worth discussion. TL;DR It might be an excellent feature to allow a plan.sh to define a package as well as a derivative package e.g. postgresql and postgresql-libs.

In pacman packages (PKGBUILD packages like those used by Arch linux) theres a first class concept of package "splits". Package splits in PKGBUILD scripts allow a single script to output two separate packages. Initially this may seem odd, but we already support similar behavior in a strange way.

The example of a package in pacman land thats a similar example in habitat would be the postgresql package.

In postgresql for the PKGBUILD package and arch repo you'll notice "Split Packages" in the details of the package metadata:

Architecture:   x86_64
Repository:     Extra
Split Packages:     postgresql-docs , postgresql-libs
Description:    Sophisticated object-relational DBMS
Upstream URL:   https://www.postgresql.org/
License(s):     custom:PostgreSQL

A split effectively says, look we don't actually want to compile this whole thing three times just to generate a meta package that will include only SOME of the pieces of this build. The example specifically being postgresql-libs or postgresql-docs. The PKGBUILD scripts provide a syntax that enables the package author to split that package into multiple packages derived from the same build. Which provides at least a bit of consistency for the build number of each of those things as well as a unified interface around pulling things like a client out of a package that by default will compile both a client and a server:

# Maintainer: Levente Polyak <anthraxx[at]archlinux[dot]org>
# Maintainer: Dan McGee <[email protected]>

pkgbase=postgresql
pkgname=('postgresql-libs' 'postgresql-docs' 'postgresql')

In habitat currently we sort of have a way to do this which is sourcing a separate plan at the top of the plan that is the "split".

# shellcheck disable=SC2148,SC1091
source ../postgresql/plan.sh

pkg_name=postgresql95
pkg_version=9.5.6
pkg_origin=core

The only issue being that we're still compiling twice to get the derivative package. It might be a sweet feature to allow plans to export a package as well as a derivative package. In doing so we might see some slight optimization in build times, but will also get packages that are inherently linked, with the same build-number. Generally we just wanted to discuss whether or not this is something we should consider. I imagine that a change like this could end up being rather extensive.

[elliott-davis]

I can see where this would be helpful (gcc, linux, postgres, etc). I started typing out how I had concerns about the UX of this for the end user, but then I read the link for PKGBUILD and it makes a ton of sense. I really like this idea and it seems like something we could do a POC for since (I think) it would only touch pkg_build

[jhuntwork]

👍 This would be especially useful if you want only run-time dependencies in your package, but would like to avoid the bloat of additional things which may be installed by the full package.

The source code of the makepkg script that ships with pacman may provide some useful insight/ideas: https://git.archlinux.org/pacman.git/tree/scripts/makepkg.sh.in

[themightychris]

With split packages, would build output be copied into the split packages so both the "mother" and "child" packages had the same files, or would they be moved and the child packages added transparently as dependencies for the mother package?

i.e. does postgresql and postgresql-libs BOTH contain the libs? Or does postgresql depend on postgresql-libs? I would think the later makes more sense, isn't that what other package managers do? (don't they have to since they're not isolated and immutable? you usually can't have two packages providing the same files)

[eeyun]

I dont know if theres an implementation that has really even been talked about yet!

[fnichol]

Currently, the only example pattern of this in core-plans is core/gcc-libs. It pulls in core/gcc as a build dependency so that it can vendor certain shared libs into itself in do_install(). That way, any Plan author that needs core/gcc doesn't additionally have to know that core/gcc-libs is also required (because it isn't). In an ideal world those libraries wouldn't exist in gcc itself but because a lot of software authors link to functions in these libs, most linux distros end up creating this package for runtime concerns. I used Arch Linux as the pattern in this case, and there is an equivalent postgresql-libs package in Arch which was going to be the basis of a corresponding Habitat Plan.

[fnichol]

Not that as with Arch we don't compile those libs over a second time, only copy and patch from another built package. Initially I was building gcc twice to get this second package and then realized how crazy that was from a build time perspective.

[johnjelinek]

I've been doing this pattern, but I've placed the libs as a runtime dep instead of buildtime so that way they get copied to the right place in the {{pkg.svc_var_path}} during init. When I deploy, I don't have to install the *-libs.hart, only the final package.

[eeyun]

We're looking to get this issue either closed out or accepted in short order if possible. It seems we have a flow that functions as a workaround for splitting (the example that @fnichol used above). This change would touch a significant area of our codebase.

What I would like to know from folks who have been following along is this: Do we feel as a community that having a first class wrapper for this functionality is important and should be added or are we comfortable that we can get the outcome with the pattern fletcher expressed. Either way a change of this nature is likely to be a breaking change and thus wouldn't be implemented until next year either _with_ 1.0 or with 2.0.

Final Note on the subject. This issue has had some good conversation on and offline. We would like to have a decision on this no later than 2 weeks from now. If we haven't reached a consensus that this IS functionality we really need then we will close the issue in 2 weeks.

[johnjelinek]

I am ok with @fnichol's pattern, but I don't really like having to build habitat/*-libs and then build, assuming a folder structure like this:

habitat/*-libs/plan.sh
habitat/plan.sh

[rsertelon]

This seems to not be useful, as @eeyun posed an ultimatum, there were not many answers. I think that if it is "just to" save some build time, it might not be worth the complexity that would be added IMHO.

Copying from a package build dependency feels weird too, even if it saves package developers the core/gcc-libs from most packages... Although it would be nice to have them explicitly stated instead :) (Because, what makes some libs authorized to be implicit, and some not then?)

[themightychris]

It's not just about saving build time. There are issues arising from having the same libraries in both core/gcc and core/gcc-libs. With both available during build time, multiple copies of the same libraries are available and builds can end up linked to the core/gcc copies that you don't want to have to make available at runtime.

A declarative splits syntax could provide a consistent way for a build of core/gcc to break out and depend on core/gcc-libs in sync. It seems like it would be sloppier solving this with two totally independent builds to have one plan only include some files and another exclude the same ones and stay in sync on that with essentially two inverted copies of the same configuration.

[eeyun]

Since this RFC hasn't gotten a significant amount of traction and the theoretical change is major in nature we're going to close this out.

Should any community members or users feel passionate enough about the issue we're always willing to reopen and revisit. For now I think folks are a bit middling and nobody is blocked by current functionality. As such, it's time to close it. Thank you everyone for contributing thoughts and dialogue!