Habitat: [Design] Binstubs for Shell based systems

[smacfarlane]

Replacing binlinks with binstubs on shell based systems.

This discussion is a followup to an issue reported in #6791. Additional context can also be found in #6796, #6700, #6689 and #6670.

Background

When a user invokes hab pkg binlink on Linux it will generate a symbolic link to the executable command in the specified package. For compiled software with no additional environmental concerns this is a relatively safe operation. However, it sets the expectation that all executable commands in Habitat packages can be binlinked. This breaks down when the command is a script that invokes other commands that are not present in the system paths or if those commands take different options (ex: busybox vs gnu tools), and when there are special environmental concerns that the package has (ex: PYTHONPATH). In other words, when these commands are symbolically linked they no longer have the environmental guarantees that hab pkg exec or the supervisor give them.

On Windows systems, we provide a "binstub" instead of the equivalent of a symbolic link. This affords us the opportunity to set the environment correctly so that the behavior should be identical to hab pkg exec.

Difficulties

The binstub approach, while arguably safer than a symbolic link, does present numerous challenges in Linux.

The #! line in particular is problematic. It's reasonable to assume /bin/sh will be present on systems, but that creates a dependency on system tools which goes against the overall design of Habitat. In addition, /bin/sh won't necessarily be the same across platforms. A quick survey of latest container images for common distributions found that /bin/sh is in all cases a symbolic link:

Alpine:      busybox
Arch Linux:  bash 5.0.7
Centos:      bash 4.2.46
Fedora:      bash 5.0.7
Ubuntu:      dash 

Finally, all habitat exporters would need to be aware that they would need to provide /bin/sh, as we wouldn't know ahead of time if a package expected to use binlink functionality.

Another difficulty is when the #! becomes self-referential, and ends up calling itself leading to #6791. We can attempt to detect this by determining if the link target is the same as the #!, but as we discovered with the chroot studio, when your / moves after the stub is created, it's much more difficult, maybe impossible, to make this assertion when generating the stub. It's also possible that there are other cases where a stub could become self-referential we are unaware of at this time.

A potential solution to the above issue would be to always use a fully-qualified Habitat path to a shell, but that creates an implicit coupling between the hab cli and the package we use, and opens up the potential for broken stubs if the requisite packages are deleted. It would also have ramifications on version upgrades as you could potentially have binstubs pointing at different releases of packages.

In addition to the above, there are some unknowns:

• Are there additional cases where #!/bin/sh might lead to errors?
• Is the environment setup we perform in the script correct across all possible shells #!/bin/sh might be?
• Is this a value we need to allow users to override?
• What else do we need to take into account?

Motivation

As as Habitat developer I want to understand the use case of binlinks on a Linux system, so that we can implement a solution that is safe and doesn't lead to unexpected behavior. Additionally, I want to make note of potential edge cases and document workarounds and any unsupported scenarios.

Specification

None at this time.

[baumanj]

The hab binlink command makes invoking habitat-installed commands more convenient, but at the cost of transparency. If a user runs hab pkg exec foo, it is clear that the command foo may have special behavior according to the hab ecosystem, whereas a naked foo doesn't give any indication and requires the operator to remember whether foo was installed and binlinked by habitat or a package manager or other means. Certainly hab pkg exec foo is a bit cumbersome, but that can generally be addressed with shell aliases or other means. Are there use cases where binlinking is _necessary_ to enable certain use cases?

Given the inherent complexity and proven potential for serious defects, It seems worth considering the possibility of removing the binlink feature and weighing its value against its continued development cost and risks.

[raskchanky]

Given the inherent complexity and proven potential for serious defects, It seems worth considering the possibility of removing the binlink feature and weighing its value against its continued development cost and risks.

One downside here (other than the disappointment this would cause) is that this would break our install mechanism. Right now, we download hab from bintray and then use that to do a hab pkg install --binlink --force core/hab. That's convenient because future hab upgrades can be accomplished via the same mechanism. While the installation mechanism is certainly fixable, I'd argue the upgrade mechanism is quite elegant, and it would be a shame to lose.

A potential solution to the above issue would be to always use a fully-qualified Habitat path to a shell, but that creates an implicit coupling between the hab cli and the package we use, and opens up the potential for broken stubs if the requisite packages are deleted.

I might be in the minority here, but I think it's reasonable to expect a binstub to create a tight coupling between the hab CLI and the package we use. If the package that the stub points to is deleted, having the stub fail seems fine to me. Maybe I'm naive, but I can't imagine people deleting packages and then being surprised later when something breaks.

It would also have ramifications on version upgrades as you could potentially have binstubs pointing at different releases of packages.

I would imagine that once a binstub is created against version X of package Y, it will always point to version X, right? If you installed a new version Z of package Y, you'd need to regenerate your binstubs in order to have them point to the right version. This seems totally reasonable and expected behavior, imo.

On the whole, I think binstubs provide a reasonable middle ground between the error prone but highly convenient nature of binlinks and the safe but verbose nature of doing everything via hab pkg exec.

[baumanj]

While the installation mechanism is certainly fixable, I'd argue the upgrade mechanism is quite elegant, and it would be a shame to lose.

True, but that mechanism doesn't work on macOS. Maybe we should have a proper hab command for a self upgrade.

[raskchanky]

True, but that mechanism doesn't work on macOS. Maybe we should have a proper hab command for a self upgrade.

That's a good point.

[stevendanna]

I think binlink is an important feature to keep, but it would be very helpful to clean up what appears to many users as inconsistency over where the links actually get created and how they work. I've tended to use binlinks in two ways:

1) To construct minimal systems from _just_ hab packages. If I want to be able to run non-habitatized packages and scripts, it is often helpful to be able to use hab to bootstrap a minimal environment for those tools to run against.

2) For software being distributed as hab packages, being able to binlink the main command-line tool of the package makes the software much more usable. For example, when users install Chef Automate, we binlink the chef-automate command for them and then update that binlink on upgrade.

I might be in the minority here, but I think it's reasonable to expect a binstub to create a tight coupling between the hab CLI and the package we use. If the package that the stub points to is deleted, having the stub fail seems fine to me. Maybe I'm naive, but I can't imagine people deleting packages and then being surprised later when something breaks.

I agree with this sentiment. Especially since I think we can probably get away with it being a single package that we depend on. That said, I think I might have a way for our only binlink requirement to be hab itself.

A potentially bad design for binstubs

You can give 1 argument to your #! line, hence why #!/usr/bin/env bash works.

What if we had a new hab command. hab binstub that worked as a valid shebang.

#!/hab/bin/hab binstub
origin/package/version/release command

The hab binstub command would know how to parse this minimal file format and then run hab pkg exec origin/package/version/release command, passing through other arguments to the underlying command. I've written a small POC of this. I didn't use full idents here out of laziness, as I just wanted to prove to myself it could work:

> cat ./ls-binstub-test 
#!/hab/bin/myhab binstub
core/coreutils ls
> ./ls-binstub-test -al /tmp/
total 24
drwxrwxrwt 4 root root  4096 Aug 13 08:25 .
drwxr-xr-x 1 root root  4096 Aug 13 08:25 ..
drwxr-xr-x 2 root root  4096 Aug 13 08:25 aliases
drwxr-xr-x 2 root root 12288 Aug 13 08:25 docs
> cat ./bash-binstub-test     
#!/hab/bin/myhab binstub
core/bash bash
> ./bash-binstub-test -c 'exit 13'
> echo $?
13

Note here that I've assumed that we have a /hab/bin/myhab there might be some work to do to make sure we always know where the hab executable will be, but we could potentially rely on the PATH for that or develop a new feature such that we always had a hab at /hab/bin/hab.

We would likely need a slightly different file format than I've used here to account for commands with spaces in the name. However, if we were to have a restriction that idents and command names couldn't have newlines then I think the file format could be a simple line-oriented one.

I also think it might be advantageous to not place these stubs directly on the PATH but rather in a separate folder in /hab/ (maybe /hab/binstubs) that we would then symlink into.