Habitat: Allow Supervisor to purge old versions of a service

[christophermaier]

As the Supervisor updates its services (and itself), we can accumulate a large number of old versions of the package on disk. Over time on long-lived systems, this can result in excessive disk usage. This is exacerbated with larger numbers of services, as well as with frequently-updating services.

Since we have (almost?) all the necessary logic for removing packages and dependencies present in the hab pkg uninstall command, we should be able to utilize this in the Supervisor directly. For instance, imagine a mode in which the Supervisor keeps only the most recent 5 releases of services it is running (configurable, of course). As it updates for the 6th time, it would delete the oldest version on disk, as well as any dependencies of that release that weren't being used by any other service that was running.

When a service is unloaded, we could remove all releases of that service on disk.

I'm not sure if it makes more sense to have this capability enabled Supervisor-wide, or on a service-by-service basis. The former seems to be more straightforward at the moment, but arguments could be made for the latter.

[christophermaier]

One idea that has been floated internally is the notion of a "package registry" that could track things like when a package was installed, by what means (e.g., explicit hab pkg install or by hab svc load), when, etc. This could be useful for Supervisor-initiated deletions because you would have a better idea of what is safe to be deleted and what should remain on disk (e.g., because something was also installed manually, which could indicate it is being used by a human; we'd only want the Supervisor to delete things that it is fully responsible for).

This is, of course, not a trivial task.

[eeyun]

As was also discussed internally, we can probably learn some from the prior art of the NixOS project and a few others. The nix manual covers their garbage collection here: https://nixos.org/nix/manual/#sec-garbage-collection (this wont give you the full ins and outs of the way it all works). Effectively though NixOS handles removal of a package by creating a new "generation" which is at its highest level a metadata snapshot of the symlinks in use by the system, the new generation will not include that package's symlinks. So when you run the garbage collector it takes the arguments you pass WRT the desired age of things to remove and Nix compares the generations to determine what directories can get blasted away.

Some more in-depth info is probably worth perusing if only for our own edification.

[davidMcneil]

This is my proposed implementation. I think it gets us most of the functionality we need while being straightforward.

Edit: See a simplified proposal here. And an even more simplified proposal here.

Add an --auto-uninstall CLI option to the hab pkg install, hab pkg uninstall, hab svc load, and hab sup run commands. This option would take an unsigned integer as an argument indicating the number of most recent packages to keep. Below is the behavior for each command:

Note - when the term "uninstall" is used it includes removing all unused dependencies of the package.

hab pkg install --auto-uninstall=3 - Install the specified package. After the installation, uninstall all but the 3 most recent versions of the package. The user could set --auto-uninstall=1 to remove all versoins except the version they just installed.

hab pkg uninstall --auto-uninstall=5 - This command makes the most sense if a non-fully qualified package ident is used. That would remove all but the 5 most recent packages from the system. Using --auto-uninstall=0 would remove all versions of the package from the system. The name --auto-uninstall feels awkward for this command. Something like --keep-latest would make more sense, but I like the consistency across all commands. Not sure what is best to do here?

hab svc load --auto-uninstall=2 - This command really only makes sense if a service update --strategy is also used. After the service updates, the Supervisor would uninstall all but the two latest packages.

hab sup run --auto-uninstall=2 - This command would behave identically to the hab svc load case for the package the supervisor was instructed to immediately load. This command would need another option, --self-auto-uninstall, that would apply the same logic when the Supervisor itself updated.

Questions

• What to actually name the --auto-uninstall option? Other potential options: --auto-cleanup, --keep-latest, --auto-uninstall-down-to... Naming is hard. Very open to other suggestions.
• Is having --auto-uninstall awkward for the hab pkg uninstall command?

When a service is unloaded, we could remove all releases of that service on disk.

• This implementation does not cover this use case. I do not fully understand when this behavior would be needed/useful?

One idea that has been floated internally is the notion of a "package registry"

• While I think a registry could enable some really cool user experiences, I think this implementation gets us most of the functionality we went with considerably less complexity.

we can probably learn some from the prior art of the NixOS project

• This is a really cool solution! I did not see a direct correlation to solving this particular problem without adding several new and potentially unnecessary concepts to habitat, but I would love to have more discussion around this.

[ericcalabretta]

@davidMcneil For naming at-least, I like the idea of --auto-cleanup since it best describes the new behavior. Auto-uninstall seems deceiving to me, most users that do hab svc load probably don't even understand or need to understand what the install behavior is, so uninstalling would be confusing.

Would the option be at a supervisor & service level? I suppose users could find examples for either, but for me I like blanket setting it at the supervisor level. Not adding an additional thing to think about when loading a supervisor, it's solving more of a global disk problem than an individual service concern.

How do we know what unused dependencies are? An old Dep of Service A could be a current Dep of Service B.

[davidMcneil]

so uninstalling would be confusing

Agreed. --auto-cleanup does a better job getting at the why the feature is useful as opposed to how the feature is implemented.

Would the option be at a supervisor & service level?

This initial proposal was at the per service level. I definitely see your point that it would be nice to have a single setting that you could set and forget. You could technically have both a global setting that individual packages could override, but that might be adding unneeded complexity.

How do we know what unused dependencies are?

This comment in the code has good details explaining that. Essentially, we make a graph of all packages on disk so we know exactly what the dependency chain is.

[davidMcneil]

Based on the feedback from @ericcalabretta, I think the original proposal may have been two large and scope. My new proposal is as follows:

Add --auto-package-cleanup and --self-auto-package-cleanup as options to the hab sup run command. Both of these options take an unsigned integer indicating the number of latest packages to keep. Any time a package is updated the Supervisor uninstalls packages older than the number of latest specified.

There are two options because it seems likely that users would want the behavior for habitat core packages to be different than their application's packages, but I could also see further simplifying this by having a single flag that applies to all packages.

[mwrock]

I actually like --keep-latest. It feels odd to have the number of versions to keep as an argument to --auto-package-cleanup while --keep-latest makes it clear in my mind that I want the command to uninstall all but what I specify in --keep-latest. I would also think that if no argument is passed to --keep-latest that the default would be 1.

[davidMcneil]

After more discussion, I think the best first implementation is to add a single option --keep-latest to the hab sup run command. After the Supervisor installs a package, it uninstalls all packages later than the number of packages specified with --keep-latest.

[ericcalabretta]

@davidMcneil That makes sense to me I like keep --keep-latest too as a command flag. If I had to pick a single place to configure it'd be at the supervisor level.

Giving people the option to store more even if the default is 1 should solve for most/all use cases.

I can't really see a case where you'd want to have different values for an individual service, could be a case but we can address that later if we need to.

[mrysanek]

A supervisor-level configured count would work for us. We currently have an installed service that cleans up "all but the newest x" of all packages. Ours is set at 3 (current included), and the use-case is "upgrade v1-->v2, patch v2-->v2.1, find a major issue with v2 and have to drop back to v1"

• I would think to avoid additional system resources being dedicated to this, only do it at package load/install and unload/uninstall (as proposed here )
• For the lazy architect, an environment variable defining this would be useful (system configured, so all hab installs can stay the same and cleanup is driven by OS team) with the cli option overriding?

[davidMcneil]

@mrysanek thank you for your feedback. Agreed, I think the ability to set this with an environment variable makes a lot of sense here.

[stevendanna]

First, an assumption I am making, this will all still be subject to what I believe are the constraints of the current uninstall command:

• A package won't be uninstalled if it is the dependency of another installed package (unless that package is currently also being uninstalled)
• A package won't be uninstalled if it is a currently loaded service.

Given those constraints, I think this feature set makes sense. I also like the --keep-latest flag.

If it doesn't substantially add to the implementation complexity, I think it would be useful to be able to define the retention/cleanup settings at both the supervisor and service level. The supervisor configuration become the default if no service-level configuration is applied. However, this could always be done as a follow up if needed.

Will the cleanup that happens at upgrade include dependencies of the removed services?

In Automate's "garbage collector" code, we found that a reasonable trade-off was to have a "conservative" mode that leaves the dependencies installed. The idea here being that the service-level packages change much more rapidly than their dependencies and drive the majority of growth. This would be the equivalent of the --no-deps flag on the current uninstall command.

When a service is unloaded, we could remove all releases of that service on disk.

I agree with not implementing this. If we did, it should be an optional flag to the unload command. Right now users can replicate this behavior by invoking an unload followed by an uninstall. I know both Automate and other hab-based Automation requires fully unloading and re-loading packages which would make any uninstall on unload problematic if it wasn't hidden behind a flag.

One idea that has been floated internally is the notion of a "package registry"

Minor note, at least on the topic of binlinks (one of the things a package registry might solve), I think there is a way to solve the binlink problem by restructuring how binlinks work and that it is possible without a package registry. It involves a layer of indirection such that all binlinks would point towards stubs/binlink-metadata-files in a known location. See https://github.com/habitat-sh/habitat/issues/6812#issuecomment-520793408 for more on the binstubs idea.

Background on Automate's Garbage Collector

Automate has a similar feature, the implementation of which can be found here:

https://github.com/chef/automate/blob/master/components/automate-deployment/pkg/depot/garbage_collect.go

In that implementation, we defined 3 modes:

• Disabled: No cleanup
• Conservative: Clean up old copies of "root" packages (i.e. the loaded services) only
• Aggressive: Clean up any package that isn't a "root" package or a dependency of a "root" package.

Since only 1 copy of the Habitat supervisor can run on a machine and since Chef Automate does not currently allow other services to be loaded into its supervisor, the conservative mode is enough helps ensure that we are unlikely to be uninstalling something that the user is using (via a binlink or script).

[davidMcneil]

@stevendanna thanks for the feedback!

However, this could always be done as a follow up if needed.

Yeah, I think I will leave it as a follow up for now, but I can definitely see its usefulness and it should be straightforward to add should a specific need arise.

Will the cleanup that happens at upgrade include dependencies of the removed services?

Yes, the current working implementation removes dependencies assuming they are not depended on by any other packages and they are not loaded by the Supervisor.

I am fairly comfortable with this more aggressive approach of removing dependencies given those two constraints and the fact we are only uninstalling the package that just updated and not garbage collecting across all packages on the system.