Brew: `brew cleanup` should be run automatically

[MikeMcQuaid]

One of the most common things we hear on our Twitter account is people saying that they are pleasantly surprised that brew cleanup cleans up so much space. We've held off running it automatically in the past as it's not granular enough to be effectively scoped to a package and because we're concerned about linkage to old versions.

We now have a cached linkage database and the HOMEBREW_UPGRADE_CLEANUP variable which cleans up a bunch more things related to a given formula which means we can start to consider having HOMEBREW_UPGRADE_CLEANUP run on every upgrade/install by default after ensuring that linkage won't be broken.

[chdiza]

Will that shellvar still exist and be able to be turned off?

[MikeMcQuaid]

@chdiza If someone can make a pitch as to why it should be toggleable off: I don't see a reason why not.

[chdiza]

Because it is a disaster for anyone who wants to keep previous versions of things around. And keeping previous versions around (at least for really important stuff) is good practice, because breakage is common (here I mainly mean breakage in upstream projects, not HB breakage---though that happens sometimes too). There's a reason why brew switch exists.

To lose this fundmental ability---part of managing packages---because doing normal maintenance doesn't occur to some people would be a real shame.

Come to think of it, if such a non-toggleable change were to occur---and were this to mean that I can't carry e.g., wget 1.19.4 and 1.19.5 for a while until I'm sure that 1.19.5 fixed a problem with 1.19.4, or for something more mission-critical carry rsync 3.1.2 for a while until I'm sure that 3.1.3 won't ruin my life---I think that might mean the end of Homebrew for me.

It's also bad for anyone who tries to keep their HOMEBREW_CACHE in sync across multiple machines, but this is a much smaller concern.

There are ways to keep the aforementioned ability, like creating something akin to pin but that simply means "please don't automatically throw this away", but I'm sure they would involve much more code than adding the toggle would.

[MikeMcQuaid]

And keeping previous versions around (at least for really important stuff) is good practice

I think it is for you as you file good issues. I don't think it is for the majority of Homebrew users, particularly those who don't even know brew cleanup exists.

To lose this fundmental ability---part of managing packages

Note: the vast majority of package managers don't keep old versions around indefinitely.

but I'm sure they would involve much more code than adding the toggle would.

Agreed, you've convinced me there should be an opt-out toggle for this.

[chdiza]

Agreed, you've convinced me there should be an opt-out toggle for this.

Wonderful! :)

Note: the vast majority of package managers don't keep old versions around indefinitely.

I would be surprised if they immediately force removal of old ones with no way to override this. If they do, they're doing it wrong :)

Side-note, now that the toggle will stay: Whether keeping things around (at least for a while) is good practice or not isn't a function of the savviness of the user, but rather of the likelihood that some upstream will break something in a newer version. If e.g., wget 1.19 breaks something, then no matter how good or bad an issue-writer the user is, she will wish she still had the 1.18 keg to brew switch to.

I've been Brewing At Home since late 2012, and I think this has happened to me at least 3 times per year. If I were some sort of Homebrew Instructor, I would tell my students to never run brew cleanup without using the -n flag first, lest they land the aforementioned pickle. If there is something in that output that they're OK with trashing, but also something they don't want to trash yet, they can manually pick out the trashable ones.

[MikeMcQuaid]

I would be surprised if they immediately force removal of old ones with no way to override this. If they do, they're doing it wrong :)

They do. apt-get dist-upgrade on Debian replaces an old version with a new one.

If e.g., wget 1.19 breaks something, then no matter how good or bad an issue-writer the user is, she will wish she still had the 1.18 keg to brew switch to.

Again, this doesn't apply to you, but I'd argue getting that version update right is the responsibility of the maintainers rather than the user.

[cig0]

In addition to:

@chdiza If someone can make a pitch as to why it should be toggleable off: I don't see a reason why not.

I just hit this issue earlier today: #4942

[MikeMcQuaid]

@cig0 That's unrelated to this issue.

[maxnordlund]

I've been running brew update/upgrade/cleanup in a script for years in order to conserve space, and it works like a charm. This would be a nice change 👍

The only time I had to reinstall an old version is when Postgres updates in order to run its database upgrade tool. Now days we have brew postgresql-upgrade-database, which I really appreciate. Having cleanup run automatically _shouldn't_ break that. So perhaps the current, and immediate last version should be kept?

That way the user can switch/pin fairly easily if something goes wrong. Another option would be to cleanup that last version on a second run, to allow for tools like postgresql-update-database, but it might be too confusing behavior.

[MikeMcQuaid]

The only time I had to reinstall an old version is when Postgres updates in order to run its database upgrade tool. Now days we have brew postgresql-upgrade-database, which I really appreciate. Having cleanup run automatically _shouldn't_ break that.

That script will use the versioned postgresql formulae so should be fine 👍. We may need to make a note to always have the previous version of postgresql around when we do major upgrades for that case alone.

[Wojonatior]

I'd love to help with this! I just wanted to clarify I know what's wanted out of this first. Is the intent of this to run a brew cleanup on install and update unless HOMEBREW_UPGRADE_CLEANUP is set to false?

[sjackman]

Could it be run after brew install, so that the result of brew install happens quickly without holding up the user? So the order of operations is: user runs brew install hello, Homebrew runs brew update automatically, then brew install hello, then brew cleanup.

[Wojonatior]

Yeah, I follow that. I'll try to get to it this week, and I'll have a status update by the weekend.

[MikeMcQuaid]

@GauthamGoli was also planning on working on this. @GauthamGoli can you let us know what you've done already (if anything)? @Wojonatior I still encourage you to work on this.

I'd suggest that we run the same cleanup that's currently on HOMEBREW_UPGRADE_CLEANUP for install, upgrade and reinstall if HOMEBREW_INSTALL_CLEANUP it set. This could be extended to do a fuller cleanup with perhaps another variable being set after a certain number of brew installs.

[Wojonatior]

Hey all, I haven't made any progress on this yet. I have some time set aside this week. But if someone else starts on this, just let me know.

[GauthamGoli]

@Wojonatior Feel free to open a PR. I haven't started working on this YET.

[MikeMcQuaid]

Folks: in general we don't "assign" issues to anyone until there is an open PR so you don't need to keep us up to date here; just open a PR if you can or don't 😁

[Wojonatior]

Hey all, I spent some time working on this over the weekend. I want to get a test in place for the current upgrade functionality. I'm unable to get a working test with the environment variables for that. Any guidance?

[MikeMcQuaid]

@Wojonatior Open a PR as-is; it'll be easier to discuss code than something more abstract.

[Wojonatior]

@MikeMcQuaid Is the ENV intended to be setup as an opt-out as originally discussed, or as an opt-in, as the previous ENV was setup?

[MikeMcQuaid]

@Wojonatior Opt-in for now.

[MikeMcQuaid]

I've implemented the last piece of this in https://github.com/Homebrew/brew/pull/5472. HOMEBREW_INSTALL_CLEANUP will opt you in to this behaviour when it is merged and HOMEBREW_NO_INSTALL_CLEANUP will opt you out forever (even after it becomes default/opt-out).

Reanswering previous questions:

Will that shellvar still exist and be able to be turned off?

Yes using HOMEBREW_NO_INSTALL_CLEANUP.

Now days we have brew postgresql-upgrade-database, which I really appreciate. Having cleanup run automatically _shouldn't_ break that.

This script now only uses versioned formulae (whose presence is enforced with an audit) and now old versions (as there were linkage issues) so will not be broken by this behaviour.

Could it be run after brew install, so that the result of brew install happens quickly without holding up the user? So the order of operations is: user runs brew install hello, Homebrew runs brew update automatically, then brew install hello, then brew cleanup.

This is the implemented behaviour.

[knowtheory]

Hi there,

I seem to have been opted into this automatically and without any notification.

Consequently i'm going to have to spend some time reinstalling a number of formulae which i need to keep and switch between. This has ruined my evening and possibly some deadlines, so i would like to strongly advocate that the behavior is changed to at least prompt users for confirmation the first time this behavior happens.

Also, as ever, thank you for Homebrew, and <3 to the maintainers.

[MikeMcQuaid]

so i would like to strongly advocate that the behavior is changed to at least prompt users for confirmation the first time this behavior happens.

Sorry, no. Homebrew does not prompt the user. Changing this now would break scripts that rely on this assumption.

This behaviour was announced on our Twitter, our 1.9.0 release notes and man brew. man brew provides details for opting out of brew update going to non-stable tags and this cleanup behaviour.

Also, as ever, thank you for Homebrew, and <3 to the maintainers.

Thanks for the kind words ❤️

[MikeMcQuaid]

Closing this issue as it has now been implemented for those on the master branch. Set HOMEBREW_NO_INSTALL_CLEANUP to opt-out of this behaviour.

[abraithwaite]

Ouch. Guess I should read the release notes. I frequently go back and forth between versions of software in a rolling release model.

Glad there's a flag at least! Would strongly urge a prompt for future changes of this nature though. Even if it does break scripts, I believe it's better than having your entire version cache wiped out.

Perhaps if it were easier to re-install an old version of a formula once it's been removed, then it wouldn't be such an issue. Unless I'm mistaken you still have to do the git song and dance to install older versions of software (unless that version also has a tap). I'd be delighted to learn otherwise. I did not see anything in the man page.

[knowtheory]

Sorry, no. Homebrew does not prompt the user. Changing this now would break scripts that rely on this assumption.

This behaviour was announced on our Twitter, our 1.9.0 release notes and man brew. man brew provides details for opting out of brew update going to non-stable tags and this cleanup behaviour.

It's unfortunate that the cat's out of the bag, but this is something worth keeping in mind for future. From my perspective as a user, i updated homebrew (which is what i'm supposed to do), installed something, and suddenly the tool's behavior has changed in a way that i didn't expect, runs automatically, and the only way i would have known what's happening is if i followed Homebrew on twitter, or if i happened to be perusing the man pages.

There's no opportunity in there for me to have known to configure my environment differently.

This is obviously all open source software, and maintained by lovely people who enable my life to be much smoother than it'd otherwise be, so, like, i guess i'll just keep a more careful eye out in future. But this feels like a shift from my POV. Before I was making sure to continue to brew update because i needed to keep my formulae up to date (and it happened to be that tooling updates came along with that). This issue has made it clear that before i do anything with a brew, i need to first make sure that i know what's going on in homebrew's release notes, and proceed with the knowledge that any possible brew command could pull down new and different tooling behavior. 🤷‍♂️

❤️

[MikeMcQuaid]

There's no opportunity in there for me to have known to configure my environment differently.

Well, there was but without you reading the release notes, Twitter or GitHub: no, there was not.

proceed with the knowledge that any possible brew command could pull down new and different tooling behavior

Not any brew command but: yes, brew install, brew upgrade also previously would always run an auto-update and they now also run a cleanup. You are safe in the assumption that brew --prefix will not do that.

[MikeMcQuaid]

This is obviously all open source software, and maintained by lovely people who enable my life to be much smoother than it'd otherwise be

Thanks for the kind words. They genuinely mean a lot ❤️

[sjackman]

@abraithwaite wrote…

Perhaps if it were easier to re-install an old version of a formula once it's been removed, then it wouldn't be such an issue. Unless I'm mistaken you still have to do the git song and dance to install older versions of software (unless that version also has a tap). I'd be delighted to learn otherwise. I did not see anything in the man page.

You can install older bottles like so:
Find the bottle on Bintray: https://bintray.com/homebrew/bottles/coreutils
Install the bottle:
brew install https://homebrew.bintray.com/bottles/coreutils-8.29.high_sierra.bottle.tar.gz

[abraithwaite]

@sj26 Neat, thank you!