Brew: More responsive outdated & upgrade commands through scheduled tasks and caching

[zbeekman]

A detailed description of the proposed feature

I’m interested in adding the ability for Homebrew users to enable:

1. Background/scheduled updates (makes running brew commands faster if brew update happens intermittently in the background)
2. Background pre-fetching of bottles for outdated formulae
3. Background unattended upgrades for outdated formulae
4. Automated caching of outdated formula on disk. (so that you can have login messages like those present in many linux distros telling you how many/which packages are out of date)

These would all be utilizing launchctl plists, possibly through brew services. I have launchctl plist files that I use on my machine that implement 1, 2, and 4 at https://github.com/zbeekman/launchctl-tasks. Should we decide to add these capabilities, I would like feedback on some implementation details:

1. The service label. Currently I think sh.brew.<name> is a good label, but I don’t know if this is appropriately canonical and if we’ve used a different label naming scheme for services in the past
2. Whether these should be implemented as
   
   
   
   • a) Formula that just configure and install the plist files, that could be added to homebrew-core,
   
   
   • b) as functionality within Homebrew/brew that users would then opt in to enable, or
   
   
   • c) as a separate Tap which can define additional commands
   
   
3. Where the scheduled tasks should write logs & data about what is upgraded, outdated and fetched, and if running brew commands can and should update & cleanup this data. E.g., on my system I print the list of outdated packages upon launching a shell, however, this list is only managed by the the scheduled tasks I have running and brew aliases I have that will update and clean it up when I run e.g, brew upgrade <pkg>
4. How the UI should work. Do users set an environment variable to enable this? Do they run brew services start brew-unattended-upgrades? Do they run a special new command, like brew unattended-upgrades-on?

The motivation for the feature

More responsive and seamless user experience. More information available to users and sys admins.

How the feature would be relevant to at least 90% of Homebrew users

Prefetching outdated bottles will make brew upgrade faster. Background/scheduled brew update will prevent brew update from running when the user invokes an arbitrary command, or at least reduce the frequency of this happening. This in turn will make the command run significantly faster, especially if there is a slow internet connection, etc. Unattended upgrades will allow users who want the latest and greatest version of a package to always have it, without manual effort. Logging and caching outdated packages (and updating this when they are upgraded or uninstalled) will allow the user to know which installed packages are out of date without running brew outdated which can be slow. (This could be done for mas app-store software and casks as well.)

What alternatives to the feature have been considered

1) Using formulae instead of changes to core
2) Using a tap instead of changes to core
3) A ledger of outdated software could be maintained and updated whenever the brew update command is run, and, entries to this ledger could be pruned when brew upgrade or brew uninstall are run. This would not necessarily require using a scheduled task with launchctl.

I would love feedback from @Homebrew/brew and @Homebrew/core maintainers on what they think of this proposal, and what the implementation should look like.

[jonchang]

I'm intrigued by this idea. Is there some sort of capability on macOS to detect metered connections and suspend background updates? Also, would it be possible to just attach certain actions to complete "after" other commands run? I'm thinking of something like, a forked and backgrounded brew fetch that runs after brew update as a starting point.

cc @Homebrew/tsc for their thoughts

[MikeMcQuaid]

1. The service label. Currently I think sh.brew.<name> is a good label, but I don’t know if this is appropriately canonical and if we’ve used a different label naming scheme for services in the past

homebrew.mxcl. is what we use for all formulae. homebrew.mxcl.background. could be used for these. Would be nice to share a prefix.

• b) as functionality within Homebrew/brew that users would then opt in to enable, or
• c) as a separate Tap which can define additional commands

Either of these seem good. I'd consider a tap to start with (perhaps with analytics data enabled) and then could be moved to Homebrew/brew if widely used.

3. Where the scheduled tasks should write logs & data about what is upgraded, outdated and fetched, and if running brew commands can and should update & cleanup this data.

Somewhere under $(brew --prefix)/var/homebrew/.

4. Do they run brew services start brew-unattended-upgrades? Do they run a special new command, like brew unattended-upgrades-on?

One of these sounds good to me.

Background/scheduled brew update will prevent brew update from running when the user invokes an arbitrary command, or at least reduce the frequency of this happening.

I'm unconvinced by this. We'll currently run auto-update by default every 60 seconds if a user hasn't run it before. Presumably we're not planning on running this every 60 seconds (and I'd rather keep the default super low as it avoids issues from people who don't know better).

(This could be done for mas app-store software and casks as well.)

I would strongly advise against scope creep for this. I'd either decide this is going to be for e.g. everything brew bundle supports from the outset or just for formulae/brew update as it'll violate expectations to start with one and move to another.

---

All that said I think this is a cool idea and something that clearly some people will want to use at some level.

[zbeekman]

@jonchang:

Is there some sort of capability on macOS to detect metered connections and suspend background updates?

Not that I know of, but I'm no expert. That seems like more of an ios concern than macOS, no? Also, that would be specific to your ISP. Maybe you could detect if you're tethering to an iPhone? But my plan is to have it run by default when most users are asleep, say between 2:30 and 3:30 AM.

Also, would it be possible to just attach certain actions to complete "after" other commands run?

Interesting idea. This proposal is focused more on the use of services though, so that the user's connection and CPU power spends less time doing homebrew stuff while they're awake.

@MikeMcQuaid:

I'd consider a tap to start with (perhaps with analytics data enabled) and then could be moved to Homebrew/brew if widely used.

Do you have a recommendation for a tap to look at as an example for someone who's relatively new to contributing to homebrew/brew (and not super proficient with Ruby too)? Right now I'm probably going to work from brew/homebrew-bundle or brew/homebrew-alias or similar. How does one enable tap analytics?

I'd either decide this is going to be for e.g. everything brew bundle supports from the outset or just for formulae/brew update as it'll violate expectations to start with one and move to another.

I'll stick to dealing with formulae that brew update and brew upgrade would pull in by default. (i.e., core and user taps)

We'll currently run auto-update by default every 60 seconds if a user hasn't run it before. Presumably we're not planning on running this every 60 seconds (and I'd rather keep the default super low as it avoids issues from people who don't know better).

I may have miss-spoken. I'm not sure if it's due to git (or my particular git settings) homebrew, or something else, but I find, with just two background updates a day, when I run brew update by hand, it typically returns Already up-to-date. It's possible that other brew commands have triggered a background update. I'm not super familiar with the internals, of brew, as I mentioned above. But things seem more responsive. (Maybe it's just confirmation bias.)

[vitorgalvao]

Is there some sort of capability on macOS to detect metered connections

macOS comes with great tools to manage network settings (like networksetup) so it seems feasible.

On the other hand, maybe it isn’t. There’s an app called TripMode to limit app access to the internet when on metered connections, but it doesn’t seem like they auto-detect said connections. Might be worth sending them an email and asking.

It's possible that other brew commands have triggered a background update. I'm not super familiar with the internals

Basically, just about every command runs update unless you tell it not to. Even if you just brew tap to see your current taps, it may run.

[zbeekman]

Basically, just about every command runs update unless you tell it not to. Even if you just brew tap to see your current taps, it may run.

Is there logic within update to see how recently github was queried and return if a git fetch/pull has happened recently? (I'll go code spelunking to answer this....)

[MikeMcQuaid]

Do you have a recommendation for a tap to look at as an example for someone who's relatively new to contributing to homebrew/brew (and not super proficient with Ruby too)?

Sure. Homebrew/bundle, Homebrew/test-bot and Homebrew/services are the best supported external command taps e.g. with style enforcement, CI, etc.

Right now I'm probably going to work from brew/homebrew-bundle or brew/homebrew-alias or similar. How does one enable tap analytics?

Call a Utils::Analytics. method. I'd note in your README you're doing this just so people aren't surprised.

I may have miss-spoken. I'm not sure if it's due to git (or my particular git settings) homebrew, or something else, but I find, with just two background updates a day, when I run brew update by hand, it typically returns Already up-to-date. It's possible that other brew commands have triggered a background update. I'm not super familiar with the internals, of brew, as I mentioned above.

No, that makes sense, I don't think you've misspoken. There's basically three cases:

1. no auto-update is run
2. auto-update is run but no changes have happened
3. auto-update is run and some changes have happened

I thought you were suggesting 2 and 3 would be sped up whereas 2 will remain the same speed and 3 will be faster 👍

Basically, just about every command runs update unless you tell it not to. Even if you just brew tap to see your current taps, it may run.

These are the relevant commands:

https://github.com/Homebrew/brew/blob/master/Library/Homebrew/brew.sh#L417-L418

In the "no argument brew tap case": we could handle that more intelligently and not run brew tap.

Is there logic within update to see how recently github was queried and return if a git fetch/pull has happened recently? (I'll go code spelunking to answer this....)

Yes: https://github.com/Homebrew/brew/blob/master/Library/Homebrew/cmd/update.sh#L450-L454

Note also this API that was added by GitHub (specifically by me 😉) to allow a HTTP call to see if a git fetch is necessary: https://developer.github.com/v3/repos/commits/#get-the-sha-1-of-a-commit-reference

We use this in brew update and it produces a non-trivial speedup.

[zbeekman]

Note also this API that was added by GitHub (specifically by me 😉) to allow a HTTP call to see if a git fetch is necessary: developer.github.com/v3/repos/commits/#get-the-sha-1-of-a-commit-reference

😲🤯

VERY nice

OK, well I'm off to finish working my way through the CodeCademy Ruby lessons, so I can do this in a non-dumb way. (Otherwise it would be almost all bash scripts, although very well-written bash scripts...)

[iMichka]

Just a small comment: please try to implement this so that it works on Linux. Or at least decouple the part of the code that does Mac-specific stuff from the rest. It it easier to work with that code later if this is planned right from the beginning :)

[vitorgalvao]

In the "no argument brew tap case": we could handle that more intelligently and not run brew tap.

Was going to work on a PR to not run update on tap with no arguments but then I though “why run on tap at all”? Why not remove that case entirely? What’s the use case for tap that ever benefits of an update beforehand?

[MikeMcQuaid]

Just a small comment: please try to implement this so that it works on Linux. Or at least decouple the part of the code that does Mac-specific stuff from the rest.

It's going to end up generating and running launchctl plists. Each "command" should be a standalone Ruby or shell script (it can be shell @zbeekman) but actually running them will be system-specific.

[MikeMcQuaid]

Was going to work on a PR to not run update on tap with no arguments but then I though “why run on tap _at all_”? Why not remove that case entirely? What’s the use case for tap that ever benefits of an update beforehand?

We don't allow taps to be tapped that have invalid syntax (i.e. brew readall $TAP fails). Updating before a brew tap means that people do not file bugs saying they are unable to tap a tap with invalid syntax because they are on an older Homebrew.

[scpeters]

In the "no argument brew tap case": we could handle that more intelligently and not run brew tap.

https://github.com/Homebrew/brew/pull/5766

[zbeekman]

Just a small comment: please try to implement this so that it works on Linux. Or at least decouple the part of the code that does Mac-specific stuff from the rest.

It's going to end up generating and running launchctl plists. Each "command" should be a standalone Ruby or shell script (it can be shell @zbeekman) but actually running them will be system-specific.

Yes, in my mind there are two parts to this:

1. The "easy" part. Enabling varying levels of background updating:
   
   
   
   1. Minimal: brew update happens in the background
   
   
   2. Medium (package pre-fetching): brew update happens in the background and brew outdated packages are brew fetched
   
   
   3. Full: unattended upgrades--brew upgrade happens in the background.
   
   
2. To make cases 1.1 and 1.2 convenient for users (i.e., so they can have a list or number of outdated packages printed when they open a terminal or ssh in) it would be nice to maintain a list of outdated formula on disk. This list would ideally be updated by brew update as well as brew upgrade, brew uninstall and others.

I plan to start with just the easy part (1) which is mac specific since it will use plist files with launchctl. A similar approach could probably be hashed out for Linuxbrew using either launchd or crontab but IIRC, there is not 100% adoption of launchd among distros. (This might be wrong or outdated.)

For 1, aka the easy part, it will be important that:

1. Decent logging happens in a sane way, you should be able to look at older logs, but without gumming up the user's drive space
2. The paths to brew are encoded into the plists since they may be run as agents or daemons etc. (i.e. the user may not even be logged in)
3. The time should be randomized (maybe just once, i.e. when first enabled) that the updates & fetch happen, so that not all macs on the east coast of North America are hitting Github and Bintray's servers at the same time.

[MikeMcQuaid]

1. Decent logging happens in a sane way, you should be able to look at older logs, but without gumming up the user's drive space

Advice: rely on https://formulae.brew.sh/formula/logrotate.

[danielbayley]

(This could be done for mas app-store software and casks as well.)

I'd either decide this is going to be for e.g. everything brew bundle supports from the outset or just for formulae/brew update as it'll violate expectations to start with one and move to another.

I'll stick to dealing with formulae that brew update and brew upgrade would pull in by default. (i.e., core and user taps)

@MikeMcQuaid I would definately vote for the former… I think it would be inevitable that this will be wanted for everthing, rather than only brew formulae… and having it only available for those surely complicates matters for users wanting to have casks and stuff installed withmas kept up to date too?

homebrew.mxcl. is what we use for all formulae.

Is there any reason for this other than @mxcl being the creator of Homebrew? Seems like brew.sh. would be more appropriate…

I have launchctl plist files that I use on my machine that implement 1, 2, and 4 at https://github.com/zbeekman/launchctl-tasks.

@zbeekman We seem to share mostly the same goal here, which is to have everything stay up to date automatically in the background, utilising launchctl, while being able to see what has changed with logs (also my motivation behind https://github.com/Homebrew/brew/issues/5744). Although I would extend this to casks and stuff installed with mas as well, as mentioned above.

it would be almost all bash scripts, although very well-written bash scripts…

actually running them will be system-specific.

I personally use zsh scripts for anything mac specific… they are so much cleaner and nicer to write than bash!

[zbeekman]

Is there any reason for this other than @mxcl being the creator of Homebrew? Seems like brew.sh. would be more appropriate…

Typically, you reverse the FQDN, so I think sh.brew.blah would be more accurate.

I personally use zsh scripts for anything mac specific… they are so much cleaner and nicer to write than bash!

I've never ventured over to the dark side. However, the goal is to have fairly minimal contents in the plists, so, most likely, I would embed commands directly in the plist without any intermediate scripts. I've been working on my ruby, so I think I'll just use string interpolation and a template of some sort to accomplish this. In fact, I'd like to use your hash to plist functionality open in the other PR, but I have to figure out a sane implementation.

After thinking about this some more, I think it would most naturally fit into brew-services; when this is tapped it would provide some default services from Homebrew. i.e., brew services start brew-unattended-upgrades. But I'm still prototyping.

[danielbayley]

• a) Formula that just configure and install the plist files, that could be added to homebrew-core

Yeah I've also been wrestling with the best way of implementing this overall… so far my thinking has been along the same lines as this… service only formulas that don't actually download anything (maybe url stage_only or something). I also actually made a simple command line utitlity to easily write these files (thinking I could depends_on "max-os/launch", and use it in post_install at one point…

The use case of being able to have launchctl automatically keep everything up to date is only one of many though for me… I also want to have homebrew install a bunch of other scripts and plists that do other things, using my own taps… sometimes running an actual script file, but in other cases can just be a couple of commands written directly into the plist file, as you mention.

But my update script for example also handles stuff unrelated to Homebrew, like global npm packages, and RubyGems… so is a seperate script. I think the more we can seperate concerns here and keep the functionality more general, the more useful it will be for users.

[MikeMcQuaid]

Although I would extend this to casks and stuff installed with mas as well, as mentioned above.

If that ends up being the case then my suggestion be it all be wrappers around brew bundle and most of the logic moved there. The codebase there is pretty easy to follow and well tested.

Is there any reason for this other than @mxcl being the creator of Homebrew? Seems like brew.sh. would be more appropriate.

That's the legacy reason, yes, but consistency is valuable for its own right.

But my update script for example also handles stuff unrelated to Homebrew, like global npm packages, and RubyGems… so is a seperate script. I think the more we can seperate concerns here and keep the functionality more general, the more useful it will be for users.

Global npm and rubygems installation could also be added into brew bundle, perhaps.

[szb512]

Well, global npm and rubygems installation would be benefital for brew bundle, but there would be a higher chance that npm would fail, and we would have to write more files.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[zbeekman]

please go away stalebot, I'm still working on this...

[MikeMcQuaid]

@zbeekman You can set an "in progress" label if you're working on it. I'd suggest this can perhaps be closed out if it's something you're working on yourself rather than writing up for others to implement, though.

[vitorgalvao]

You can set an "in progress" label if you're working on it.

Another way is to delete the bot’s message. Doing so auto-removes the label, the issue stays cleaner, and the countdown resets.

[claui]

@vitorgalvao Nice trick, didn’t know that!

[zbeekman]

@MikeMcQuaid and @vitorgalvao thanks for the advice! I'd be happy to delete the message or remove the label, but I'm not currently a maintainer on Homebrew/brew, so I can't do that.

FWIW, I've been practicing my ruby, and am no longer terrified of pushing a catastrophic commit, so I'd happily take on that responsibility should you choose to entrust me with it... but I'm happy either way.

[zbeekman]

I'd suggest this can perhaps be closed out if it's something you're working on yourself rather than writing up for others to implement, though.

Fair. I think we hit most of the discussion points. I've put this on the back burner as I've been watching the plist work from @danielbayley progress; I figure it makes sense to let that stabilize a bit and then use it for this. We can close it if you like, but it's a nice reminder to me.

[MikeMcQuaid]

We can close it if you like, but it's a nice reminder to me.

Yeh, I think so, thanks. The issue tracker is best placed for tracking bugs or, in the case of features, those we're actively soliciting the community work on.