Brew: Handle Versions Better

[MikeMcQuaid]

Extracted from https://github.com/Homebrew/brew/issues/60.

• [x] Add code to note in the INSTALL_RECEIPT what the name was of the alias used to install (if any) and check that alias in future for upgrades (so e.g. boost.rb can be an alias to [email protected] and upgrades to [email protected] when the alias points to that).
• [x] Do not accept new versions in homebrew/versions for any versions above those in core and move old versions back into core.
• [x] Document all this for formula/tap authors (https://github.com/Homebrew/brew/pull/1204)
• [x] Give unversioned formulae that have versioned formulae aliases e.g. [email protected]
• [x] Upgrade immediately to new formulae versions but provide old versions when needed to avoid blocking large upgrades
• [x] At some point in the future if we're on e.g. [email protected] or something (having had [email protected] etc.) and [email protected] no longer compiles we remove any formulae from core that still depend on [email protected] and remove the [email protected] formula itself.

This will allow us to immediately merge e.g. boost upgrades and handle the upgrade of dependent formulae on a formula-by-formula basis rather than being blocked on a single formula being upgraded or removed.

[DomT4]

I keep trying to get the ball rolling here without much success. Tried to talk people into doing an llvm38 core formula before giving up & waiting for the core llvm formula to be ready to update. Probably my fault for not drawing up a proper timetable outside of my head, but there we go.

Do not accept new versions in homebrew/versions for any versions above those in core and consider moving old versions back into core.

Would like to get the problem with brew install/info homebrew/versions/ducks150 resolved on moving formulae before we start reverse migration, but yeah, some stuff will have to come out of Versions in the medium term.

At some point in the future if we're on e.g. boost30 or something (having had boost21 etc.) we remove any formulae from core that still depend on boost15 and remove the boost15 formula itself.

For the sake of keeping the core actually maintainable I'd like us to follow the same rules we've applied to new Versions submissions, i.e. +/-2 from current release, formulae should be pegged to a specific release branch & not for example autoconf213.

[MikeMcQuaid]

For the sake of keeping the core actually maintainable I'd like us to follow the same rules we've applied to new Versions submissions, i.e. +/-2 from current release, formulae should be pegged to a specific release branch & not for example autoconf213.

Agreed. Should add another point once we decide about naming. I'm increasing tempted by @UniqMartin's suggestion of e.g. autoconf+2.1 format.

[DomT4]

One big question we'll need to resolve is how to balance formulae against each other. It doesn't make sense for say boost+160 to be keg_only if boost is currently installed & boost points to boost+159 or such, but if both are installed there will be a conflict and we need to work out how best to handle that without trashing UX.

@UniqMartin's suggestion of boost+160 or similar makes enough sense. I don't have a strong preference between boost160 and boost+160. We've used the former but I don't think anyone is particularly attached to it & perhaps changing the format will more strongly indicate change on how we handle Versions.

[MikeMcQuaid]

One big question we'll need to resolve is how to balance formulae against each other. It doesn't make sense for say boost+160 to be keg_only if boost is currently installed & boost points to boost+159 or such, but if both are installed there will be a conflict and we need to work out how best to handle that without trashing UX.

I don't think that's a blocker on this but we could infer keg_only status based on the alias.

@UniqMartin's suggestion of boost+160 or similar makes enough sense. I don't have a strong preference between boost160 and boost+160.

I have a strong preference for something with a . in it; it's not obvious from a glance whether boost160 is boost1.6.0 or boost16.0 or boost1.60 or what 😆

[DomT4]

I don't think that's a blocker on this but we could infer keg_only status based on the alias.

That seems reasonable. I'm hesitant to introduce a bunch more keg_only formulae given they're not massively user friendly but I can't think of an obviously better workaround. Maybe an extension to the conflicts_with mechanism to account for run rather than build or something.

conflicts_with "boost" => :run, :because => "Boost alias currently points to boost+159"

[MikeMcQuaid]

@DomT4 Yeh but I'm thinking that it should be inferred rather than specified in the formula.

[DomT4]

Aye. It shouldn't be difficult to check the current alias & assume anything matching a regex of that name can't be linked on top of that.

[MikeMcQuaid]

Another reason for using boost+1.2.3 or something similar; it allows us to use boost+ in the regex rather than the problematic boost.

[sjackman]

Debian separates the name and version with a hyphen, e.g. libstdc++-4.8 and libusb-1.0

[RandomDSdevel]

I'm just a lurking Homebrew beginner watching this issue, but I just came up with the thought that another option would be to separate the formula name and version with '-v' or '_v' — i. e.: 'boost-v1.61.0' or 'boost_v1.61.0' for Boost v1.61.0 — since it might be a good idea to explicitly specify that the number suffixed to a formula name is a version number using the letter 'v' as an abbreviation.

[chdiza]

Please don't stick punctuation in the formula names. That makes things really annoying to type.

[CamJN]

Maybe if it was a ❣ but ascii punctuation should be fine.

[MikeMcQuaid]

We're going to have to stick some sort of character we don't currently use in formula names to delimit them. You shouldn't need to type the non-aliased version, though.

[tdsmith]

Should this be in the -evolution process?

[MikeMcQuaid]

Maybe. It predates it and we've/(I've) got a fairly clear bunch of stuff to implement so I don't think so. We could perhaps reword it but the way I see it the evolution process is there to submit detailed designs for others to implement and a tracking issue like this is the way for maintainers to signal to each other "this is work I currently have in progress".

[sjackman]

We're going to have to stick some sort of character we don't currently use in formula names to delimit them. You shouldn't need to type the non-aliased version, though.

There's a few unfortunately named packages that include + in the name:

bonnie++.rb            kimwitu++.rb           mysql-connector-c++.rb
crf++.rb               libsigc++.rb           pcre++.rb
dvd+rw-tools.rb        libxml++.rb            png++.rb
gtk+.rb                libxml++3.rb           vera++.rb
gtk+3.rb               mysql++.rb

My preference is - for its similarity to Debian. (Definitely bike shedding here though.) Both + and - are used in formula names, though granted there's only 14 that use + and 587 that use -.

[CamJN]

No formula names use '@' or 🐢 for that matter.

[MikeMcQuaid]

@CamJN @ is the forerunner in my mind.

[RandomDSdevel]

@MikeMcQuaid: Forerunner to _what?_

[tdsmith]

I think Mike meant "frontrunner," i.e. his favorite of the proposed options?

[RandomDSdevel]

@tdsmith: Ah, _that_ makes sense!

[MikeMcQuaid]

I think Mike meant "frontrunner," i.e. his favorite of the proposed options?

Yes 😀

[UniqMartin]

Even though I'm still fond of + as a delimiter, that's going to create some ambiguity with existing formula names and will complicate any automatic detection of related formulae.

I'm very much in favor of using @ as it reads very nicely (<formula>@<series>, e.g. boost at the 1.59 release series), is an unproblematic character in file names (it's easy to type, too), and not something we're currently using or accepting in our formula names.

On a related note, I think we should be stripping @<series> from the formula name before constructing a Ruby class name, e.g. [email protected] would still contain a Boost class derived from Formula, just like any other boost@<series>.rb formula file.

One major issue I'm not entirely sure how to handle is how we're going to upgrade stuff that depends on a versioned formula to use the next major release, e.g. to stick with the Boost example, do we use [email protected] in every depends_on "boost" line and then bump this in every using formula, assuming it is already compatible with Boost 1.62.x, to [email protected] after the [email protected] formula has been merged?

[MikeMcQuaid]

I'm very much in favor of using @ as it reads very nicely (@, e.g. boost at the 1.59 release series), is an unproblematic character in file names (it's easy to type, too), and not something we're currently using or accepting in our formula names.

Yes, sounds like that's the best option.

On a related note, I think we should be stripping @ from the formula name before constructing a Ruby class name, e.g. [email protected] would still contain a Boost class derived from Formula, just like any other boost@.rb formula file.

I don't agree, this will add code complexity and deviation from the current patterns for no real benefit.

to stick with the Boost example, do we use [email protected] in every depends_on "boost" line and then bump this in every using formula, assuming it is already compatible with Boost 1.62.x, to [email protected] after the [email protected] formula has been merged?

Basically, we do the current revisioning process after the merge which means we can merge new versions quickly. The two ways of doing that are to either:

1. use boost@ in depends_on as you mentioned
2. only change the boost alias to point to [email protected] when everything has been revisioned/updated to support the new version and then move the stuff that's blocking the upgrade to be depending on [email protected] instead.

I don't think we need to mandate which way to go as I think 2. will be better for core and 1. for e.g. science.

[UniqMartin]

On a related note, I think we should be stripping @<series> from the formula name before constructing a Ruby class name, e.g. [email protected] would still contain a Boost class derived from Formula, just like any other boost@<series>.rb formula file.

I don't agree, this will add code complexity and deviation from the current patterns for no real benefit.

It's a new pattern that hasn't been used before, thus we should be free to adjust the rules as necessary. And there's no real benefit in encoding the release series in the class name. It just unnecessarily duplicates this information, forces us to invent a new rule for translating @ to a valid formula class name component, and overall increases the number of places that need to be changed when creating, e.g., [email protected] from [email protected].

In contrast, stripping off @<series> is a simple change in Formulary.class_s that implements this mapping for all of Homebrew in a single location (unless I'm overlooking something).

Basically, we do the current revisioning process after the merge which means we can merge new versions quickly. The two ways of doing that are to either:

1. use boost@ in depends_on as you mentioned
2. only change the boost alias to point to [email protected] when everything has been revisioned/updated to support the new version and then move the stuff that's blocking the upgrade to be depending on [email protected] instead.

I don't think we need to mandate which way to go as I think 2. will be better for core and 1. for e.g. science.

Sounds good!

[MikeMcQuaid]

It's a new pattern that hasn't been used before, thus we should be free to adjust the rules as necessary.

This pattern has been used in homebrew/versions and other taps for a long time.

And there's no real benefit in encoding the release series in the class name.

• It stops having multiple classes with the same name in the same tap which, even if it doesn't break current code assumptions or introduce strange bugs, makes it harder to search for a single classname
• It's what we've done with no complaints in homebrew/versions and taps for a long time.

It just unnecessarily duplicates this information, forces us to invent a new rule for translating @ to a valid formula class name component, and overall increases the number of places that need to be changed when creating, e.g., [email protected] from [email protected].

In contrast, stripping off @ is a simple change in Formulary.class_s that implements this mapping for all of Homebrew in a single location (unless I'm overlooking something).

Both versions require code changes so I don't think that's a strong argument either way.

[UniqMartin]

It's a new pattern that hasn't been used before, thus we should be free to adjust the rules as necessary.

This pattern has been used in homebrew/versions and other taps for a long time.

I disagree: "fooXY" != "[email protected]".

[MikeMcQuaid]

I disagree: "fooXY" != "[email protected]".

I was referring to having the class name and filename both containing the version.

[DomT4]

Ref: https://github.com/Homebrew/brew/pull/812

[xu-cheng]

On a related note, I think we should be stripping @ from the formula name before constructing a Ruby class name, e.g. [email protected] would still contain a Boost class derived from Formula, just like any other [email protected] formula file.

I don't agree, this will add code complexity and deviation from the current patterns for no real benefit.

There is actual one important benefit. It will make the formula [email protected] and [email protected] share the same prefix. Which in turns can solve many potential problems:

• It handles conflicts among different version automatically without the need of manually declaring conflicts_with(which it's not scalable)
• It supports brew switch. And allow users to manage which version is active one(i.e. exposed to the PATH).
• In long term, we can even replace --devel, --HEAD with aliases like foo@devel and foo@HEAD. So I can run brew install foo@devel bar@HEAD baz (which it's impossible for now).
• For brew bundle, it means that we can finally support version check. For example, brew 'foo', version: "~>1 should be considered to be satisfied if either of [email protected] and [email protected] is installed.

The main point is to let homebrew understand foo.rb and [email protected] as well their installation kegs are indeed the same formula with different version. And handle conflict in the process using existing codes.

However, this does require us to store @1.0 suffix in the tab file.

This is just my thought to be shared.

[MikeMcQuaid]

@xu-cheng How would you depend on a given major/minor but not patch version? The patch version will be stored in the Cellar PATH but only the latest/switchd will be linked to by opt. The implementation complexity will be much higher even if both those problems are solved. I'd prefer to use the version that's already been working for us in homebrew-versions.

[tdsmith]

@xu-cheng: that's a compelling list of features but do we always need the class name to be "foo" to achieve that?

[MikeMcQuaid]

@penman has done great work pushing this forward 👏. The code for Homebrew/brew should all be there for now, it's just a matter of adding the aliases and trying this approach out.

[HaraldNordgren]

@UniqMartin:
On a related note, I think we should be stripping @ from the formula name before constructing a Ruby class name, e.g. [email protected] would still contain a Boost class derived from Formula, just like any other boost@.rb formula file.

I agree with this and what @xu-cheng said.

Although there is a technical problem since brew doesn't allow file and class names to diverge. One solution is to create an Alias structure with symlinks, like this:

 % tree homebrew-versions
homebrew-versions
|-- Aliases
|   `-- [email protected] -> ../Formula/[email protected]/media-info.rb
|-- Formula
|   `-- [email protected]
|       `-- media-info.rb
|-- ...

Inside Formula/[email protected]/media-info.rb we then have class MediaInfo < Formula. This allows files to be installed in the same Cellar as the core version, and for versions to be changed through brew switch.

The directory structure prevents brew from directly accessing the .rb file so there won't be any mix-ups:

 % brew install homebrew/versions/media-info
Error: No available formula with the name "homebrew/versions/media-info"
==> Searching for similarly named formulae...
This similarly named formula was found:
homebrew/versions/[email protected] ✔

The installation is done through brew install [email protected]. The regular one is still installed through brew install media-info. And then versions are handled like this:

 % brew switch media-info 0.7.70
Cleaning /usr/local/Cellar/media-info/0.7.70
Cleaning /usr/local/Cellar/media-info/0.7.90
10 links created for /usr/local/Cellar/media-info/0.7.70

[ahundt]

@scpeters described cross-tap breakage in https://github.com/Homebrew/brew/issues/1416#issuecomment-257956759, reproduced here:

My guess is that it's related to formulae that have dependencies that live in other taps. If the dependency gets rebuilt, the downstream taps will need to rebuild their bottles. For example, formula revisions for boost users in core were bumped in Homebrew/homebrew-core#5364, but boost users in homebrew/science were probably broken right when that was merged.

Cross-tap bottle breakages seems like a good topic for a brew-evolution pull request to me. I'll be happy to participate there, since this affects a tap that I maintain. I don't have any immediate ideas though.

Essentially, if one formula is updated all of its dependencies need to be updated, even across taps. I encountered this problem via the following steps:

1. eigen3.rb in homebrew-core was updated
2. I updated homebrew, which updated my installed version of eigen3
3. ceres-solver.rb in homebrew-science remained unchanged
4. I ran a cmake script in one of my libraries that calls find_package(Ceres)
5. CeresConfig.cmake checks Eigen header versions as detailed in this link to ensure the headers supplied with Eigen match the version used when it was compiled to prevent silent failures by violating the One Definition Rule.
6. I ran brew reinstall ceres-solver
7. No change in steps 2-4 because it was reinstalled from the bottle, which remained unchanged
8. I successfully ran brew reinstall ceres-solver --build-from-source, since ceres-solver built directly with the correct eigen version.

This occurred at least twice to my knowledge, here are the issues and prs:

• https://github.com/Homebrew/homebrew-science/issues/3454
• https://github.com/Homebrew/homebrew-science/pull/4524
• https://github.com/Homebrew/homebrew-science/issues/4543

I'm sure this has happened with other packages too, but I don't have references. Perhaps there is some approach to detect and avoid this problem?

Essentially, bottles can become stale even if all the packages involved are correct and compile from source successfully. As I understand it the creation of bottles is triggered based on pull requests, so updating the bottles of one repository based on a CI build of bottles in another may be challenging...

[MikeMcQuaid]

@HaraldNordgren Just a note that that alternate approach is valid but we're too far along the road with the current one and as it's pretty much just me doing this work now I'm not going to go back yet.

Essentially, bottles can become stale even if all the packages involved are correct and compile from source successfully. As I understand it the creation of bottles is triggered based on pull requests, so updating the bottles of one repository based on a CI build of bottles in another may be challenging...

Yes. This is a (perhaps "the") known problem and I won't consider this issue closed until it's resolved.

[ahundt]

Yes. This is a (perhaps "the") known problem and I won't consider this issue closed until it's resolved.

@MikeMcQuaid Great, thanks! Hopefully I wasn't belaboring the point. I figured a step by step would be useful while it was fresh in my mind. Plus, deps built with a different version is slightly different failure case from deps that don't support a certain new version.

[MikeMcQuaid]

The first test package (Elasticsearch 1.7) has been migrated from homebrew/versions to homebrew/core: https://github.com/Homebrew/homebrew-core/pull/6533. After a few days and no issues (and some personal dogfooding I've been doing for a while) I'll move on to migrating the rest of the Elasticsearch and MySQL formulae: https://github.com/Homebrew/homebrew-core/pull/7181

After that the goal will be to start migrating everything that's being actively used and makes sense in homebrew/versions into homebrew/core with the goal of shutting down homebrew/versions. Relatedly, I plan on also shutting down some other homebrew/* taps that are maintained exclusively by current Homebrew maintainers and merging them into homebrew/core too.

After this is done I'll publish guidelines on which old versions are acceptable for homebrew/core and how we're going to tackle upgrades for new versions e.g. [email protected] (preview: much the same as we do currently but with some symlinks to ease migration).

[HaraldNordgren]

I went ahead and created a tap according to my idea above:

1) A Heroku process polls homebrew-versions for new commits
1) Runs a script to separate versions from classnames
1) Converts every formula to 'class@version' form
1) Replaces references in other formulae
1) Automatically pushes to the new tap

So hombrew-versions that now work effortlessly with brew switch + All the older versions of packages if you need those since I designed the bot never to remove .rb files.

Link: HaraldNordgren/homebrew-versions

[MikeMcQuaid]

@HaraldNordgren Please add a license and copyright notice as required by the Homebrew Versions license: https://github.com/Homebrew/homebrew-versions/blob/master/LICENSE

[HaraldNordgren]

Done.

[MikeMcQuaid]

@HaraldNordgren Thanks!

[MikeMcQuaid]

Just a note that I've been migrating various Homebrew/versions formulae into Homebrew/core and fixing the issues that crop up when doing so. We've also got checks now to ensure recursive dependencies don't have mismatched versions. The strategy that we'll personally take in Homebrew/core to get things up to date will likely change a bit but I still plan on trying to make the migration path easier for 3rd-party taps to avoid breakages.

You can check the documentation changes in https://github.com/Homebrew/brew/pull/1767 to see where we're at with this today.

[pinepain]

It was brought up few time before, but what about creating set of [email protected] formulae? I already have to use my own (https://github.com/pinepain/homebrew-devtools/blob/master/Formula/v8%405.7.rb) to move forward and in recent PRs (#9343 and #7168) it came up that it's not only me who interested having more recent v8 version that default v8 provides (5.1).

I guess I can help maintain v8 formulae on daily basis and keep them up to date as I track v8 public api changes on daily basis for my php-v8 extension and also package v8 for Ubuntu (see libv8-* PPAs at https://launchpad.net/~pinepain).

At this time I guess we would be far more than OK to have [email protected] and [email protected] as building all from 5.1 will abuse build system as v8 is a tough piece of code. We might consider to maintain -2 minor release to stable, current stable and upcoming major releases. For upcoming releases we will be OK to upgrade to patch release which breaks BC, while for current stable and below - no. This small trade-off should keep PITA level to work with v8 low and move forward. For non-stable releases we would update no more than once per week, as patch releases pop up almost every few hours. When needed and upon request we might update in-between, but less likely such changes appears which can't wait till the week cycle release date.

As to d8 tool which provides basic v8 shell, I propose to expose it (if we decide to do) as a symlink or proxy shell script, as d8 requires icudtl.dat alongside it, see my comment for details. Naming for exposing will be v8-x.y, where x and y are the same as formula version number. Using system ICU at this time is not what v8 allows, there was few discussion on v8 mailing lists (and maybe even in nodejs github issues).

[MikeMcQuaid]

@pinepain If you're using [email protected] specifically then we'd review (and likely accept) a PR to add just that.

[pinepain]

@MikeMcQuaid I'd rather go with [email protected] as at this time it public API BC-compatible with 5.7 (at this time), and 5.7 will likely be a stable soon, which prevent it from getting new ES features. Will submit a PR. Thank you!

[MikeMcQuaid]

Cross-posting https://github.com/Homebrew/brew/issues/1937 here as people here are likely interested in the discussion on how the new versions system should be used.

[MikeMcQuaid]

Now that Homebrew/versions is migrated into Homebrew/core and we have various audit checks, aliases and migrations in place I'm considering this done. Thanks for the input everyone.