Brew: Simplify bottling for taps

[davidchall]

Without a BrewTestBot (or similar), it's labor intensive for tap maintainers to build bottles, publish them on Bintray, and update the formula with the bottle DSL. This could potentially be achieved on Travis CI using the brew test-bot and brew pull --bottle commands.

I've hacked together a partial solution by making changes only to Homebrew/brew-test-bot, but I'm looking for feedback from the experts to find a better solution (e.g. @MikeMcQuaid @sjackman).

I think there are two main points to discuss:

• best workflow without a TestBot user
• code refactoring in Homebrew/brew

Once we arrive at a solution, it can be added to brew tap-new and documented.

Suggested workflow

• Encrypted env vars: cannot upload bottles from forked PRs
• I suggest uploading bottles from master branch builds (i.e. test builds), since this is the point at which they are "accepted" into the tap
• In the absence of a TestBot user, I suggest using the owner's account as the TestBot user (creating testing-#{build_id} tags and branches)
• Exploit the new Travis CI stages, so that bottles are built and uploaded for each OS version in the first stage, and if all jobs complete successfully then a second stage runs brew pull --bottle

Code refactoring

Here are some variables that need to be edited to upload bottles and push formula changes:

• bintray-org (brew test-bot and brew pull --bottle)
• HOMEBREW_BINTRAY_USER (brew test-bot)
• HOMEBREW_BINTRAY_KEY (brew test-bot)
• git-name (brew test-bot)
• git-email (brew test-bot)
• test-bot-user (brew pull --bottle)
• BottleSpecification::DEFAULT_DOMAIN (to get correct root_url)

These are currently passed as cmdline arguments or env vars, which could instead be properties of the tap object (perhaps read from a .brew-tap.yml file in the tap repo). Of course, the HOMEBREW_BINTRAY_KEY will need to remain an encrypted env var.

These lines also hardcode where to find bottle patches. If we're going to support Travis CI push builds, then we'd need something similar.

[sjackman]

I've successfully used CircleCI to create macOS and Linux bottles for a tap:
https://github.com/Linuxbrew/homebrew-bioinformatics/blob/master/.circleci/config.yml
I created a user named LinuxbrewTestBot. If your tap were in an organization Foo, the test-bot user would be called FooTestBot and your Bintray organization would be called foo.

[MikeMcQuaid]

I suggest uploading bottles from master branch builds (i.e. test builds), since this is the point at which they are "accepted" into the tap

This would need to not just upload bottles but also push a commit with the bottle checksums.

In the absence of a TestBot user, I suggest using the owner's account as the TestBot user (creating testing-#{build_id} tags and branches)

This seems sensible 👍

These are currently passed as cmdline arguments or env vars, which could instead be properties of the tap object (perhaps read from a .brew-tap.yml file in the tap repo).

I think environment variables for these would be ideal for when they can't be inferred automatically. I agree automatic inferring would be ideal, though.

If your tap were in an organization Foo, the test-bot user would be called FooTestBot and your Bintray organization would be called foo.

I think the FooTestBot pattern makes sense in terms of overhead for orgs but not taps with a single user as maintainer.

---

As a general comment: thanks for documenting and experimenting with this @davidchall. I'd definitely like a generic process we can support for people to be able to use Travis CI and other CI providers to build bottles. I doubt this will be something that ever scales to official taps given the 60m build timeout but it should work otherwise and I'm open to reviewing and making changes to get us there.

[ilovezfs]

I'm not sure encouraging bottling in third-party taps is going to have a salutary impact on the third-party Homebrew ecosystem overall.

If there is a breaking upgrade to a library dependency of a bottle, then the bottle breaks. That means the software is broken for anyone who has already installed the bottle, and for anyone who is freshly installing the bottle. The bottle will remain broken until the tap owner gets around to building and publishing a new bottle. In the meantime anyone doing a fresh install will get a broken bottle and have to uninstall it and start over with a source install, assuming they can figure out what is going on.

By contrast, in a tap without bottles, preexisting installations will break when there is a breaking upgrade to a library dependency. But any fresh installations, since they will be source installs, will be fine, and there's no waiting on the tap owner to find out about and fix the problem.

So encouraging bottling in third-party taps will mean spreading the problems that have been endemic to the science tap more broadly, and triggering a constant cycle of break-notify-rebuild-republish-break-notify-rebuild-republish increasing the burden on tap owners and making the user experience more inconsistent.

[scpeters]

@ilovezfs I maintain the osrf/simulation tap and build bottles for it, and we have daily jenkins jobs that test the bottles, so that we know when to build new ones. For software that isn't quick to compile, even if the bottles fail 1% of the time, I think the average user experience is better.

Maybe I could complement my bottle-testing job with a job that searches github for homebrew-core pull requests that modify dependencies of our formulae so we aren't surprised when they get merged.

[ilovezfs]

we have daily jenkins jobs that test the bottles, so that we know when to build new ones.

I guess that proves the point.

[ilovezfs]

Maybe I could complement my bottle-testing job with a job that searches github for homebrew-core pull requests that modify dependencies of our formulae so we aren't surprised when they get merged.

subscribing to homebrew/core and using a mail filter is probably the simplest approach

[scpeters]

subscribing to homebrew/core and using a mail filter is probably the simplest approach

I'm doing that as we speak. It works for individuals but doesn't scale as well for teams wanting these notifications.

[scpeters]

we have daily jenkins jobs that test the bottles, so that we know when to build new ones.

I guess that proves the point.

It's true, and I agree that it's a hassle, but it's worth the hassle for me since our software takes a long time to compile. I guess I'm just saying you don't have to encourage bottling for third-party taps, but I hope you don't disable it, since we find it useful.

[ilovezfs]

Yeah, we wouldn't disable it :)

[sjackman]

If there is a breaking upgrade to a library dependency of a bottle, then the bottle breaks.

A solution that addresses some (but not all) of this breakage is…
after brew installing a bottle, automatically run brew linkage on that keg, and if it fails, fall back to installing from source. This handles renamed dynamic library files. It currently does not handle removed/renamed symbols in those libraries, though it could be extended to do that as well.

[ilovezfs]

That's pretty much papering over the problem with an inconsistent UX.

[scpeters]

I have a confession to make: I was running Yosemite on my development laptop until just a few weeks ago. I backdated my homebrew-core to a point that still had yosemite bottles for the packages I needed and set HOMEBREW_NO_AUTO_UPDATE=1 in my shell profile. I kept brew up to date with cd $(brew --repo) && git pull, and I could still install old yosemite bottles. I just had to remember to never brew up. This is not supported and no one should ask for help if they do it, but it seemed to work for me.

If 3rd-party taps could temporarily backdate homebrew-core, this would allow them to keep working until their bottles are rebuilt. I don't know how this would be implemented; I think it's possible, though it may not be desired functionality.

If this functionality was built, I could imagine brew doctor complaints if people have homebrew-core backdated.

[stek29]

Or bottles can include some metadata on what versions of dependent formulas (&their deps) they were built with, and brew install would check that metadata and fallback to source build if it doesn't match

[alyssais]

The metadata is there in INSTALL_RECEIPT.json inside the bottle, but I don’t think version checking is a good solution. Most version changes won’t require building from source, and rebuilding bottles for all dependents every version change would be unsustainable for us.

[stek29]

@alyssais placing metadata in separate file would allow checking it without downloading whole bottle. It might be placed as separate tag, so it doesn't break older brew.

Are dependent formulas rebotlled manually now?

Also, if it's just a version bump, only metadata on dependent formulas can be updated. Not ideal, but still better than rebuilding.

[stek29]

Or another option -- add some revision number to formula, which is bumped every time a breaking change requiring rebuild of dependent bottles occurs, and use that as a reference instead if version. Not sure on how to do that without surprising older brew versions though.

[alyssais]

Are dependent formulas rebotlled manually now?

Not by default, but they often are when the maintainers expect them to be broken.

Also, if it's just a version bump, only metadata on dependent formulas can be updated. Not ideal, but still better than rebuilding.

This would require a lot of manual testing if I’m understanding it right.

add some revision number

This is how we currently do this.

[MikeMcQuaid]

@stek Discourse is a better forum for questions and ideas if you've not personally written any code to improve this. Note we're not going to change the bottle file format.

[stek29]

@MikeMcQuaid I've not personally written any code to improve this, but I was planning to.
(Has anybody? It's an open issue which wasn't worked on yet, isn't it? it also has help wanted and is even listed in gsoc "ideas", so I expected it to be open for discussion relevant to the issue)

After looking through other issues, I got an impression that you've meant "don't leave comments unless you contributed to Homebrew and/or planning to work on this issue" -- if I'm right, then let me say that I did make some minor contributions to homebrew-core, and recently made a PR for brew :)

What I was suggesting is this:

• Bottle (as the tarball) format stays unchanged
• Bottle (as in Formula source) gets additional property specifying file with metadata for current bottle -- which would include copies of INSTALL_RECEIPTs of other tags
• Optional (to avoid changing all existing formulas) revision number, which is only bumped on breaking change is added to Formula (not the same as Formula.revision -- Formula.version / Formula.revision may change while keeping binary compatibility, while revision number I'm suggesting is bumped only if breaking changes requiring rebottling happen -- I'll refer to it as binary_revision)
• INSTALL_RECEIPT includes that "binary_revision" per required runtime dependency
• When installing from bottle:
  
  
  
  • Check for presence of metadata tag, if it exists get needed INSTALL_RECEIPT from it (if it doesn't, fallback to old behaviors (just install hoping that nothing breaks))
  
  
  • Check if binary_revisions of all installed Formulas are matching: if they are, fetch bottle and install it. If they're not, show a warning and install from source
  
  
• When checking formula's binary_revision, infer it from version / version+revision if it's not present
• Optionally require binary_revision for all new/updated formulas in future so eventually all formulas have it (unless they never update -- then they don't need it)
• When updating & rebottling a formula: add that new tag with all INSTALL_RECEIPTs to it
• When formula is updated and it's binary_revision is changed, it suggests maintainers to rebottle all dependent formulas. Actually, it could've been done automatically, but there's one thing I'm not sure about:

Problem: Dependencies of Dependencies. Let's say there's:

• A: depends on B
• B: depends on C
• C

Now, B and C can be some libraries, and so if C's binary_revision is changed, both A and B should be rebuilt.
However, if C is a library and B is some kind of language (I can't come up with any better analogy), then A probably should not be rebuilt.

There's an option -- when formula's binary_revision is changed, maintainers trigger rebuild on all dependent formulas, and optionally change their binary_revision's if "they [maintainers] expect them [dependencies of rebottled formula] to be broken".

[MikeMcQuaid]

@MikeMcQuaid I've not personally written any code to improve this, but I was planning to.

@stek29 Consider you just made two excellent PRs today: consider my comments revoked and apologised for ❤️

Optional (to avoid changing all existing formulas) revision number, which is only bumped on breaking change is added to Formula (not the same as Formula.revision -- Formula.version / Formula.revision may change while keeping binary compatibility, while revision number I'm suggesting is bumped only if breaking changes requiring rebottling happen -- I'll refer to it as binary_revision)

Bottles have a rebuild that's used if you're rebuilding the same version/revision but want a new bottle.

Bottle (as in Formula source) gets additional property specifying file with metadata for current bottle -- which would include copies of INSTALL_RECEIPTs of other tags

I'm not sure what this would be for? Can you elaborate? Checking all versions within a bottle and building from source if they don't match? There's no real way of detecting this automatically, unfortunately, and doing so manually would be a lot of work.

[stek29]

@MikeMcQuaid to avoid downloading whole bottle to only find out that it links to incompatible versions of formulas.

There's no real way of detecting this automatically, unfortunately, and doing so manually would be a lot of work

I'm suggesting adding a property "binary_revision" to formula which is manually bumped only in case of breaking changes which would require rebottling formulas depending on it.
If I understood @alyssais correctly, currently maintainers manually monitor each formula update and rebottle formulas depending on updated formula if they expect it to break.
Instead, they can bump formula's binary_revision and get all dependencies rebuilt.

However, as I noticed previously, "deps of deps" aren't that simple to handle.
I can only think of this: bump binary_revision manually on all rebottled formulas if needed, causing rebottling of their deps, do that on deps, and so on recursively.

I just discovered brew linkage, so I guess it could be used to simplify the work for maintainers.

brew linkage output is bundled in bottle metadata file along with binary_revision's.
When formula's binary_revision is bumped, maintainer runs a command, which:

• Triggers rebottling of all direct dependents of updated formula
• Goes through all dependents recursively, fetching metadata file, and checking if they link to formula which was bumped. If they do, suggests bumping that dependent's binary_revision, and marks that formula as bumped, so when going through other dependents it would suggests bumping for anything that linked to that formula.

[ilovezfs]

FYI we don't actually revision bump or want to revision bump and rebottle every recursive reverse dependency, only the ones that actually break without the revision bump.

[stek29]

@ilovezfs That's why it should "suggest" to bump it.

Also, could there be a case when A and B depend on C, but only A needs to be rebuilt when C is updated with some breaking changes?

[ilovezfs]

Yup, that happens all the time.

[stek29]

Ok, then what about providing both option to fully rebuild formula and to only rebuild metadata?

Again, A and B depend on C.
C is updated.

• If C update is non-breaking, binary_revision is not changed
• If C update is potentially breaking (breaks A but not B), binary_revision is changed, and:
  
  
  
  • A is fully rebuilt and it's metadata file is updated in result. Additionally, it's binary_revision is bumped, and the same process is done for A.
  
  
  • B only has it's metadata file updated to mark it tested and confirmed to be working with new binary_revision of C.
  
  

That means that after update all dependents are tested and either marked "just working" with new version, are rebuilt (but not marked changed so their dependents don't need to be rebuilt), or rebuilt and rebuild is marked as breaking, so their dependents need to be checked too.

[ilovezfs]

B only has it's metadata file updated to mark it tested and confirmed to be working with new binary_revision of C.

That means either

a) the INSTALL_RECEIPT, which is stored inside the prebuilt bottle, not apart from it, so cannot be updated without rebuilding the bottle
b) the formula file, which means you're maintaining "it works OK" state per dependency per version in the formula
c) some other files that don't exist yet

None of these sound like they can be handled automatically.

[stek29]

@ilovezfs c -- the new property/tag which is added to the bottle.

Also, they reference binary_version and not version/revision -- if there's some minor change which doesn't break anything, then binary_revision isn't bumped. If it breaks at least one formula in core, it may also likely break third-party taps, so binary_revision is bumped and all dependent formulas are reviewed (with assistance of automatic tool?..), and either rebuilt & binary_revision bumped, just rebuilt without bumping binary_revision (it's dependents aren't broken), or only metadata file is rebuilt to mark it compliant with current binary_revision without rebuilding.

It's certainly is not easier or prettier for maintainers, and it introduces an overhead of metadata file & rebuilding that metadata file from time to time, but I think that it would make bottle system more robust and make third-party bottles easier to use for end users and to make for third party tap maintainers.

If you don't think it's worth it -- I guess some other solution is needed :)

[MikeMcQuaid]

@MikeMcQuaid to avoid downloading whole bottle to only find out that it links to incompatible versions of formulas.

@stek29 I think it's not worth optimising for this case just to avoid downloading a single bottle, for what it's worth.

[sjackman]

placing metadata in separate file would allow checking it without downloading whole bottle.

Though I'm not arguing for or against the purpose of this comment, it is worth mentioning that the INSTALL_RECEIPT.json and .brew/formula.rb can be placed first in the tarball, so that those files can be downloaded without downloading the entire bottle. It is a pretty minor patch to place those meta-files first in the archive, by placing them first in the tar c command line.

In fact, by chance, INSTALL_RECEIPT.json is usually early in the archive on macOS, due to the default file system sort order on macOS being alphabetical—sadly not on Linux, do to the default file system sort order being different.

❯❯❯ tar tf ~/Library/Caches/Homebrew/gcc-7.3.0.el_capitan.bottle.1.tar.gz | head -n7
gcc/7.3.0/
gcc/7.3.0/.brew/
gcc/7.3.0/bin/
gcc/7.3.0/ChangeLog
gcc/7.3.0/COPYING
gcc/7.3.0/include/
gcc/7.3.0/INSTALL_RECEIPT.json

The following command exits after downloading the enough of the bottle to fetch extract INSTALL_RECEIPT.json.

❯❯❯ curl -L https://homebrew.bintray.com/bottles/gcc-7.3.0.el_capitan.bottle.1.tar.gz | gtar -xzO --occurrence=1 gcc/7.3.0/INSTALL_RECEIPT.json | jq . | wc
      60      92    1486
curl: (23) Failed writing body (0 != 16384)

BSD tar doesn't unfortunately have the same behaviour.

❯❯❯ curl -L https://homebrew.bintray.com/bottles/gcc-7.3.0.el_capitan.bottle.1.tar.gz | tar -xzO gcc/7.3.0/INSTALL_RECEIPT.json | jq . | wc
     60      92    1486

[sjackman]

Also, could there be a case when A and B depend on C, but only A needs to be rebuilt when C is updated with some breaking changes?

Yup, that happens all the time.

That may caused by A having a direct linkage dependency on C, but no direct dependency specified in the formula, relying instead on the indirect dependency of A on B, and B on C. If A has a direct linkage dependency on C, it could instead specify that dependency in its formula, rather than rely on the indirect dependency through B. brew linkage now reports these indirect dependencies thanks to PR https://github.com/Homebrew/brew/pull/3785.

[sjackman]

When HOMEBREW_DEVELOPER is set, brew install runs brew linkage to check for broken linkage.
https://github.com/Homebrew/brew/blob/144356e23846ca5332891b358d2a0a5789546dbe/Library/Homebrew/formula_installer.rb#L569
https://github.com/Homebrew/brew/blob/144356e23846ca5332891b358d2a0a5789546dbe/Library/Homebrew/extend/os/mac/formula_cellar_checks.rb#L90

Currently that checks for missing libraries, which catches most cases of incompatible library names (install names on macOS or SONAME on Linux), that correctly bump the library name when their ABI changes. For misbehaved libraries that do not bump their library name when changing the ABI, brew linkage could check for missing symbols as well, which would be a slower check. Finally, neither of these would catch an incompatible ABI change that does not remove symbols (that is, changes the behaviour of an existing symbol without changing its name). brew test foo may catch some of those cases.

[scpeters]

Automating the process would be nice, but the bigger issue to me is that there is no grace period when a bottle in a 3rd-party tap gets broken by a new upstream dependency. There is inherent downtime when a core PR is merged that breaks downstream bottles.

Is it possible to build downstream bottles for a 3rd party tap before the homebrew-core PR is merged?

[stek29]

@scpeters Putting homebred-core PRs on hold for 3rd party taps seems unreasonable for me, but they probably can "watch" homebrew-core and check every new PR seeing if it changes anything that is in dependency tree of your formula

[scpeters]

they probably can "watch" homebrew-core and check every new PR seeing if it changes anything that is in dependency tree of your formula

That's what I'm currently doing, but there's inherent downtime if I have to wait for my bottles to be broken in order to fix them. I wouldn't propose to block merging homebrew-core PR's until 3rd party taps are ready, but a 1 hour grace period requested by a Homebrew contributor might not be too much of a burden? If I can't finish the job during the grace period, then that's on me.

For comparison, I think this is more reasonable than my other idea ( https://github.com/Homebrew/brew/issues/3346#issuecomment-369708453 ) of allowing taps to request to pin homebrew-core to an older version. Something similar could also be done if we made tags for homebrew-core, but I saw a recent @MikeMcQuaid comment stating that isn't a desirable idea.

The grace period is the best idea I have for now that would make my life a little easier as a 3rd-party bottler.

[scpeters]

I'll add that I think my company would be willing to pay money to improve the stability of bottles in our tap. I do try to contribute my time to brew, but if money would help, then that's something we could talk about too.

[ilovezfs]

@scpeters any reason you haven't vendored all of your deps in the tap? Then you could merge the upgrades at your convenience.

[sjackman]

I'm not a fan of vendoring dependencies, because it results in duplication of maintainer effort. It would be great if formulae in third party taps would build from source until their bottles were updated, when a core formula on which it depends is updated in a way that is known to break ABI compatibility, as suggested by @davidchall.

[scpeters]

I'm not a fan of vendoring dependencies, because it results in duplication of maintainer effort.

Yeah, vendoring feels like a shame when there are already compatible binaries that have been built.

[MikeMcQuaid]

Though I'm not arguing for or against the purpose of this comment, it is worth mentioning that the INSTALL_RECEIPT.json and .brew/formula.rb can be placed first in the tarball, so that those files can be downloaded without downloading the entire bottle. It is a pretty minor patch to place those meta-files first in the archive, by placing them first in the tar c command line.

This would be quite clever and useful if integrated with Homebrew itself.

BSD tar doesn't unfortunately have the same behaviour.

This would limit our use here unfortunately.

Is it possible to build downstream bottles for a 3rd party tap before the homebrew-core PR is merged?

Not really, unfortunately, without putting the onus on Homebrew/homebrew-core maintainers to fix up third-party taps (which is one of the biggest reasons in a month there will be no Homebrew organisation formula taps any more).

Automating the process would be nice, but the bigger issue to me is that there is no grace period when a bottle in a 3rd-party tap gets broken by a new upstream dependency. There is inherent downtime when a core PR is merged that breaks downstream bottles.

Yep, this is the fundamental problem that I think should be solved here. @sjackman has previously talked about third-party taps downloading old bottles and installing them to avoid linkage issues. What's the latest on that?

I'll add that I think my company would be willing to pay money to improve the stability of bottles in our tap.

Thanks for mentioning that. Certainly if we have any @Homebrew/maintainers who would be interested in spending a bit of paid time on this I think that would be a good call. I would offer but I kinda need the time more than the money right now.

It would be great if formulae in third party taps would build from source until their bottles were updated, when a core formula on which it depends is updated in a way that is known to break ABI compatibility, as suggested by @davidchall.

When https://github.com/Homebrew/brew/pull/3720 is merged brew linkage should be able to be fast enough to allow something similar.

[MikeMcQuaid]

That's what I'm currently doing, but there's inherent downtime if I have to wait for my bottles to be broken in order to fix them. I wouldn't propose to block merging homebrew-core PR's until 3rd party taps are ready, but a 1 hour grace period requested by a Homebrew contributor might not be too much of a burden? If I can't finish the job during the grace period, then that's on me.

Thinking a bit more about this: something here could be possible if there was a simple Ruby/Heroku service that polled (or, ideally used webhooks) a homebrew-core for new commits, ran brew test-bot to verify if anything is broken in a given tap and open a PR with a revision bump if so.

[scpeters]

Yep, this is the fundamental problem that I think should be solved here. @sjackman has previously talked about third-party taps downloading old bottles and installing them to avoid linkage issues. What's the latest on that?

Downloading old bottles sounds nice to me. The following will add bottle url and sha256 to INSTALL_RECEIPT.json for each dependency, so you could see what a bottle was built against:

~
diff --git a/Library/Homebrew/tab.rb b/Library/Homebrew/tab.rb
index 0fed724a8d..635b70502e 100644
--- a/Library/Homebrew/tab.rb
+++ b/Library/Homebrew/tab.rb
@@ -34,7 +34,13 @@ class Tab < OpenStruct
"aliases" => formula.aliases,
"runtime_dependencies" => formula.runtime_dependencies.map do |dep|
f = dep.to_formula
- { "full_name" => f.full_name, "version" => f.version.to_s }
+ bottle_hash = if f.bottled?
+ r = f.bottle.resource
+ {"url" => r.url, "sha256" => r.checksum.hexdigest }
+ else
+ {}
+ end
+ { "full_name" => f.full_name, "version" => f.version.to_s, "bottle" => bottle_hash }
end,
"source" => {
"path" => formula.specified_path.to_s,
~

Would this help?

[ilovezfs]

Old bottles sounds like a mistake to me. We don't want people using old bottles for dependencies that are used by stuff in core that the users may have installed too.

[ilovezfs]

Yeah, vendoring feels like a shame when there are already compatible binaries that have been built.

Whether it's a shame or not, I can tell you without question this is what I would do if I wanted stable bottles in a third-party tap, especially if this is for business purposes.

[sjackman]

BSD tar doesn't unfortunately have the same behaviour.

This would limit our use here unfortunately.

It's possible for Brew (Ruby) to scan the output curl | tar and exit the pipe once the INSTALL_RECEIPT.json file is output.

[sjackman]

Downloading old bottles sounds nice to me. The following will add bottle url and sha256 to INSTALL_RECEIPT.json for each dependency, so you could see what a bottle was built against

I like this idea.

Old bottles sounds like a mistake to me. We don't want people using old bottles for dependencies that are used by stuff in core that the users may have installed too.

The old bottles can be installed but not linked, if the library linkage is to files in the keg rather than HOMEBREW_PREFIX/lib or HOMEBREW_PREFIX/opt/foo/lib.

[ilovezfs]

The dylib library id is the opt path.

[sjackman]

It could be rewritten at install time to the version specified by INSTALL_RECEIPT.json.

[sjackman]

It would be great if formulae in third party taps would build from source until their bottles were updated, when a core formula on which it depends is updated in a way that is known to break ABI compatibility, as suggested by @davidchall.

When #3720 is merged brew linkage should be able to be fast enough to allow something similar.

The solution of building from source until the bottle is rebuilt may be simpler to implement, particularly if the situation is detected using brew linkage.

I have an idea of including an abi value in the formula DSL. The abi of the dependencies of a formula is stored in its INSTALL_RECEIPT.json. If the abi value of any of its dependencies is different than what is what built against, then that formula is built form source.

For formulae that do a good job of using semantic versioning, the value of abi is simply the major version number, or the library SONAME (install name).

[ilovezfs]

And get blown away when the user runs brew cleanup.

[ilovezfs]

I have an idea of including an abi value in the formula DSL. The abi of the dependencies of a formula is stored in its INSTALL_RECEIPT.json. If the abi value of any of its dependencies is different than what is what built against, then that formula is built form source.

I am not eager to maintain these, honestly.

[sjackman]

And get blown away when the user runs brew cleanup.

brew cleanup can be improved to be knowledgable of these dependencies.

I am not eager to maintain these, honestly.

Fair enough. That's certainly a consideration. We can use the result of brew linkage as a proxy.

[ilovezfs]

The old bottles can be installed but not linked, if the library linkage is to files in the keg rather than HOMEBREW_PREFIX/lib or HOMEBREW_PREFIX/opt/foo/lib.

Also, note that only keg_only bottles are expected or tested to work while unlinked, so again I really don't see this old bottle idea going anywhere.

[MikeMcQuaid]

Something I considered a long time ago for Homebrew's bottling as a whole was for each bottle to include all of the recursive dependencies within its Cellar prefix. This approach has worked well for macOS .app bundles. It would clearly be overkill for Homebrew/homebrew-core at this point but it could work as an approach for taps to have their bottles be standalone.

[ilovezfs]

How is that really distinct from vendoring? The binaries would still need to be rebuilt rather than reused from homebrew-core.

[sjackman]

Vendoring requires duplicating the formulae of a formula's recursive dependencies in that formula, and then maintaining those duplicated formulae, which requires significant effort. Mike's suggestion avoids that duplication.

[ilovezfs]

Rebuilding n deps every time there's an upgrade or revision bump of the main formula seems way over the top compared with having separate formulae.

[ilovezfs]

Take Qt for instance. Are you seriously proposing rebuilding Qt every time you upgrade something using it?

[sjackman]

I thought that's what you were proposing, if you're suggesting vendoring Qt into a formula that depends on Qt.

[ilovezfs]

Yes, building Qt as often as you need/want Qt to be upgraded, not as often as the rev dep using it!

[scpeters]

Yes, building Qt as often as you need/want Qt to be upgraded, not as often as the rev dep using it!

I had interpreted this as @sjackman did after looking at how several core formulae are using resources. Your clarification sounds better to me now, though I'm not sure how I would implement it without seeing an example or some guidance.

[sjackman]

A 25-line formula that depends on Qt (145 lines) becomes 170 lines if you vendor Qt by adding its formula to your formula. I'm not a fan of maintaining that duplication of code. I'd be more inclined toward a solution that uses an unlinked older version of its keg.

[ilovezfs]

@sjackman using non-keg-only bottles as if they're keg-only sounds like code smell.

[scpeters]

Something I considered a long time ago for Homebrew's bottling as a whole was for each bottle to include all of the recursive dependencies within its Cellar prefix. This approach has worked well for macOS .app bundles. It would clearly be overkill for Homebrew/homebrew-core at this point but it could work as an approach for taps to have their bottles be standalone.

I think this is perfect for leaf applications, but I suspect it's less ideal for developers who want to build against mid-level libraries. It's one thing if the only cost is increased disk usage due to duplicate installed dependencies, but symbol collisions between different library versions could be more complicated. That's just speculation of course.

[MikeMcQuaid]

I think this is perfect for leaf applications

Yep, that was the intended application. Note that anything that's cellar :any could likely be included as-is in the bottle without needing rebuilds (and libraries are much more likely to be in this category).

---

I'm open to any solution and I don't think as a four we'll reach any agreement on the best course of action. The only requirements really are:

• a non-maintainer will do most of the implementation
• no changes in workflow are required for Homebrew/brew and Homebrew/homebrew-core
• a working source build is prioritised over a broken bottle build

And ideally:

• as little as possible work is required for tap maintainers
• as much as possible is doable on e.g. Travis CI's free tier

[sjackman]

I'm okay rebuilding a bottle in a tap that depends on qt each time qt is updated in core. That's precisely what's done in Homebrew/core, and it works there. I want to avoid the short-term breakage in the interim before that bottle is rebuilt. Other less actively maintained taps may have other priorities, where vendoring is a better solution for them.

[sjackman]

a working source build is prioritised over a broken bottle build

:+1: Preferring a source build over a broken bottle build is the simplest path forward, to avoid interim breakage.

[MikeMcQuaid]

I would also likely to gently suggest that unless @scpeters and @sjackman can reach some sort of agreement on the best approach the wider community is unlikely to benefit from a single, implementable recommendation.

[scpeters]

I've reached out to @sjackman on twitter and will report back

[ilovezfs]

I'm okay rebuilding a bottle in a tap that depends on qt each time qt is updated in core. That's precisely what's done in Homebrew/core, and it works there. I want to avoid the short-term breakage in the interim before that bottle is rebuilt. Other less actively maintained taps may have other priorities, where vendoring is a better solution for them.

@sjackman I think you've literally been saying the opposite with the use-old-bottles proposition.

[ilovezfs]

I had interpreted this as @sjackman did after looking at how several core formulae are using resources. Your clarification sounds better to me now, though I'm not sure how I would implement it without seeing an example or some guidance.

@scpeters so for example osrf/simulation/haptix-comm depends on protobuf, protobuf-c, zeromq, and cppzmq from homebrew/core. Instead you can create

• osrf-protobuf
• osrf-protobuf-c
• osrf-zeromq
• osrf-cppzmq

by copying each of those formulae from homebrew/core into osrf/simulation with a different name (e.g. prepend osrf-), and adding keg_only to them.

Then use

  depends_on "osrf-protobuf"
  depends_on "osrf-protobuf-c" => :build
  depends_on "osrf-zeromq"
  depends_on "osrf-cppzmq"

That should go a long way to making your bottles more stable.

[scpeters]

Thanks @ilovezfs that makes sense. I was interpreting "vendoring" as requiring the use of the resource blocks, but forked formulae is a way of doing it.

Falling back to a build from source if the bottle breaks would be less effort / less performant, but I think that's where I'm currently leaning.

[sjackman]

Thanks @ilovezfs that makes sense. I was interpreting "vendoring" as requiring the use of the resource blocks, but forked formulae is a way of doing it.

That's also how I was interpreting your use of the word vendoring. This proposal is better than that.

[sjackman]

I'm okay rebuilding a bottle in a tap that depends on qt each time qt is updated in core. That's precisely what's done in Homebrew/core, and it works there. I want to avoid the short-term breakage in the interim before that bottle is rebuilt. Other less actively maintained taps may have other priorities, where vendoring is a better solution for them.

@sjackman I think you've literally been saying the opposite with the use-old-bottles proposition.

There's multiple possible technical solutions to this problem. One such solution is using old bottles. My current preferred solution is building from source until the tap has updated the bottle.

[ilovezfs]

any reason you haven't vendored all of your deps in the tap?

Ah. Yeah, that's why I said "in the tap" not "in the formula" but I guess that was not clear.

[ilovezfs]

My current preferred solution is building from source until the tap has updated the bottle.

That of course doesn't do anything for people who already have something installed when the breaking brew upgrade hits, unless there's some revdep-rebuild magic that kicks in for existing installations of the reverse dependencies, not just a source build fall back on _new_ installs where a bottle is noticed to be broken on pour.

[sjackman]

Yes, good point.
brew upgrade could use brew linkage with the fast caching to detect that upgrading foo would break bar.
brew upgrade could rebuild bar from source.
brew upgrade could modify bar using install_name_tool to link to Cellar/foo/…/lib rather than opt/foo/lib.
brew upgrade could wait on upgrading foo until bar is rebottled, if that rebottling would happen in the near future, perhaps triggered automatically as Mike suggested. This option is not feasible if the rebottling of bar takes a long time or never happens.

[ilovezfs]

Some new brew magic that triggers a bar source build would be fine so that taps with absentee landords aren't totally broken. I still think best practice is bottling your own deps.

[scpeters]

Some new brew magic that triggers a bar source build would be fine so that taps with absentee landords aren't totally broken.

👍

I still think best practice is bottling your own deps.

If there was an easy way to rename a working core bottle, say boost, to osrf-boost without rebuilding, it would be much more attractive. Not sure how viable that is.

[wjwwood]

If there was an easy way to rename a working core bottle, say boost, to osrf-boost without rebuilding, it would be much more attractive. Not sure how viable that is.

That'd be nice, but it doesn't solve the proverbial framework integration issue, which is like "gazebo depends on osrf-boost which is at x.y and opencv depends on just boost which is at x.y+n and so gazebo and opencv can potentially not be linked together now". So then you end up basically needing to fork all of homebrew if you want to let your users combine your software arbitrarily with any other software in homebrew. Note, that's not a real example, just something I made up as an illustration, but something like this is realistic (imo) and has happened many times before to me.

[ilovezfs]

So then you end up basically needing to fork all of homebrew if you want to let your users combine your software arbitrarily with any other software in homebrew.

Yeah that would expressly need to not be a goal of the tap.

[wjwwood]

Yeah that would expressly need to not be a goal of the tap.

For this discussion, I agree. I didn't mean to get this issue off topic.

It's probably a somewhat unique requirement given that ROS integrates with some many other libraries at the user level while also using them internally, but it's also a big reason why we don't maintain a tap for ROS. That unfortunately means our users are more likely to avoid using macOS and Homebrew, and therefore also unlikely to contribute to Homebrew, though some do anyways.

[ilovezfs]

If there was an easy way to rename a working core bottle, say boost, to osrf-boost without rebuilding, it would be much more attractive. Not sure how viable that is.

It should be possible if you shadow the name instead of renaming. So

  depends_on "osrf/simulation/protobuf"
  depends_on "osrf/simulation/protobuf-c" => :build
  depends_on "osrf/simulation/zeromq"
  depends_on "osrf/simulation/cppzmq"

[ilovezfs]

(Note I wouldn't recommend trying that unless you're planning on expecting people to use tap-pin)

[sjackman]

Wouldn't that break formulae in Homebrew/core that depend on homebrew/core/protobuf, since they would see an older out-of-date version of protobuf?

[ilovezfs]

@sjackman yup.

[ilovezfs]

Is it possible to build downstream bottles for a 3rd party tap before the homebrew-core PR is merged?

@scpeters You should be able to pull the core PR into your own PR before we actually merge it to master.

[scpeters]

@scpeters You should be able to pull the core PR into your own PR before we actually merge it to master.

So brew pull --bottle should do it. Right. When I asked the question I had only ever used brew pull without --bottle (we modify bottle sha256's in our tap pull requests before merging). But it's clear in the maintainer docs; it sounds obvious now. Thanks for answering.

I'll try adding a core PR to our jenkins job parameters to see if this works.

[ilovezfs]

You'd need to brew pull without the --bottle since there won't be a published bottle yet, so you'd need to build the relevant dep yourself as well as part of the job.

[scpeters]

You'd need to brew pull without the --bottle since there won't be a published bottle yet, so you'd need to build the relevant dep yourself as well as part of the job.

Right when the pull request is made, there wouldn't be any published bottles, but they are uploaded once the jenkins CI jobs go green, right? If I was lucky with the timing to start the build after bottles had been uploaded, I think it might work? This is like an unofficial version of the "grace period" I discussed above. I'm not actually sure if that's how it works though.

[ilovezfs]

You could

brew pull --bottle --no-publish

and then manually download the bottles from Jenkins if they're already built and put them in your cache or upload them somewhere and set the root_url in the bottle block. They won't be on Bintray yet.

[scpeters]

They won't be on Bintray yet.

This was a little confusing to me, but I think I understand that they would be uploaded to bintray at this point but not published. So bintray credentials would be needed to download them.

You could brew pull --bottle --no-publish and then manually download the bottles from Jenkins if they're already built and put them in your cache or upload them somewhere and set the root_url in the bottle block.

Downloading the bottles from jenkins would take a little more work, crawling through the list of recent jenkins builds or getting the CI status from the GitHub API, but there's a limited number of recent builds, so they would potentially be deleted.

Is there room for a role in between contributor and maintainer (of homebrew-core) for 3rd-party tap maintainers who could have read-only access to unpublished bottles on bintray? I would consider this to be a privilege and that it wouldn't include an expectation of support from maintainers.

[sjackman]

I use CircleCI to build bottles for both Linux and macOS. You can download the bottle artifacts from CircleCI using their REST API. See https://github.com/Linuxbrew/homebrew-developer/blob/master/cmd/brew-pull-circle.rb#L50

You can then download the bottles form CircleCI and upload them to Bintray. I automated that step using a web hook and AWS Lambda. See https://github.com/Linuxbrew/linuxbrew-lambda

[scpeters]

I use Jenkins with machines in our office to build our bottles. In talking about unpublished homebrew-core bottles, I was referring to the unofficial "grace period" idea and trying to build bottles for a downstream tap before the homebrew-core pull request has been merged and bottles published to bintray.

[sjackman]

Ah, right. Thanks for the reminder, Steven.

[fredemmott]

I'm hoping to get HHVM's stuff in to a shape soon where I can send a PR so it's not third-party; for others, an alternative to full vendoring is to use static linking for dependencies. You can do this for everything, but what we've found best is to statically link dependencies that are particularly problematic (for example, ICU4C bumps the major version really often) and keep the stable stuff dynamically linked. Things still break occasionally but it's manageable.

[MikeMcQuaid]

I'm open to (and have been) accepting more specific PRs for this but closing this out as a bit too much of a meta-ask with no real completion requirements.