Brew: Merging homebrew-core and linuxbrew-core

[MikeMcQuaid]

Eventually we may want to merge homebrew-core and linuxbrew-core. Even if not, minimising the diff between these repositories makes life easier for the linuxbrew-core maintainers.

To do this will require a few things (not in any particular order):

• [x] Migrate Homebrew/homebrew-core CI to GitHub Actions CI https://github.com/Homebrew/brew/issues/6255 (which linuxbrew-core already is)
• [x] Have bottled as many linuxbrew-core formulae as possible (in linuxbrew-core)
• [x] Add more Linux DSLs to avoid if OS.linux? blocks (partly for readability and partly so these blocks can be parsed on macOS) like uses_from_macos e.g.
  
  
  
  • [x] a way to applying patches only on a single platform
  
  
  • [x] a way to have dependencies only on a single platform
  
  
  • [x] a block in def install that's only run on a single platform
  
  
• [ ] Finish the xorg migration, a few formulae need to be migrated
• [ ] Allow applying cellar :any to bottles on a single platform
• [ ] Reduce the diff between the two repositories to 0

[iMichka]

@sjackman I would like to make gcc@5 the main gcc formula on linux.
brew install gcc would then install gcc 9, like it does on mac. This would allow to cleanup the gcc formula on linux, and ease the transition. Do you think this is doable?

[issyl0]

As of #7280 we've got an audit for legitimate uses_from_macos usage.

[sjackman]

@sjackman I would like to make gcc@5 the main gcc formula on linux.
brew install gcc would then install gcc 9, like it does on mac. This would allow to cleanup the gcc formula on linux, and ease the transition. Do you think this is doable?

My intuition is that on Linux gcc should be an alias to gcc@5.

[sjackman]

As of #7280 we've got an audit for legitimate uses_from_macos usage.

Thanks, Issy!

[iMichka]

My intuition is that on Linux gcc should be an alias to gcc@5

This means that we need os-dependent aliases. Not sure I like that idea. And I still think it is surprising to brew install gcc and get something different than on mac.

[MikeMcQuaid]

I'm with @iMichka that I think the cleanest solution is probably to make the gcc formula the same on both OSs. It's easier to have Homebrew/brew compiler selection code pick a different compiler when needed than it's going to be to have homebrew-core have different logic for GCC versions with the same name.

[sjackman]

Makes sense. I'm on board.

[fdr]

FYI having the command "gcc" (on the system: 10.1.1, Fedora 32) clobbered by linuxbrew's gcc (5.5) caused some unexpected problems that I only started to encounter when compiling a gem with a c++ dependency.

It's also contra the documentation, "Homebrew does not use any libraries provided by your host system, except glibc and gcc if they are new enough."

I do not mind brew clobbering with new versions, but ancient ones? Hmmm.

My workaround has been to uninstall the "gcc" formula, ignoring the dependency requirement by "llvm" (which in turn is brought in by Postgres and maybe Crystal language).

[sjackman]

I do not mind brew clobbering with new versions, but ancient ones? Hmmm.

A fair comment. I'll keep this in mind when we address this issue.

[fmeyer]

@iMichka I recently stumbled in a very ancient GCC's ThreadSanitizer data race bug when compiling a Go program which relies on cgo. on linux

Just to understand, why does homebrew defaults to 5.5?

[iMichka]

Because we build on Ubuntu 16.04 which is a LTS version, and gcc5 is the default there.
You can modify a formula to make it use any other brewed gcc if needed.

[sjackman]

You can also use brew install --cc=gcc-9 hello

[fdr]

What about using the system GCC if it's adequate, per documentation?

[sjackman]

If the brewed GCC-5 is installed, it will always take precedence. If the brewed GCC-5 is not installed, and if the host GCC is new enough, it will be used.

[fdr]

Yeah, but when I install(ed) llvm, it decided to opt me into installing gcc. I took no special action to install GCC of any version myself. And LLVM is a dependency of Postgres and Crystal. Maybe it's a formula problem with llvm?

[iMichka]

Yes, because llvm currently depends on gcc as it provides libstdc++. If you know how to ship llvm without gcc, a PR is welcome.

Could we discuss this somewhere else, this issue was not about llvm and gcc intially: https://discourse.brew.sh

[iMichka]

Short status update, as some huge progress has been made lately:

• I update the checklist on the top of this issue
• Most of the important xorg formulae have been migrated to homebrew-core (which had the nice side-effect that some of maintainers started to work on getting rid of the x11 requirement :)). Around 13 remaining xorg formulae need to be migrated.
• 13 linux-only formulae (out of 43) have been added to homebrew-core and have linux CI there.
• When I started reducing the diff between both repos, there were more than 2000 OS.mac? statments. We are now below 1400.

[iMichka]

There was a proposal from @SMillerDev to start adding linux bottles and more CI to homebrew-core as the migration is ongoing. Initially I was thinking to add all the bottles in one step at the end of the migration; but we can also take a step-by-step approach. We can not re-use linuxbrew-core bottles as-is, as the revisions are sometimes different there.

• Where do we store the new linux bottles on bintray for homebrew-core? I guess in https://bintray.com/homebrew, and we keep the old linux bottles in https://bintray.com/linuxbrew?
• I think we do not need to adapt the merge script from homebrew-core to linuxbrew-core, as the linux sha256 line will conflict anyway, so this is a good thing.
• We will do double bottle builds for some time, in homebrew-core and then in linuxbrew-core. I hope that's fine
• Remains the final question: how will the transition happen from linuxbrew-core to homebrew-core for our users, knowing that revisions might be different between both repos. We do not need to answer this question right now.
• What is the impact of the migration on the shelf-life of old linux bottles? Is this still a thing? I know that @sjackman always thought it was important, but I personally do not have any need for this.

Thoughts?

[SMillerDev]

I think the best option to have would be:

We bottle the formula that are explicitly enabled to be tested in homebrew-core and use those bottles in linuxbrew where we can having less and less bottles in the linuxbrew bintray.

Second best option:

We bottle the formula that are explicitly enabled to be tested in homebrew-core and rebuild the bottles in linuxbrew. Until we flip the switch and we disable linuxbrew.

Worst option:

We don't bottle in homebrew-core, but we keep the same flow until all formulae are upstreamed from linuxbrew and then we start bottles in homebrew-core.

[dawidd6]

The first option looks great, but how can we know on which bintray account the Linux bottle actually is? This will become irrelevant when we merge the core taps together, but for now I think it would be a problem that needs to be sorted out first.

[sjackman]

We bottle the formula that are explicitly enabled to be tested in homebrew-core and rebuild the bottles in linuxbrew. Until we flip the switch and we disable linuxbrew.

I like this option.

[SMillerDev]

The first option looks great, but how can we know on which bintray account the Linux bottle actually is? This will become irrelevant when we merge the core taps together, but for now I think it would be a problem that needs to be sorted out first.

I'd imagine we can try the main one and fall back if it fails?

[iMichka]

I would go for second option. It's less work, and probably easier to handle.

We just need to think about the final migration when we flip the switch, but that's all.