Homebrew-core: Migrate popular formulae from Homebrew/homebrew-* taps

Created on 14 Jul 2017  路  22Comments  路  Source: Homebrew/homebrew-core

Migrate the most popular formulae from https://brew.sh/analytics/install/ (the top 500 of the top 1000 listed installed formulae):

The imported formulae should be treated like any new homebrew/core formula. This means they should:

  • Have their options paired down to the minimum ones that are used (based on brew formula-analytics data)
  • Become versioned formulae
  • Split into multiple formulae or become a single formulae as sensible
  • Not use OS.linux? and assume macOS everywhere
  • Remove unnecessary caveats
  • Not use any sort of AbstractFormula stuff

The primary motivation is to be able to improve the support and discoverability of this software. The secondary goal is to consolidate Homebrew/homebrew-core and eventually consider moving some or all non-Homebrew/homebrew-core formula taps out of the Homebrew organisation into their own organisations or users.

CC @Homebrew/apache @Homebrew/science @Homebrew/php

picture MikeMcQuaid
❤4

Most helpful comment

samtools and htslib will stay in science since they're bioinformatics

picture ilovezfs on 23 Jul 2017
👍2

All 22 comments

We originally removed the php ones from the core because no one on the core has time to maintain them. I can't imagine the support nightmare that would occur if we had two different places to install stuff from, so I am 馃憥 .

josegonzalez picture josegonzalez on 14 Jul 2017

I can't imagine the support nightmare that would occur if we had two different places to install stuff from, so I am 馃憥 .

We'd be removing them from the taps in favour of having them in core. We'd not be duplicating them in multiple places.

MikeMcQuaid picture MikeMcQuaid on 15 Jul 2017

There are two problems:

  • Some of our taps come in multiple versions for the various php things. For instance, we have php55 on your list but none of the extensions. Updating them in two places would be a pita and error-prone.
  • People will inevitably file issues upstream or in our tap for php stuff, causing issues for maintainers. We'd have to look at two places for issue tracking, and potentially need to move issues as appropriate. Thats a small issue now but will be more annoying and make it less likely for maintainers to, well, maintain the tap.

If we didn't have taps or there was a way to auto-tag/move issues, _maybe_ I'd be okay with it, but I don't think this is going to work out well for the current homebrew-php maintainers.

josegonzalez picture josegonzalez on 16 Jul 2017

@josegonzalez Ok. I'm interested in reasons that are user-focused rather than Homebrew/php maintainer focused and to hear from tap maintainers who have been more active in the last few years.

MikeMcQuaid picture MikeMcQuaid on 16 Jul 2017
👍1

I think if done right, it could make a lot of sense for the core php formulas, but it would take a lot of work.

After an import, does the core team then maintain the formulas going forward? Do the current tap maintainers just become contributors (pull-requests submitted for updates)?

The main PHP formulas would take some work to get them into an acceptable state for the core repo. We'd also have to figure out how to make sure that when the core version is update that the bundled extensions are also updated (the reason for the current abstract version handling).

ablyler picture ablyler on 19 Jul 2017

After an import, does the core team then maintain the formulas going forward? Do the current tap maintainers just become contributors (pull-requests submitted for updates)?

Yep. If there's regular, solid contributions to formulae then tap maintainers are likely to be offered core maintainership too.

MikeMcQuaid picture MikeMcQuaid on 19 Jul 2017

For php the only way we could possibly do this is to bundle the binaries and core extension into one formula - it doesn't really make sense to do it any other way. It would increase the number of dependencies of course.

I also started this issue in the php tap https://github.com/Homebrew/homebrew-php/issues/4005 to try and get my head around some of the challenges in making this into one package. I will give it some more thought and see if I can come up with something next week.

javian picture javian on 19 Jul 2017

Can you elaborate a bit on what extensions are? Are they options in formulae, resources or separate formulae?

MikeMcQuaid picture MikeMcQuaid on 19 Jul 2017

@MikeMcQuaid to quote somebody else PHP extensions are compiled libraries which enable specific functions to be used in your PHP code.. There are 2 different types of extensions. 1 type is maintained in the PHP source https://github.com/php/php-src/tree/master/ext and the the other type can be found on https://pecl.php.net that can be managed with the pear tool that you can compile when compiling the php source. Pear is currently not included by default in homebrew-php.

99% of the extensions that are in the homebrew-php tap should be available in pecl . One option could be to make pear be included by default and have the home-brew users manage their extensions with that rather than using formulas for it.

javian picture javian on 20 Jul 2017

One option could be to make pear be included by default and have the home-brew users manage their extensions with that rather than using formulas for it.

This seems like a very good idea, regardless of what happens with taps.

MikeMcQuaid picture MikeMcQuaid on 20 Jul 2017

There are 2 options in php that is mutually exclusive when installing php. They are both used and called thread safety and using debug symbols . The extensions needs to be compiled with the php version that supports 0 or one or many of those two options. If we were to support all of them we would need 4 separate formulas for php. One without any of the options, one with both enabled and one for each of them.

Looking at php 7.1 and the stats over the past two months it looks like this.

Option used | Unique install over past 2 months
------------ | ------------
None | 63280
Thread-Safety | 404
Debug | 1266
Both TS and DBG | 123

TS and Debug are currently only offered as options to the main php formula and I assume this could remain as it currently is working where the users requiring these options would manually reinstall. If pear was the default extension installer it would also take care about the installation.

Debian has made the choice of appending the version to all the executables to avoid have a naming conflict. https://packages.debian.org/stretch/amd64/php7.0-cli/filelist if we were to do this in the Home-brew Formula this would allows us to install multiple versions side by side and developers could easily test different version with ease. Would there be anything preventing us from designing it like this ?

javian picture javian on 21 Jul 2017

Unique install over past 2 months

I think given these numbers and that we don't install debug symbols/source properly it's worth just providing the main formula and the rest can be handled by other taps.

Would there be anything preventing us from designing it like this ?

If it can still work in a useful fashion like this (i.e. in a web browser) then: sure 馃憤

MikeMcQuaid picture MikeMcQuaid on 21 Jul 2017
👍1

Other candidates from homebrew-science:

fxcoudert picture fxcoudert on 23 Jul 2017

samtools and htslib will stay in science since they're bioinformatics

ilovezfs picture ilovezfs on 23 Jul 2017
👍2

Maybe also

  • [x] homebrew/science/ceres-solver (https://github.com/Homebrew/homebrew-core/pull/16465)
  • [x] homebrew/science/netcdf (https://github.com/Homebrew/homebrew-core/pull/16223)
ilovezfs picture ilovezfs on 25 Jul 2017

I opened a PR for netcdf here: #16223

iMichka picture iMichka on 30 Jul 2017

Other candidates, which are used as dependencies on other core formulas:

  • [x] science/cfitsio
  • [x] science/hwloc
fxcoudert picture fxcoudert on 9 Aug 2017

Here are analytics for installs in the last 100 days for all formulas in homebrew-science.

The top 20 looks like:

57761 opencv3
32979 opencv
 9687 matplotlib
 4541 pillow
 3303 htslib
 2879 samtools
 2059 mumps
 1927 pymol
 1867 root6
 1751 pcl
 1272 bowtie2
 1257 blast
 1140 lammps
 1072 graph-tool
 1050 ipopt
  951 ampl-mp
  942 igv
  926 cminpack
  921 dlib
  916 gmt
fxcoudert picture fxcoudert on 11 Aug 2017

That is instructive but I notice that some formulas are largely simplified when they are migrated to core. I think it's also important to look at the most-used options when the most-installed formulae are installed. Otherwise, simplifying the formulas could defeat the purpose and their usage might drop. For example, all Matlab extensions of suite-sparse were removed during the migration, but speaking for myself, the formula has basically become useless (suite-sparse is at the heart of Matlab).

dpo picture dpo on 11 Aug 2017

@dpo when migrating formulas to core, options are simplified/removed based on existing usage. We cannot test every option, and so each option is a risk of more bugs coming in due to lack of testing.

Regarding suite-sparse specifically, the analytics shows that 0.22% of users used the matlab extensions.

fxcoudert picture fxcoudert on 11 Aug 2017

when migrating formulas to core, options are simplified/removed based on existing usage. We cannot test every option, and so each option is a risk of more bugs coming in due to lack of testing.

I understand the rationale.

Regarding suite-sparse specifically, the analytics shows that 0.22% of users used the matlab extensions.

Ok, that's good to know. Thanks for confirming. Where can I have access to those specifics?

dpo picture dpo on 11 Aug 2017

Only PHP remains, which is being handled in https://github.com/Homebrew/homebrew-core/pull/16067 so I'm going to close this out.

ilovezfs picture ilovezfs on 24 Dec 2017
Was this page helpful?
0 / 5 - 0 ratings

Related issues

tejasmanohar picture tejasmanohar  路  3Comments
ghost picture ghost  路  3Comments
kiendang picture kiendang  路  3Comments
oli-laban picture oli-laban  路  3Comments
faraazkhan picture faraazkhan  路  3Comments