Homebrew-core: Formulae fail to build

[xu-cheng]

I have finished the massive bottle build. Below is the list of formulae fail to build.

• [x] blucat
• [x] cvs2svn
• [x] dwarf
• [x] file-roller
• [x] fsw (test timeout)
• [x] geocouch — #475
• [x] gexiv2
• [x] henplus
• [x] hqx
• [x] juise
• [x] kjell — #870
• [x] libcanberra
• [x] libmarisa
• [x] libreadline-java
• [x] mapserver
• [x] mercury
• [x] mikmod
• [x] mmix
• [x] ninja-ide
• [x] ocamlsdl (build timeout like 12hrs)
• [x] pg_top
• [x] pillar
• [x] pylucene
• [x] recode
• [x] sam2p
• [x] sdf
• [x] sile
• [x] skytools
• [x] slony
• [x] snow
• [x] soccerwindow2
• [x] sproxy
• [x] ssreflect
• [x] suricata
• [x] swatchbooker
• [x] valkyrie
• [x] vcodex
• [x] vmalloc
• [x] wartremover
• [x] wsmancli — fails because its dependency openwsman is broken: Homebrew/legacy-homebrew#44047
• [x] zint

Without an El Capitan bottle, but not necessarily broken:

• [x] apt-cacher-ng — depends_on :osxfuse
• [x] class-dump — MaximumMacOSRequirement => :mavericks
• [x] ctunnel — depends_on :tuntap
• [x] fastd — depends_on :tuntap
• [x] horndis — depends_on UnsignedKextRequirement
• [x] iodine — depends_on :tuntap
• [x] ld64 — MaximumMacOSRequirement => :snow_leopard
• [x] osxfuse — depends_on UnsignedKextRequirement
• [x] peervpn — depends_on :tuntap
• [x] shrewsoft-vpn-client — depends_on :tuntap
• [x] tuntap — depends_on UnsignedKextRequirement
• [x] vpnc — depends_on :tuntap

[MikeMcQuaid]

Failed to build on what platform/platforms? Do they have sandbox violations?

[DomT4]

I presume apt-cacher-ng was a 404, was being worked on here but hit other issues.

geocouch is likely to be this old bundle of fun, which is both a sandbox problem and a "Upstream like to break the checksum" problem.

[xu-cheng]

Failed to build on what platform/platforms?

Fail to build on El Capitan at least.

Do they have sandbox violations?

I don't know. Because the list is generated by subtraction. i.e. All the formulae without El Capitan bottles. (I have removed the formulae that explicitly not work on El Capitan, e.g. MaximumMacOSRequirement and UnsignedKextRequirement).

[DomT4]

I'd be tempted to get rid of geocouch FWIW. It has been broken for a fairly long time now and only a handful of users have noticed, and we've fixed the checksum before only for that to quickly break again. I'm unconvinced it's worth the effort to fix and likely keep having to fix, unless anyone really wants to.

[xu-cheng]

FYI, this is the script used to generate the list.

require "tap"
require "formula"
require "extend/ARGV"
tap = Tap.fetch(ARGV.named.first || "homebrew/core")
ff = tap.formula_files.map { |f| Formula[f] }
ff.reject! { |f| f.bottle_disabled? }
ff.reject! { |f| f.bottle_specification.collector.key? :el_capitan }
ff.reject! do |f|
  f.recursive_requirements.any? { |r| r.is_a?(UnsignedKextRequirement) || r.is_a?(MaximumMacOSRequirement) || r.is_a?(OsxfuseRequirement) || r.is_a?(TuntapRequirement) }
end
puts_columns ff.map(&:name)
puts ff.size / tap.formula_files.size.to_f

[ilovezfs]

apt-cacher-ng: osxfuse
ctunnel: tuntap
fastd: tuntap
iodine: tuntap
miredo: tuntap
peervpn: tuntap
shrewsoft-vpn-client: tuntap
vpnc: tuntap

So that subset is the unsigned kext requirement, too.

[ilovezfs]

This seems to fix fsw https://github.com/Homebrew/homebrew-core/pull/367

[ilovezfs]

Not having any issue with zint. The developer does still appear to be actively working on the project but mostly on SourceForge now. @xu-cheng was there a specific failure?

[ilovezfs]

OK, zint issue reproduced: sandbox violation https://github.com/Homebrew/homebrew-core/pull/386

Install the project...
/usr/local/Cellar/cmake/3.5.2/bin/cmake -P cmake_install.cmake
-- Install configuration: "Release"
-- Installing: /usr/local/Cellar/cmake/3.5.2/share/cmake/Modules/FindZint.cmake
CMake Error at cmake_install.cmake:39 (file):
  file INSTALL cannot copy file
  "/tmp/zint20160417-39600-b9whtp/zint-2.4.3/cmake/modules/FindZint.cmake" to
  "/usr/local/Cellar/cmake/3.5.2/share/cmake/Modules/FindZint.cmake".

[ilovezfs]

Seems there's no issue with suricata: https://github.com/Homebrew/homebrew-core/pull/482

[ilovezfs]

valkyrie seems to need unistd.h #502

[ilovezfs]

ssreflect is superseded by https://github.com/math-comp/math-comp which has coq 8.5 support

[DomT4]

pg_top and sam2p were both more or less fine for me locally and on CI so not sure what happened with them. Added bottles and tests for those in https://github.com/Homebrew/homebrew-core/pull/542, as well as fixing pillar and recode.

[ilovezfs]

Here's a fix for file-roller: https://github.com/Homebrew/homebrew-core/pull/716

[ilovezfs]

vcodex like ksh seems to have been subsumed by https://github.com/att/ast for which I've already requested a tag https://github.com/att/ast/issues/3 but received no response. Since the underlying code is still the same old bits as always, maybe stable should be defined as a specific revision until someone upstream agrees to create a tag?

[apjanke]

wartremover: sandbox violation by sbt. #748.

Is it okay if I just edit your original issue description to add references to individual issues as appropriate? I'm going to run through a few more of these over the weekend.

vmalloc: broken download on server side: #749
vcodex: also broken download on server side: #749

swatchbooker: just needs external dependency pillow.

$ brew install -s swatchbooker
swatchbooker: Unsatisfied dependency: pillow
Homebrew does not provide Python dependencies; install with:
  pip install pillow
Error: An unsatisfied requirement failed this build.

[ilovezfs]

This fixes dwarf: https://github.com/Homebrew/homebrew-core/pull/768

[ilovezfs]

It seems like although henplus actually builds fine, it's currently tag-less on GitHub. The download link in the formula doesn't appear to be published anywhere except in the formula, and is apparently the 2009-05-01 0.9.8 from Sourceforge (https://sourceforge.net/projects/henplus/files/henplus/).

However, upstream does still appear to be alive and latest commit was Feb 10, 2016 (https://github.com/neurolabs/henplus/commits/master). So maybe upstream needs to be pestered to create a new tag?
Requisite pestering commenced: https://github.com/neurolabs/henplus/issues/41

[martinlschumann]

ocamlsdl now builds correctly: #809

[ilovezfs]

hqx just needs devil bottling re-enabled, so https://github.com/Homebrew/homebrew-core/pull/839 is prerequisite, if anyone would like to look that over.

839 is now merged, so hqx builds again.

[martinlschumann]

mapserver also builds correctly #811

[ilovezfs]

Problem with juise is a sandbox violation: #865 which needs review.

[ilovezfs]

kjell is also a sandbox violation: https://github.com/Homebrew/homebrew-core/pull/870, which needs review

[ilovezfs]

libcanberra dies with "configure: error: Unable to find libltdl" unless it depends on libtool: https://github.com/Homebrew/homebrew-core/pull/871

[ilovezfs]

libmarisa builds ok with clang and gcc5, but make check fails with clang: https://github.com/Homebrew/homebrew-core/pull/875

[ilovezfs]

Only problem with mercury is that it needs a longer timeout than one hour set for CI: https://github.com/Homebrew/homebrew-core/pull/898

Successful test-bot run here locally: https://gist.githubusercontent.com/ilovezfs/b27ef778438f92118360a40a7724c95b/raw/c2ae6dfd158d1e99ebb7430c150b2fa25ca5d632/gistfile1.txt

[ilovezfs]

libcanberra PR updated #871; needs review.

[ilovezfs]

mmix seems to have been fine if anyone wants to review #979

[ilovezfs]

Upstream has now merged a patch for the file-roller bug fixed above: #980

[ilovezfs]

pylucene is a sandbox violation attempting to write to $HOME/.ant: https://gist.github.com/anonymous/68f02ea63c329312993809589777aee7

[UniqMartin]

pylucene is a sandbox violation attempting to write to $HOME/.ant: https://gist.github.com/anonymous/68f02ea63c329312993809589777aee7

I'd say this can be rectified with either ENV.java_cache or fixing up the HOME as seen by Java by using something like ENV["_JAVA_OPTIONS"] = "-Duser.home=#{ENV["HOME"]}" depending on the type of material placed in that path by the build system.

[DomT4]

The former, yeah. It's actually fetching remote modules/etc by the looks of it. Sort of thing ENV.java_cache was designed for this.

[apjanke]

fixing up the HOME as seen by Java by using something like ENV["_JAVA_OPTIONS"] = "-Duser.home=#{ENV["HOME"]}"

Been seeing a good amount of stuff like this lately. Maybe we should start doing this by default as part of superenv. And maybe ENV.java_cache should become the default, too. Are there any cases where we specifically want to _not_ do ENV.java_cache, as opposed to just not bothering to type it in?

[UniqMartin]

Are there any cases where we specifically want to _not_ do ENV.java_cache, as opposed to just not bothering to type it in?

I think the discussion attached to 8280d0f3a0b726d1a74c950f201be4d2af9e7895 shows a case where I intentionally didn't use ENV.java_cache and the reasons for that (if you'd like to incorporate that into your pondering).

[DomT4]

Are there any cases where we specifically want to not do ENV.java_cache, as opposed to just not bothering to type it in?

It still breaks a few things, which is why it's not the default yet. The intention is/was to make it the default as soon as possible, and consequently deprecate having to write it into every applicable formula manually.

[ilovezfs]

The problem in 8280d0f seems to suggest a need for a distinction between ENV.java_cache and ENV.java_tmp maybe?

[apjanke]

Or just a rename, and recognizing that Maven's default caching location is a side effect of setting the user home directory. IIRC, most of the uses of ENV.java_cache I've seen have not actually been about caching anything; I just thought the name was for historical reasons. But if the intent is to deprecate it eventually, no reason to bother.

[apjanke]

Okay, can someone point me to some cases where we want to avoid ENV.java_cache because it actually breaks a build or test, and not because it's a misleading name? (I'm a Java-head so wouldn't mind a couple more items on my TODO list here.)

[DomT4]

IIRC, most of the uses of ENV.java_cache I've seen have not actually been about caching anything; I just thought the name was for historical reasons.

Per my comment on the commit Martin referenced it is actually intended to be used as a cache.

The problem in 8280d0f seems to suggest a need for a distinction between ENV.java_cache and ENV.java_tmp maybe?

This feels a bit contributor-hostile but I could be talked into it.

Okay, can someone point me to some cases where we want to avoid ENV.java_cache because it actually breaks a build or test

I sent you a couple of references on Slack the other day on this. I can try and dig them out but should still be in the chat I'd imagine.

[ilovezfs]

This feels a bit contributor-hostile but I could be talked into it.

Yes, discussing the distinction between cache and temp over and over could get old. The main question is what harm is caused when something that "should" only be temporary is allowed to linger. If the answer is none, then no reason to draw a distinction.

[apjanke]

I sent you a couple of references on Slack the other day on this. I can try and dig them out but should still be in the chat I'd imagine.

Don't worry; I'll hit the scrollback. I must have just spaced out on this. Thanks.

[DomT4]

The main question is what harm is caused when something that "should" only be temporary is allowed to linger.

To be honest, I can see where Martin is coming from in terms of it being silly to cache things like .oracle_jre_usage but I think we're close to over-engineering a solution to a problem that well, isn't much of a problem, IMO. Retaining things like that isn't ideal, but the disk space impact is minuscule and to date we've had zero complaints about it.

[UniqMartin]

To be honest, I can see where Martin is coming from […]

That's great. 😀 To me it's less about about the wasted disk space and more about lingering and hard to debug issues that _might_ be caused by this (not that I'm aware of any), thus about correctness. But since my interest in Java and the number of Java-using formulae I have installed is close to zero, I'll shut up, as I had enough space to present my concerns and ideas.

Let me just finish off by mentioning that https://github.com/Homebrew/homebrew-core/commit/8280d0f3a0b726d1a74c950f201be4d2af9e7895#commitcomment-17277627 presents a solution that I think should to provide both caching and temporary storage to Java-using formulae and, if implemented, requires neither ENV.java_cache nor a not yet existing ENV.java_tmp to be added to formula methods, thus making things more convenient for both contributors and maintainers alike. (See? I did it again. I'm over-engineering things. Le sigh …)

[apjanke]

Yeah, that solution seems to me like the Right Way to handle this.

Since I _do_ have significant interest in Java, I'll pick that up and see if I can make it happen.

[DomT4]

I'm happy to help review proposed changes, but I am wary of making this... complicated. This will stick around permanently, so we need to keep in mind we're promising to look after it permanently, and if one of us gets whacked by a bus the others need to understand what the mechanism is doing and why.

[apjanke]

Seems kinda straightforward: Java doesn't respect the $HOME environment variable, but it does provide a System Properties hook to set its equivalent for all Java runs, and in turn a $_JAVA_OPTS hook to set System Properties, so we should repoint that along with $HOME whenever we do so for sandboxing or superenv purposes. And maybe pre-create the config dirs for well-known Java build tools for those that cannot auto-create their own dirs.

By creating a sandbox environment that can't use the user's actual home directory, I think we've already effectively signed up to do this permanently... :)

[DomT4]

We _do want_ a lot of this to be retained though, and if we point it in the same place as we do $HOME it won't be. $HOME is discarded at the end of every build and test, but we don't want that system for Java.

We also can't (and IMO shouldn't) use the user's own $HOME directory, because Homebrew's writing outside of its prefix was always pretty much an accident that we couldn't prevent, and now we can prevent it we try to do so.

[MikeMcQuaid]

It still breaks a few things, which is why it's not the default yet. The intention is/was to make it the default as soon as possible, and consequently deprecate having to write it into every applicable formula manually.

Seems like a good approach 👍

[ilovezfs]

sdf looks like it's dead and superseded by http://www.rascal-mpl.org/ which we don't have yet. sdf does seem to build fine, though: https://github.com/Homebrew/homebrew-core/pull/1038

[ilovezfs]

sile cannot be bottled presently due to LuaRocks requirements. There is a new version though: https://github.com/Homebrew/homebrew-core/pull/1039

[ilovezfs]

skytools is a sandbox violation:

/bin/sh /usr/local/lib/postgresql/pgxs/src/makefiles/../../config/install-sh -c -d '/usr/local/share/postgresql/contrib'
mkdir: /usr/local/share/postgresql/contrib: Operation not permitted
make[2]: *** [installdirs-old-contrib] Error 1
make[1]: *** [install] Error 2
make: *** [sub-install] Error 2
==> Sandbox log
May 10 03:00:07 sandboxd[137]: mkdir(49045) deny file-write-create /usr/local/share/postgresql/contrib

[ilovezfs]

slony is also a sandbox violation:

/bin/sh /private/tmp/slony-20160510-51357-1nbbfrp/slony1-2.2.4/config/install-sh -c -m 755 slony1_funcs.2.2.4.so /usr/local/lib/postgresql/
cp: /usr/local/lib/postgresql//#inst.52720#: Operation not permitted
make[2]: *** [install] Error 1
make[1]: *** [install] Error 2
make: *** [install] Error 2
==> Sandbox log
May 10 03:02:54 sandboxd[137]: cp(52726) deny file-write-create /usr/local/lib/postgresql/#inst.52720#

[ilovezfs]

This fixes soccerwindow2: https://github.com/Homebrew/homebrew-core/pull/1044

[ilovezfs]

This fixes ssreflect https://github.com/Homebrew/homebrew-core/pull/1166

I figured it might be a good idea to get the ssreflect 1.5/Coq 8.4 series a working formula+bottle before replacing it outright with the mathcomp/ssreflect 1.6/Coq 8.5 new thing. The latter will depend on https://github.com/Homebrew/homebrew-core/pull/1142 being merged first.

[MikeMcQuaid]

Boneyarding a bunch of these in https://github.com/Homebrew/homebrew-core/pull/2194.

[ilovezfs]

@MikeMcQuaid not sure swatchbooker needs to go as @apjanke said "swatchbooker: just needs external dependency pillow," and pillow is vendored in several formulae in core currently so that should be trivial.

[MikeMcQuaid]

@ilovezfs I got it running locally but it has no installations and is a pretty terrible port so I'd rather we just killed it.

[ilovezfs]

@MikeMcQuaid In that case kill it with :fire:

[UniqMartin]

I updated the top comment with a second list of formulae that don't have an El Capitan bottle, but are not necessarily broken. I also included the reason for why they don't have a bottle. Not sure if this should be a separate issue, but I figured since this is all about completing our bottling, it fits. Generated with:

brew for-each-formula 'puts f.full_name if !f.tap? && f.bottle_defined? && !f.bottle_specification.collector.keys.include?(:el_capitan)'

[MikeMcQuaid]

🚢d.

[ilovezfs]

HOMEBREW_SANDBOX=1 for all!

[MikeMcQuaid]

@ilovezfs see https://github.com/Homebrew/brew/issues/713 😉