Homebrew-core: GASNet patching help

[zbeekman]

GASNet's default means of helping the user build code is for the user to include stub makefiles depending on what conduit they want to use. One problem, is that the linker and compilers that get written in these makefiles are like /usr/local/homebrew/Library/ENV/4.3/clang which don't work outside of the homebrew installation environment.

My questions are as follows:

1. How does one determine the path to the real compiler and linker being used during the homebrew installation?
2. It is not known _a priori_ which conduits are available on a system, so it is not known how many/which makefile includes/stubs will need patching. Is there a way to include patches for all possible files that may need patching, but not fail if there is no file present to patch?

[UniqMartin]

Sorry for leaving this unanswered for so long. I guess it's somewhat irrelevant now that gasnet has been boneyarded in #1368.

How does one determine the path to the real compiler and linker being used during the homebrew installation?

That's essentially unsupported. I think the way we tend to deal with these rare cases is to patch the affected files once installation finishes to use just the binary, e.g. clang, instead of the full path. (The full path isn't that helpful anyway, as CLT and/or Xcode is probably going to be updated after the formula gets installed, which changes the Clang version but not the location of the binary in the file system.)

It is not known _a priori_ which conduits are available on a system, so it is not known how many/which makefile includes/stubs will need patching. Is there a way to include patches for all possible files that may need patching, but not fail if there is no file present to patch?

The way we tend to deal with this is to use inreplace instead of patching, possibly wrapped in a Dir["<glob>"].each do |file| … end block to apply the same replacement to a bunch of related (and similar) files selected by the shell globbing pattern <glob>.

[zbeekman]

Sorry for leaving this unanswered for so long.

No worries and thanks for getting back to me @UniqMartin

I guess it's somewhat irrelevant now that gasnet has been bone yarded in #1368.

Well, GASNet is still actively developed and it would be nice to get it into a homebrew tap if I can run down these issues. Perhaps it belongs somewhere else, in homebrew-science perhaps, rather than core... what do you think?

How does one determine the path to the real compiler and linker being used during the homebrew installation?
That's essentially unsupported. I think the way we tend to deal with these rare cases is to patch the affected files once installation finishes to use just the binary, e.g. clang, instead of the full path. (The full path isn't that helpful anyway, as CLT and/or Xcode is probably going to be updated after the formula gets installed, which changes the Clang version but not the location of the binary in the file system.)

Yes that makes sense. So it seems like I can just use ENV.cc etc. as the replacement text... the somewhat tricky part will be determining where the replacements need to occur and how much needs to be replaced. Perhaps I could look for full paths ending in #{ENV.cc} and strip the leading directories?

It is not known a priori which conduits are available on a system, so it is not known how many/which makefile includes/stubs will need patching. Is there a way to include patches for all possible files that may need patching, but not fail if there is no file present to patch?
The way we tend to deal with this is to use in replace instead of patching, possibly wrapped in a Dir[""].each do |file| … end block to apply the same replacement to a bunch of related (and similar) files selected by the shell globbing pattern .

Great, thanks

[UniqMartin]

Well, GASNet is still actively developed and it would be nice to get it into a homebrew tap if I can run down these issues. Perhaps it belongs somewhere else, in homebrew-science perhaps, rather than core... what do you think?

I'm not at all familiar with it, but I suspect it's mostly used in a scientific context, so the homebrew/science tap sounds like a good fit initially. On the other hand and judging from its description on the homepage, it's fairly generic and provides services similar to MPI implementations, that we continue to host in homebrew/core. If you can revive it at some later stage, I'd be tempted to put it where it came from, as I don't see a strong argument why it shouldn't be in homebrew/core.

Maybe the only thing that feels a bit troubling to me is that none of the formulae in Homebrew, not even any from homebrew/science depend (even if only optionally) on it. That makes me wonder if it's not too niche to be included in one of the official taps.

Perhaps I could look for full paths ending in #{ENV.cc} and strip the leading directories?

Yep, something like that, or maybe even a match for paths starting with #{HOMEBREW_LIBRARY}/ENV. The replacement can get pretty sophisticated, if need be. Ideal, of course, would be to have some upstream support for not hard-coding the paths discovered during the configuration and installation of the package.

[zbeekman]

I'm not at all familiar with it, but I suspect it's mostly used in a scientific context, so the homebrew/science tap sounds like a good fit initially. On the other hand and judging from its description on the homepage, it's fairly generic and provides services similar to MPI implementations, that we continue to host in homebrew/core. If you can revive it at some later stage, I'd be tempted to put it where it came from, as I don't see a strong argument why it shouldn't be in homebrew/core.

Yes, it's a very generic library that is mostly maintained for UPC, but other PGAS languages use it (OpenCoarrays, titanium? I can't remember off the top of my head but I believe there are a bunch more.) It is able to leverage a number of different "conduits" and will probably be faster than competing MPI implementations on most Macs... plus a lot of HPC users (scientists & engineers) have Macs and it is helpful to be able to test things locally before trying to run in a HPC environment, or to implement CI across multiple OSes for their codes.

Maybe the only thing that feels a bit troubling to me is that none of the formulae in Homebrew, not even any from homebrew/science depend (even if only optionally) on it. That makes me wonder if it's not too niche to be included in one of the official taps.

Well, to be fair, it hasn't been in homebrew that long, it isn't necessarily going to be consumed by other homebrew packages if scientists and engineers are using it to help them develop code locally before moving to an HPC environment, and it more or less has never worked correctly as installed by homebrew... (my fault for not catching the absolute paths being embedded in the makefile includes) I know at least one other user (@jeffhammond of Intel) has expressed interest in having it in Homebrew.

Perhaps I could look for full paths ending in #{ENV.cc} and strip the leading directories?
Yep, something like that, or maybe even a match for paths starting with #{HOMEBREW_LIBRARY}/ENV. The replacement can get pretty sophisticated, if need be. Ideal, of course, would be to have some upstream support for not hard-coding the paths discovered during the configuration and installation of the package.

It would be great if variables like HOMEBREW_LIBRARY were documented in the cookbook. This is exactly what I was looking for thanks for mentioning it!

[UniqMartin]

Fair. I was only drawing my conclusions from the (very little) research I've done on that package. I have sympathy for the use cases you describe and agree that having the library without an (obvious) consumer is still a useful thing to have in Homebrew.

It would be great if variables like HOMEBREW_LIBRARY were documented in the cookbook. This is exactly what I was looking for thanks for mentioning it!

I might get some flak for even suggesting it, as everything that's not documented yet is free to change arbitrarily. :wink: This might change a bit once we reach 1.0.0, but we aren't there yet.

A safer bet would probably be to rely on #{HOMEBREW_REPOSITORY}/Library/ENV as that gives you the same path and HOMEBREW_REPOSITORY is already used in a few formulae. (In fact there's already this snippet in the colorsvn formula that is somewhat related to your use case.)

[jeffhammond]

@zbeekman GASNet isn't really a competitor to MPI. It implements different features. It may be true that Put and Get with GASNet will be faster than MPI under some conditions, but in many other respects, they can't be compared (collectives and active-messages are supported only by MPI and GASNet, respectively).

In any case, GASNet is high-quality software that is meticulously maintained by the folks at Lawrence Berkeley National Laboratory. They do extensive QA, although perhaps not on Mac. I've been testing all of the relevant GASNet conduits via Travis CI in the PRK project and there is one issue, but not in the recommend conduit.

As for the clients of GASNet, they include Berkeley UPC and UPC++, Intrepid's GCC UPC and Clang UPC, Chapel, the OpenSHMEM reference implementation, OpenCoarrays, Stanford's Legion, and others.

A few folks who might want to +1 this issue include: @bradcray, @elliottslaughter, @magnatelee, @sidjana, in case it makes life easier on their projects.

[zbeekman]

@zbeekman GASNet isn't really a competitor to MPI. It implements different features. It may be true that Put and Get with GASNet will be faster than MPI under some conditions, but in many other respects, they can't be compared (collectives and active-messages are supported only by MPI and GASNet, respectively).

Yes, I realize they are very different and can't be directly compared... poor phrasing and over simplification on my part.

In any case, GASNet is high-quality software that is meticulously maintained by the folks at Lawrence Berkeley National Laboratory. They do extensive QA, although perhaps not on Mac. I've been testing all of the relevant GASNet conduits via Travis CI in the PRK project and there is one issue, but not in the recommend conduit.

Which conduit/OS combos have you tested, and is the problematic one the MPI conduit?

As for the clients of GASNet, they include Berkeley UPC and UPC++, Intrepid's GCC UPC and Clang UPC, Chapel, the OpenSHMEM reference implementation, OpenCoarrays, Stanford's Legion, and others.

A few folks who might want to +1 this issue include: +bradcray, +elliottslaughter, +magnatelee, +sidjana, in case it makes life easier on their projects.

Great, thanks for weighing in here. You may want to @ mention them (instead of +)

[zbeekman]

@UniqMartin

What do you think about this at the end of the install block:

    Dir["#{include}/*.mak","#{include}/*-conduit/*.mak"].each do |file|
      inreplace file do |s|
        s.gsub! %r{#{Regexp.escape(HOMEBREW_LIBRARY)}/ENV/[0-9]+(\.[0-9]+)*/#{Regexp.escape(ENV.cxx)}}, ENV.cxx
        s.gsub! %r{#{Regexp.escape(HOMEBREW_LIBRARY)}/ENV/[0-9]+(\.[0-9]+)*/#{Regexp.escape(ENV.cc)}}, ENV.cc
      end
    end

Note that the order is important here, because sometimes ENV.cc is a substring of ENV.cxx (like for clang) so if you do it in the other order inreplace will fails because the ENV.cc line will match the ENV.cxx lines as well for clang and possibly some other compilers.

It seems like these makefiles get written during configure using automake, but I don't think there's an easy way to do this via patching or at configure time.

[elliottslaughter]

Sorry in advance if I'm jumping in without understanding context, but why can't you just set CC and CXX on configure? This is roughly what we do in our install script for Legion:

CC=... CXX=...   ./configure ...

From https://github.com/StanfordLegion/gasnet/blob/master/Makefile

[zbeekman]

Sorry in advance if I'm jumping in without understanding context, but why can't you just set CC and CXX on configure? This is roughly what we do in our install script for Legion:

CC=... CXX=... ./configure ...
From https://github.com/StanfordLegion/gasnet/blob/master/Makefile

@elliottslaughter Perhaps it was a bit of a context lacking leap, but I don't mind and I'll do my best to explain... with the caveat that my Ruby is limited and I am a Homebrew contributor but not a homebrew maintainer, so some of my explanations may be a bit imprecise.

Setting CC and CXX is effectively what homebrew does. However, during the Homebrew build process a special sandbox environment is setup (the "superenv"). CC and CXX are set to clang and clang++ respectively but due to the special sandbox/environment clang resolves to a special shim script in /usr/local/homebrew/Library/ENV/3.4/clang which is only functional when this sandbox is activated by the homebrew build process. Everything seems to build and install fine, except that the full paths to the shim scripts clang and clang++ are embedded the makefile snippets that users are meant to include in their makefiles. As a result, when a user includes the GASNet provided conduit specific makefile the compiler and/or linker resolves to /usr/local/homebrew/Library/ENV/4.3/clang etc. which are non-functional outside of the homebrew install environment...

TL;DR homebrew installation seems to be fine, except the makefile includes in the include directory and the conduit-specific include directories which have full paths to the wrong compiler/linker due to Homebrew magic 🎩 🐇

[UniqMartin]

What do you think about this at the end of the install block:

This looks about right.

Sorry in advance if I'm jumping in without understanding context, but why can't you just set CC and CXX on configure?

That's effectively what we do. Prior to invoking ./configure we export CC=clang and CXX=clang++ to the environment. But apparently this isn't used (and written to the conduit Makefiles) unaltered and instead gets resolved to the absolute path. (@zbeekman might be able to comment on this in more detail, as I have literally zero practical experience with GASNet and trying to build it on OS X.)

[zbeekman]

@UniqMartin yes that is what is happening, I'm guessing a Makefile function like realpath or abspath might be the culprit here but I have not dug around enough to say that with absolute confidence.

[jeffhammond]

@zbeekman Correct. It was the MPI conduit that didn't work with BUPC on Mac. I noted this in https://github.com/ParRes/Kernels/blob/master/.travis.yml#L66, so it must be true 😄

[zbeekman]

@jeffhammond just curious if you ever followed up with anyone else about
this. Also, how recently did you test this. There was a new GASNet release
earlier this month.
On Wed, May 25, 2016 at 10:04 PM Jeff Hammond [email protected]
wrote:

@zbeekman https://github.com/zbeekman Correct. It was the MPI conduit
that didn't work with BUPC on Mac. I noted this in
https://github.com/ParRes/Kernels/blob/master/.travis.yml#L66, so it must
be true 😄

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/Homebrew/homebrew-core/issues/1264#issuecomment-221758803

[jeffhammond]

@zbeekman I will reenable the Travis tests, but I don't care that much, because I know the MPI conduit is a means of last resort for GASNet, especially on a single-node machine, which most Mac systems are.