Spack: Generating potential packages.yaml file?

[BarrySmith]

@tgamblin

./spack compiler find

generates a nice file with a list of compilers that it can use and that users can edit for their needs.

What about

./spack package find

generating a ~/.spack/packages.yaml file containing at a minimum python, mpi, cuda, blas/lapack, cmake, if they exist on the system? The user can then decide by editing if they want to use any of them. This will save the biggest complaint I have with spack wasting hours trying to build the basics (and then having the builds fail which is even more annoying). Users who want spack to build everything, of course, are free to ignore this feature.

[adamjstewart]

We've talked about this before, but this would be very difficult. For every package that you want to support, you need to know:

1. If the package is installed, and where
2. What version it is
3. How it was built

(1) is harder that it sounds. The package name does not always match an executable, and many packages don't come with executables at all.

(2) is also difficult. Not every executable accepts --version, and not every package comes with executables.

(3) is less important, but practically impossible.

At the very least, we would need to add functions to every package.py that we want to support. I imagine we can handle all of the "check /, /usr, /usr/local, and everything in PATH" stuff in Spack's core, which would mean at the bare minimum, something like the python package would need:

def find_installed_path(self, path):
    for executable in ['python', 'python2', 'python3']:
        if os.exists(join_path(path, 'bin', executable)):
            return True
    return False

def find_installed_version(self, path):
    for executable in ['python', 'python2', 'python3']:
        if os.exists(join_path(path, 'bin', executable)):
            python = Executable(join_path(path, 'bin', executable))
            return python('--version)

However, this gets complicated really quickly. What about Python installations that don't include python-devel headers? What about older versions of Python that don't accept --version (2.4 doesn't).

@citibeth has also suggested making the configuration file hierarchy more detailed (linux > linux-centos6-x86_64) and adding system-specific packages.yaml that ship with Spack. This is another good idea worth considering.

[citibeth]

See #2979

@adamjstewart What makes this problem tractable is the relatively small number of packages @BarrySmith wants to find. He's right... if we can do it for compilers, we can do it for NetCDF / MPI / etc. as well. This could slot into the bootstrap stuff of #3057 too.

@BarrySmith Do you care to hack together a script that finds this stuff and generates a packages.yaml? That's the hard part... once we have it, we can try it out on different systems and figure out better how to package / ship it with Spack.

[BarrySmith]

but this would be very difficult

I agree it is not easy (which is why I restricted it to a few packages), but I think it is important. For outdated packages (python 2.4 etc) one could simply put a comment in the generated packages.yaml that stated the package was to outdated to be directory incorporated as in packages.yaml and suggest upgrading. Similarly for packages that refuse to respose to --version etc they could be rejected with a commented message.

Do you care to hack together a script that finds this stuff and generates a packages.yaml?

You got me there :-), hard to resist a challenge :-) If I don't drop dead from the sheer exhaustion of working for DOE I can try to start something, pity only 3.4 percent of my funding covers this type of work :-)

[alalazo]

Should we consider integrating a feature like this into the command introduced in #2507 ?

[adamjstewart]

Yes, I think spack external find would be a good addition. It could find external packages for every package.py that declares how to find it.

[markcmiller86]

I wanna comment on this chain too but I suspect my comments are on-topic only somewhat.

I agree with Barry's criticism's regarding ease-ing or eliminating entirely the need to specify and manage a long and potentially ever growing list of 'external' packages.

My concern, which may or may not be similar to Barry's, is the arbitrarily deep dependency chains Spack is currently managing. For example, having to install pkg-config, m4, autotools, etc. simply to install hdf5 is, IMHO, insane for most HPC software users.

I think only LCF system staff and other similar power users would be interested in following dependency chains down into the bowels of what most of us consider to be "system" level software in that we believe whoever stood up the system, installed the basic set of tools we've all come to expect (things like compilers, standard libraries, linkers, shells, make, autoconf, m4, cmake, etc., etc., etc). I don't wanna have to manage an explicit list of these packages on umpty-umpt different systems and then constantly revise that list as Spack packages grow ever deeper dependency chains which I then have to take action to counteract.

Here's what I want...a way to annotate all packages as whether each "lives" typically above or below the "system library" level of dependencies. Sure, this is a vague definition. But, I'd bet if you asked 100 people who work at the level Barry Smith, or I, or Rob Falguot (developer of Hypre) or Tzanio Kolev (developer of MFEM) or Hank Childs (developer of VisIt), etc., etc. what the "system" level is, they would all pretty much agree that things like m4, autoconf, pkg-config, cmake, etc. are viewed as being "below" the system level and they generally don't wanna be bothered with those dependencies unless they absolutely must (e.g. things simply don't build with out explicitly doing something at that level).

With a simple boolean that annotates each package as being "above" or "at or below" system level, I could then do 'spack install --dependency-depth=above-system-only foobar" (or whatever incantation makes most sense in Spack speak) and have Spack resolve dependencies below that to what has already been installed on that system.

We could make it more complex and assign some arbitrary numerical "depth" to a package as in 0=just above bare metal, 1=about same as compilers, 2=standard/common tools, 3=everything else. I don't really care. What I care about is Spack offering the ability to skip these arbitrarily deep dependency chains as any of us naturally would (by default) do when we go to install some package and simply use the tools we have avialble there (again, resorting to installing them only when we discover we must).

A brief inspection of many package.py files suggests that many of the dependencis I would IMHO consider unproductively deep are "build dependencies", perhaps another way of saying what I want is to disable processing of build dependencies and simply use the tools already there.

Now, how does Spack know the paths to these tools? I am not sure the answer to that. What I do know, is when I log into Vesta or Titan or Edison or Surface (or whaterver your favorite LCF platform is), they are already in my path. If thats possible, I dunno why Spack can't simply use 'em.

IMHO, without a way to easily avert these arbitrarily deep dependency chains, I worry that many users of Spack, like me or Barry and others I have spoken too, will have to resort to maintaining their own set of package.py files apart from Spack just to avoid this issue. That would be shame and would seriously undermind a key goal of Spack...one stop shopping for installation recipes for HPC software stacks.

I don't wanna sound dramatic here. At the same time, I think this issue has been treated with far too little concern for non-system-level installers and non-system-level power uers. Looking at things from the bottom up, I see why dependencies work they way they currently do. Looking at things from the top-down, I am befuddled by and wary of these arbitrarily deep dependencies.

[BarrySmith]

I largely agree with Mark's rant. Python and below should not be built by default; hdf5 and above should be :-). MPI is tricky, if the machine has a "real MPI" that is associated with the cluster switch or batch system then Spack should not install MPI, but if there is just some random MPI installs that does not utilize switch hardware I'd like it to be ignored because it is probably broken.

Perhaps the defaults could be changed with something like

spack default external m4
spack default install python

or some better syntax.

[adamjstewart]

@markcmiller86 I agree with what you're saying. But I don't think it is possible to decide what is "system level" and what isn't.

For example, having to install pkg-config, m4, autotools, etc. simply to install hdf5 is, IMHO, insane for _most_ HPC software users.

I actually tried installing hdf one time and found that it didn't work at all because bison and flex weren't installed on my system. Simple build dependencies like this are _usually_ available, but when they aren't it's a real show stopper. In my opinion, the only thing worse than building dependencies I don't care about is _not_ installing things I need. If spack install doesn't _just work_, we're going to lose a lot more users than we will for those who don't want to install too many dependencies.

Likewise, we've seen a lot of issues lately that boiled down to "the person who packages this assumed that every OS in the world will have zlib installed", but when someone tried to build the same package on an older OS, the system zlib was too old to build the package. This could have been easily avoided by adding a zlib dependency in the first place.

Today I tried installing a package that required openssl. Obviously my system had openssl installed, and I didn't want to manage it with Spack, so I added it as an external package. Much later, I found out that the package I built didn't have HTTPS support because although openssl is installed on my system, openssl-devel is not. If I had just let Spack install things, I would have been fine.

Python and below should not be built by default

That's great, unless you are trying to maintain several Python installations. Basically, everyone has different needs. I don't think we can possibly agree on what should or shouldn't be built. The only safe way to handle things is to add every dependency to a package and let users tell Spack not to build them in packages.yaml. Could this be made easier? Of course. But by default, Spack should build everything. Everything everything.

[BarrySmith]

add every dependency to a package

I totally agree with this, no one, I think, is advocating that any packages just grab random things they find on the system. Managing all dependencies via spack is crucial. But this is slightly different than having spack install everything automatically. Even if it does install everything by default there should be a simple switch to not automatically install low level things. For example on my Mac spack works fine with the python and below on my system but it was a pain as a new user to need to create a packages.yaml manually.

Another thing you seem to be missing is sometimes spack tries to build a low-level package and fails, while there is a perfectly good one already installed and working on the machine; this is enormously frustrating as a user to see spack fail to build something they already have. This will always happen occasionally because no package configure/cmake etc is perfect and if I am trying to install some high level package for my work I don't want to spend time sending bug reports about buggy python installing.

[markcmiller86]

@adamjstewart ... I actually tried installing hdf one time and found that it didn't work at all because bison and flex weren't installed on my system.

Hmm...so because one installer encountered one problem install hdf5 one time on one system, we need to change Spack's install behavior for HDF5 for everyone everywhere? I don't think that is what you are advocating but I honestly don't know. We all have horror stories about hard installations. But, is our goal with Spack to codify bits of logic avert every possible horror story for installing a package on literally every possible system? I don't think so.

Few of comments about HDF5 specifically

• [ ] Looks to me like flex/bison are needed in High Level interface only. Not sure if that helps but if the solution is to make HDF5 depend_on("flex") and depend_on("bison"), it'd really be good to only do that when="+hl"). Or, better yet, disable hl if flex/bison are not available and/or fail the build of hl is explicitly requested and flex/bison aren't available.
• [ ] If it were me and HDF5 installation was pre-empted due to flex and/or bison issues, next thing I'd do is ask The HDF Group about how much they really need to create that dependency? A short look at the code suggests they might be able to code around it and elim. that dependency without too much effort. But, thats a guess and would, nonetheless require an updated HDF5 at that point assuming The HDF Group found it productive to provide it.
• [ ] Spack to the rescue. Looks like spack actually does install flex and bison. So, use it. Use spack to install those and then go back to the HDF5 install that but with the flex/bison you just installed in your path. And, don't modify Spack's recipe for HDF5 at all.

Likewise, we've seen a lot of issues lately that boiled down to "the person who packages this assumed that every OS in the world will have zlib installed", but when someone tried to build the same package on an older OS, the system zlib was too old to build the package. This could have been easily avoided by adding a zlib dependency in the first place.

IMHO, this kind of thing is better addressed by ensuring packages donot* introduce unnecessary dependencies in the first place. Using HDF5 as an example, almost nobody installs HDF5 without zlib. The community as a whole more or less assumes HDF5 requires zlib. But it doesn't. HDF5 builds fine without it and including a depends_on("zlib") in HDF5 is simply wrong. So, here, I think the solution is really to go back to those package files and eliminate the zlib dependency rather than proliferate it. So, I guess I kinda sorta feel like you are helping to make my point for me. Unless I am seriously missing something in what you say.

Today I tried installing a package that required openssl. Obviously my system had openssl installed, and I didn't want to manage it with Spack, so I added it as an external package. Much later, I found out that the package I built didn't have HTTPS support because although openssl is installed on my system, openssl-devel is not. If I had just let Spack install things, I would have been fine.

Sure. It worked there. My concern is not the cases where things work. Its the cases where they don't. The more we proliferate dependencies, the more we create opportunities for things to fail. So, when we use build_visit, we minimize dependencies by default. We build every TPL VisIt needs as bare bones as possible. But, thats looking at things from the top-down...what is the minimum I need to install VisIt?

So, In Spack we've now got a tool that can manage dependencies for us. And, what happens? Everyone who is developing Spack gets comfortable with dependency proliferation because it allows them to resolve yet another issue on yet another (possibly exotic) platform. But, for the rest of us who primarily hope to simply use spack to install software we need, our installations are exposed to added risk of failure not to mention longer build times, bigger install points and almost indefensible explanations to users (Oh, sorry, we couldn't build PETSc for you because Spack coudn't download NodeJs at this time...NodeJs? What the heck is that? I am trying to solve differential equations. Why do I need NodeJs to do that?) due to unnecessary dependencies.

Basically, everyone has different needs. I don't think we can possibly agree on what should or shouldn't be built.

You're right. We can't. But I think comming to agreement on what is considered "system" level software or on some numerical depth ranking of software "levels" is possible. And, IMOH, I think it is somewhat essential to simplify the proliferation of dependencies problem. Otherwise, we're all maintainining our own package.py files.

[tgamblin]

I'm not unsympathetic to the need to minimize extra dependency installations. The whole point of Spack is to allow you to:

1. Easily build many versions of packages and their dependencies;
2. Leverage system versions of libraries and compilers, since we need to do to use vendor tools on many HPC machines;
3. Enable reproducible builds, to the best of our abilities given (1) and (2). This means we need to maintain full build provenance.

(2) pretty much means yes, definitely, we should do this. And if we can do it for "system" dependencies too, then even better.

Arbitrarily picking a level where the "system" begins drastically oversimplifies the problem, and it isn't going to work. If we do that we'll be everyone's favorite broken configure script. I think @adamjstewart already adequately explained this pretty well, but to rehash, you have no idea what's installed on the system and what isn't. Just because the machines you know and love have a package doesn't mean it's also installed on everyone else's.

People use Spack on minimal OS's like Arch Linux, to build containers, on HPC systems, on their macs, and other places, and we do want to encourage those folks to contribute and to build reliably in those environments. I have no control over where people run Spack, and the only way this project is going to be stable long-term is with help from a wide range of contributors. Limiting the scope to the 6 HPC systems we like is a surefire way to lose the rest of the community. It's also a surefire way to be behind the curve when the next HPC system comes along, because we didn't deem its particular environment to be important... until now. I think that is a pitfall that a lot of HPC tools fall into.

Options

I see three options:

1. Reliably detect what's installed on the system and use it;
2. Build things (something I can control and test);
3. Punt and let the build system find things if I think it's a system dependency.

I like (1) the best. It's what the Homebrew guys do very successfully and with lots of testing and CI. Note (with no lack of reverence for what they've accomplished) that they have a simpler problem in at least one respect: they only really have to run on Mac OS.

We, on the other hand, want to run on whatever 4-year old OS IBM or Cray gives us on the front-end node, with CUDA and fancy compilers. If we're going to do (1), we need a more reliable mechanism to detect what's available. It is not even easy to do that when you rely on a stable host OS, and it is probably not a coincidence that it has taken Homebrew 8 years to be competitive, stability-wise, with MacPorts, a system that builds much more of its own stuff (see here).

(2) is my second choice because I can control it and test it. Right now we're working really hard to get CI going for packages, and honestly I think that a more important priority than (1).

(3) is one of the main reasons (if not THE main reason) HPC builds break, in my experience. The stuff you need is never where a build system expects it, it's too old (cough, cough. RHEL), or someone built it with a weird MPI or a weird compiler. @markcmiller86 claimed over email that system versions of things work for him on HPC machines 99% of the time. That's never been the case for me, and I doubt it's been the case for many other people. People have been writing papers about this since at least 2003. They've probably been complaining about it for longer than that.

What to do

So far we've done (2), because it's reliable, we can test it, and (1) is, IMHO, a much harder problem. Witness every horrible FindFoo package in CMake that never actually found Foo for you. To be fair, we have attempted to minimize dependencies by setting a number of variants to be off by default, but we're mostly doing (2).

You can agree with me or not on what we should've done, and whether things fail more one way or another. But I want both (1) and (2). In either case, though, we need to ensure, as @BarrySmith noted, that packages are managed by Spack:

no one, I think, is advocating that any packages just grab random things they find on the system. Managing all dependencies via spack is crucial.

packages.yaml is a stopgap, and I agree it requires way too much bootstrapping that isn't really friendly for your typical user.

I see several reasonable ways to proceed here:

1. Make it easier to add packages, e.g., with #2507.
2. We do distro detection in Spack via the distro package. We can maintain a mapping from system packages to Spack packages for major distributions.
3. In conjunction with (2), we could query the system package manager to see what is installed and what version it's at. It's fairly easy to run things like rpm -qa and rpm -qi <package> to get this information, and I think we could integrate this with concretization pretty easily.
4. Allow packages to have some form of autodetection, like compilers do. Each package would need to implement its own version of this.
5. Do something similar for modules, where we record common module names and what Spack packages they provide.

I think @citibeth did a nice job of discussing the tradeoffs of 2-4 in #2979, so I won't get into that here, but all of these seem like great ideas to me.

Even if we do these things, we are still left with the need to manage the found packages. That means we must start the concretizer with some preconditions (namely, the found packages) and solve for a compatible spec to install, according to the constraints in package.py files, and to fail over to a buildable spec if we can't build with what is installed.

Incidentally, that is what I'm working on right now, along with package testing. So expect to see more focus on these directions once the new concretizer is done. Sorry it's taking so long!

[citibeth]

I think we should pursue 2-4 simultaneously. At the very list, someone can
submit a PR for (4)? You don't have to add a FindFoo() method to every
package, just one or two to start. Then you have a Spack command that
scans all packages for a FindFoo() method, runs them, and generates a
packages.yaml file from it.

We could start with just Python, that would help solve some of our Python
circular dependency issues.

Some notes on the heated voices over the past few days:

• Nobody cares about Python 2.4

• I've seen plenty of cases where system versions are too old and things
  don't build. The most pernicious case here is binutils --- an "old,
  boring" package that, nonetheless, causes some builds to fail if it's not
  new enough.

• Systems where these things fail are not particularly "exotic." Ubuntu
  12.04 is the most common example of a widely-used OS with plenty of old
  versions of things. (In fact, you have to bootstrap a new OpenSSL + Git
  before you can even run Spack).

[adamjstewart]

I agree, we should get started on 4. If someone writes the framework, I would be more than happy to start adding find() methods to the packages I care about.

One hurdle: how do we define whether or not a package is installed? For example, Python works just fine as a build dependency if which python returns something, but unless python-devel is installed you can't really use it as a link dependency. How do we handle partially installed packages?

[markcmiller86]

@tgamblin ...Arbitrarily picking a level where the "system" begins drastically oversimplifies the problem, and it isn't going to work. If we do that we'll be everyone's favorite broken configure script. I think @adamjstewart already adequately explained this pretty well, but to rehash, you have no idea what's installed on the system and what isn't. Just because the machines you know and love have a package doesn't mean it's also installed on everyone else's.

Yes, but the flip side of this statement is that because one person installing one package on one system one time encountered a problem and then "solves" that problem by introducing a new dependency..._everyone_ is hit by that dependency _always_ and _everywhere_ even when its probably rarely an issue.

I think there is a basic sanity test for which the current dependency practice does not pass muster...is Spack behaving much differently than a human would behave if s/he were installing things manually?

The only time I would bother to chase down a deep dependency (like m4) is when I discover its a problem. And, my statement about "...system software working 99% of the time..." is more or less based on experiences such as having manually installed HDF5 on many different systems and never once in my life encountering a need to install m4. But, there Spack is doing it all the time everywhere for me. That behavior doesn't pass a basic sanity check.

Now, you might wanna argue that the time wasted in the rare cases where some poor sole might have to wait and discover a problem with m4 and then go fix it is worse than exposing everyone to an unnecessary m4 dependency. If so, I just wouldn't agree. Why? Again, because in my experience a bad m4 install getting in the way of any autoconf project (not just HDF5) is just not my own personal experience. I am certainly not a power user like the @BarrySmith or @tgamblin and so maybe the problem is I simply don't have the experience to know better.

However, as I think about that more, I am realizing one flaw in my thinking...I am thinking solely from the perspective of installing my favorite package, say HDF5. The _first_ time I try to do that with my own personal copy of a wholly clean checkout of Spack, I will get hit with and see the m4 dependency (and pkg-config, and autotools, and ncurses or whatever). However, thereafter, any other place it comes up in my own personal installs, I won't see it again (assuming the spec's match up). And, if I was using a publicly managed Spack install point on some shared system, I probably would _never_ get _hit_ by that dependency because some other sys-admin had already installed all that stuff (much like they do now manually) and I wouldn't even know/see it.

So, this makes me think that another bias in the way I am thinking about things comes from the perspective of always starting from a _clean slate_...a fresh checkout of Spack where literally nothing has yet been installed. Maybe I need to be thinking of that case like a new _system_ where _/_ is literally empty. In that case, it makes perfect sense that Spack needs to _bootstrap itself_ through all the deep dependencies we don't ordinarily worry about when we get an account on a new system. Hmm...now I seem to be waffling.

Regarding @tgamblin three options... 1) detect whats there, 2) build what you need, 3) punt... while I think I understand the meaning I am wondering why explicit verbiage regarding ability to _trim the dependency tree_ is not also part of the conversation? Isn't that what we are fundamentally talking about?

I understand that the three options are aimed at addressing this but, for me anyway, it just feels like that verbiage missing a key part of the picture. And, sure arbitrarily labeling packages as "system" is not a winning strategy. But, some way to tag/classify depth and branches in the tree sure sounds like a winner to me. To some extent, we already distinguish between a "build" dependency and "link" dependency. So, maybe being able to trim build dependencies from the tree (and use already available versions on this system) is the answer.

Of the three options, I agree 1 is probably best. I understand perfect automation of it is hard too. But, what about some degree of human assistance...less than packages.yaml file but more that pure automation? Perhaps a 'spack --setup' command that goes through a Q&A session with a sys admin? Just spit-ballin solutions here. Or, back to @BarrySmith original inquiry, why not some spack command to help _prime_ packages.yaml

Next, using @adamjstewart example of flex/bison issues with HDF5 installation...is it acceptable to adopt a policy where we do not then make HDF5 directly and permanently _dependent_ on flex/bison but instead allow for some kind of a mechanism/approach that enables that dependency when a failed install suggests its necessary but averts it otherwise?

Finally, in spite of my rants, I really do wanna say I appreciate the thought and effort everyone has already given to this issue. I am certainly far less aware of many of the system issues than others who have commented here. I am sorry if I have sounded like I a broken record or am comming across as overly negative or critical. That is not my intention. Its just the only time I really like extra dependencies is when I am filing my taxes.

[cyrush]

The way I view this is every implicit dependency between me and the set of packages needed is a chance for things to break. (More dependencies more problems) It could be where the tarball was hosted has randomly moved ( zlib ...), it could be that some seemingly innocuous package has never been tested on the platform I am building on, and it's not so innocuous after all.

But this context is important:

• When we need the features, we have to pay the price to resolve any problems. We are used to living in this regime.

• When we don't need the features, I prefer the ability to turn them off. Currently -- that may require a journey to understand why its there in the first place, and some crazy variant forwarding chain.

As @markcmiller86 suggests, there is a fuzzy distance away from absolute all dependencies we want to stay at. But I think its very hard to define that distance.

@markcmiller86 I have had hdf5 builds link to zlib even when passing the --no-zlib option :-(.
I think we had enough horror stories with zlib, to where we had to move it above must-be-built the threshold. (In spack and in build_visit).

I also view CMake as must-be-built, b/c different release CMake vary quite a bit in bugs, and features.

Perl building when CMake was requested was one example of something I clearly below the must-be-built threshold (note: this was resolved)

I think it's really an argument about what acceptable defaults features are enabled. Maybe there is some very very small core that could be agreed on (autotools + m4 ... ) -- but in general, there is no way to identify this across packages -- I think where the line is is really a per a package discussion, b/c it fundamentally involves what package features are enabled. I think Mark and I would prefer a constrained set of features are enabled by default, as a way to reduce what is built.

I am very skeptical of magic to detect everything on the system. Adding more complexity to try to bound complexity is scary. It would be easier for me to debug a build b/c M4 didn't exist, and enable (or add) a +m4 variant, then understand or debug how spack detects M4.

[tgamblin]

@adamjstewart:

One hurdle: how do we define whether or not a package is installed? For example, Python works just fine as a build dependency if which python returns something, but unless python-devel is installed you can't really use it as a link dependency. How do we handle partially installed packages?

The second question there is hard (-ish). I think you should open up a separate PR for it.

[BarrySmith]

Next, using @adamjstewart example of flex/bison issues with HDF5 installation...is it acceptable to adopt a policy where we do not then make HDF5 directly and permanently dependent on flex/bison but instead allow for some kind of a mechanism/approach that enables that dependency when a failed install suggests its necessary but averts it otherwise?

If the build system of HDF5 depends on flex/bison then it depends on flex/bison, you cannot remove this dependency. The question is if/how one tells Spack about this dependency? Looking at hdf5/package.py it looks like it does not tell Spack currently so it must rely on the logic in the hdf5 builder to find them.

I think hdf5/package.py should tell spack about the dependency (the more information spack has about dependencies the better a job it can do). But I also think there should be a spack option that says "find certain standard packages that are already likely installed and add them to the packages.yaml file if they pass certain simple tests (flex would be one such package with a find option). Then I submit most Spack users would want to first run this option before they start installing stuff. This search can use the suggested find option defined for the package. PETSc (with the idea and code stolen from autoconf) can do this find for many packages. As an example, PTScotch actually uses flex so PETSc first searches for flex before building PTScotch as an example of this.

I would rather have Spack do the finding then rely on the underlying package's mechanism for finding stuff (since that is usually really hard to debug and fragile).

[tgamblin]

@markcmiller86:

the flip side of this statement is that because one person installing one package on one system one time encountered a problem and then "solves" that problem by introducing a new dependency..._everyone_ is hit by that dependency _always_ and _everywhere_ even when its probably rarely an issue.

I think I already explained why we went with the dependency over not. One case has the potential to work, the other will always be broken somewhere. And we can test the extra package. I can't test arbitrary machines. The only way around that is having a reliable way to use the system dependency.

I think there is a basic sanity test for which the current dependency practice does not pass muster...is Spack behaving much differently than a human would behave if s/he were installing things manually?

A human wouldn't make a reproducible build. Spack is not a human and I don't think this is a great sanity check. I know at least 10 humans at this lab who would install things 10 different ways.

Why? Again, because in my experience a bad m4 install getting in the way of any autoconf project (not just HDF5) is just not my own personal experience.

Has m4 actually failed to build for you or is this a notional example, like petsc depending on node? FWIW, hdf5 in develop doesn't depend on m4. Are you using an external mvapich2? This is the only path I can see to m4:

hdf5 --[build, link]--> mvapich2 (via mpi) --[build, link]--> bison --[build, run]--> m4

I wonder if that is why you're pulling in m4. Spack may be pulling in bison inadvertently through an external mvapich2. That might just be a bug in the way we concretize externals, because dependencies of externals are supposed to be truncated.

In that case, it makes perfect sense that Spack needs to _bootstrap itself_ through all the deep dependencies we don't ordinarily worry about when we get an account on a new system. Hmm...now I seem to be waffling.

It will become less painful, still, when #2548 is merged.

I am wondering why explicit verbiage regarding ability to _trim the dependency tree_ is not also part of the conversation? Isn't that what we are fundamentally talking about?

My impression, and I must admit I'm a bit lost in all the verbiage, is that you're asking to trim dependencies that are actually required, assume everyone has the system version, and let things break for people who don't. That model won't work. However, if we can find system build dependencies, I believe we can trim the dependency DAG below them and assume that they have what they need if they're system installs. That may be the issue with your pulling in m4 above.

Or, back to @BarrySmith original inquiry, why not some spack command to help _prime_ packages.yaml

Options 2-5 that I proposed above all fit this description -- did you look at them?

[tgamblin]

@cyrush:

I think it's really an argument about what acceptable defaults features are enabled. Maybe there is some very very small core that could be agreed on (autotools + m4 ... ) -- but in general, there is no way to identify this across packages -- I think where the line is is really a per a package discussion, b/c it fundamentally involves what package features are enabled. I think Mark and I would prefer a constrained set of features are enabled by default, as a way to reduce what is built.

FWIW, I don't think we have a strict policy against minimal installs. We generally enable MPI by default, but in most cases I think we've made variants that incur dependencies default to False where possible. I think we can continue to do more of this, but as you say, it's a per-package thing. If you have specific instances like the perl issue where you want a dependency reduced by default, can you and @markcmiller86 file issues about them? I think that would be much more useful than fake examples like "what if PETSc depends on node.js".

I am very skeptical of magic to detect everything on the system.

I am wary of auto-detection to some extent as well, but it has worked fairly well for compilers so far, and I think packages.yaml could work similarly, particularly for build dependencies.

I'm actually reasonably confident that querying the system package manager would work well. @citibeth has already been using her own packages.yaml for CentOS7 that includes things that are already on the system with buildable=False. Querying RPM/apt would just automate that (though there are complications like @adamjstewart's question about split packages above). In any case we'd generate a local packages.yaml saying what was found, and allow it to be overridden.

[cyrush]

@tgamblin, I guess my skepticism is reflected by the fact that we are explicitly setting up the compilers we want for specific platforms :-) -- and not using compiler detection.

I think I already explained why we went with the dependency over not. One case has the potential to work, the other will always be broken somewhere.

Enabling a dependency by default because it failed on one platform doesn't mean that new dependency won't break on other platforms. Especially in the absence of extensive testing.
Can't we opt in on the platform that did have the problem, instead of default opting in everywhere? I think thats @markcmiller86's point about a sanity check. This type of conservative approach is a very pragmatic way to avoid playing wack-a-mole.

Yes -- I will be happy to report more things that might be useful to constrain when we move the a newer spack.

[adamjstewart]

Just to clarify confusion over my HDF example, I was actually talking about HDF4, not HDF5. The HDF4 package cannot be installed without flex and bison, this is a hard requirement. If the executables aren't in the PATH, then configure immediately quits. So there's really no way to work around this one. It has to declare its dependencies.

[tgamblin]

@cyrush:

Enabling a dependency by default because it failed on one platform doesn't mean that new dependency won't break on other platforms. Especially in the absence of extensive testing.
Can't we opt in on the platform that did have the problem, instead of default opting in everywhere?

In general when we add a dependency, it's because the package actually depends on the thing we added. One of the goals with Spack is to have full build provenance, and if we don't add the dependency, we're putting ourselves in a position where we're always at the mercy of the host system, and we can't make reproducible builds. If I have the dependency, I can at least manage it, and I at least have a chance at externalizing it later, as @BarrySmith has pointed out.

I agree that the lack of testing is an issue, but spack.io/cdash is happening, and I am really hoping to have per-PR build testing in the cloud and on HPC machines soon. We're also adding things like #2548, which I think will significantly reduce the number of build failures seen for the common dependencies by always building them with the default compiler. I don't really see another reasonable way to guarantee stability for 1,400 packages.

Finally, I'll point out that we'll have binary installs in the near future, which may further reduce the pain associated with building extra stuff.

[adamjstewart]

@cyrush Just to clarify, are we talking about optional or required dependencies here? Because if a dependency is required, there's really no way to get around that. You have to build the dependency, either with Spack or with your system package manager. To be fair, your system package manager doesn't care about what is already installed by other package managers either. The only difference is that rpm installations are generally much faster than building from source. There are some people working on binary installations for Spack, but in the meantime we have to install from source. If you want to use system packages, you're more than welcome to. But removing required dependencies from a package will almost certainly break the package for other users. Spack is a package manager, managing and installing dependencies is what it does best. Let's let it do that, and if you don't want to build something that is already installed you can tell it to do that. I think a way to query system package installations would help greatly with that. But a lot of Spack users are building executables that need to run on both login and compute nodes, and our compute nodes certainly don't have as many libraries installed as our login nodes. I personally err on the side of caution and let Spack build all library dependencies, while using the system packages for build-only dependencies.

[cyrush]

@adamjstewart really all I am trying to bring up is that adding dependencies also creates failure modes.

I would rather turn more knobs on to build more implicit dependencies on platforms that don't provide them, than build them everywhere because things failed somewhere. It's analogous to doing platform-specific patches. Or changing the defaults for package on a specific platform (like with python on OSX) In general, in the absence of extensive testing, I am more comfortable with surgical changes.

Testing is the real solution, and I know folks are working hard to get there.

[adamjstewart]

I understand your point, and I've certainly run into this problem myself. I can't built Perl with Intel, and I can't build pkg-config, openssl, and dozens of other dependencies with PGI (although most developers have been pretty good about providing me with patches to get PGI installs working). But nevertheless, the package still depends on these things, whether I build them with Spack (with another compiler) or use a system installation. If Spack can't install something, you can always add it to packages.yaml. But if a package doesn't declare a dependency, it might not be obvious to a user why the installation failed or how to fix it. We want the same thing here: to use a system installation instead of building it with Spack. But we're going about it very differently. What we're suggesting is an easier way to tell Spack to use system dependencies. What you're suggesting is analogous to pretending the package doesn't have these dependencies and letting it use whatever may (or may not) be in the PATH. Implicit dependencies worry me, and offer no way to ensure a reliable build that works for everyone.

[markcmiller86]

A human wouldn't make a reproducible build. Spack is not a human and I don't think this is a great sanity check. I know at least 10 humans at this lab who would install things 10 different ways.

Fair enough. But, likewise, do you know any humans who have already, in their experience, taken to _always_ installing mawk, perl, pkg-config, ncurses, zlib and openssl just to install cmake? Those are the dependencies @cyrush got hit with for a recent cmake installation.

Has m4 actually failed to build for you or is this a notional example, like petsc depending on node? FWIW, hdf5 in develop doesn't depend on m4. Are you using an external mvapich2? This is the only path I can see to m4:

Sorry. Yes, notional example. So was NodeJS for PETSc. But, that wasn't entirely pulled from my rear end. I was thinking about packages that include HTML documentation build as a default part of a _make install_. So, you can imagine Jekyll...liquid tags, NodeJS, Ruby/Gems, etc., etc.

...is that you're asking to trim dependencies that are actually required, assume everyone has the system version, and let things break for people who don't. That model won't work.

Nope. I am not asking for the ability to trim away required dependencies. At least I don't think I am. I am asking for the abiliity for Spack to trim _building_ required dependencies and just _use_ already available _installations_ (call them _system_, call them _external_, call them _pre-installed_, whatever)...and yet still keep track of the dependency provenance as @BarrySmith emphasizes.

Options 2-5 that I proposed above all fit this description -- did you look at them?

I did now. I like 4. But, then, I think that only a certain class of package would really warrant implementing a def findme(): method...those that are more or less treated as _system_. Thats a vague definition and a moving target. For example, prior to 1998, HDF5 didn't even exist...Now its actually part of many standard system installations (and because of that often winds up getting in the way of builds of HDF5 dependent packages picking up an hdf5.h from /usr/include rather than an HDF5 I just installed). When/how would the findme method get triggered in place of the standard package install pipeline? Maybe package.yaml has packages with preferences to first fire _findme_ and only resort to install if _findme_ fails? Anyways, I like this idea. Its probably represents a lot of work but I am not sure I see any other choice.

FWIW, I don't think we have a strict policy against minimal installs. We generally enable MPI by default, but in most cases I think we've made variants that incur dependencies default to False where possible. I think we can continue to do more of this, but as you say, it's a per-package thing.

Hmm. I like this notion of _variants that incur dependencies_. That seems like a useful way to classify a variant. That would mean any package where we have variant('+foo'...) and a depends_on('bar', when='+foo'), we'd want the variant to be _off_ by default, right? That makes sense. Any way to check how well the 1,400 packages currently obey this practice? Any way to maybe encourage developers to do that?

[citibeth]

I agree, we should get started on 4. If someone writes the framework, I
would be more than happy to start adding find() methods to the packages I
care about.

One hurdle: how do we define whether or not a package is installed? For
example, Python works just fine as a build dependency if which python
returns something, but unless python-devel is installed you can't really
use it as a link dependency. How do we handle partially installed packages?

I think we would need the find() method return information on whether it
thinks the installed package will work for a run dep, link dep or both.

[citibeth]

I think there is a basic sanity test for which the current dependency
practice does not pass muster...is Spack behaving much differently than a
human would behave if s/he were installing things manually?

Real human beings are really bad at concretizing large DAGs, and typically
end up with mistakes (different versions of the same dependency linked
twice), which cause unending grief. We WANT Spack to behave differently
from humans.

I think the better sanity test is... does it work? Spack is an
auto-builder; building an extra package here or there is not a big problem,
especially if the packages are low-level and quick to build.

Fair enough. But, likewise, do you know any humans who have already, in
their experience, taken to always installing mawk, perl, pkg-config,
ncurses, zlib and openssl just to install cmake? Those are the dependencies
@cyrush got hit with for a recent cmake installation.

Spack can install all that in the time it takes a real human to install one
or two of the packages. But to be fair... I add perl to packages.yaml
because it's just a run dependency and system Perl is fine; and I add
openssl because I don't want to be responsible for keeping
security-critical software up-to-date.

The only time I would bother to chase down a deep dependency (like m4) is
when I discover its a problem.

So is Spack going to parse and interpret compiler errors to discover that
the system m4 is a problem?

Now, you might wanna argue that the time wasted in the rare cases where
some poor sole might have to wait and discover a problem with m4 and then
go fix it is worse than exposing everyone to an unnecessary m4 dependency.

Actually, what happens is new users pick up Spack, find it doesn't install
HDF5 for them, and give up on it. If we include everything, they can later
decide it's installing too much and pare things down to their liking.

Look at it this way... although we all agree that manually setting up
packages.yaml is sub-optimal... if you really care about it, adding m4
to that file is pretty easy.

Another issue is, at least some of us don't have root access. Maybe my
CentOS m4 would do just fine. But if it's not already installed, it's
easier for me to type spack install m4 than to beg our sysadmins to
install in through yum. And I don't really care, becuase m4 is quick
and easy to build.

Same for zlib, which is as easy to build as anything out there, the whole
process takes under a minute, it has no other dependencies, etc. I just
don't see the problem with building zlib.

Where I DO care is with the 200 X11 packages now in Spack. My system has
them, and I don't want to build them myself. I have strategically added
certain packages to my packages.yaml to prevent X11 builds for the
packages I care about. Typcially, I will inspect the concretization of a
large build before telling Spack to go at it. spack spec is my friend...

I also care in cases where Spack packages don't build. [email protected] failed
to build for a while on my system (I think it's been fixed, but maybe not),
so I had a keen interest in making sure Spack used my system-installed
Python 2.7 for build dependencies. I don't do any "real" Python work with
that, as my "real" Python work is with Python3 --- which I want Spack to
build completely, so I have good control over it.

Regarding @tgamblin three options... 1) detect whats there, 2) build what
you need, 3) punt... while I think I understand the meaning I am wondering
why explicit verbiage regarding ability to trim the dependency tree is not
also part of the conversation? Isn't that what we are fundamentally talking
about?

Adding a record to packages.yaml is cleaner but otherwise sort of
equivalent to removing a dependency (say m4) that you "think" should be
available on the system.

As for auto-detection... in the case of CMake, it has been decided that
packages should NOT change their configuration based on what they find on
your system. Instead, you should tell them what features you want, and if
your system doesn't have it they should crash. I think this is the right
approach; and it's in contrast to many Autotools scripts that do the
former. I believe we should follow the CMake wisdom with Spack.

I understand that the three options are aimed at addressing this but, for
me anyway, it just feels like that verbiage missing a key part of the
picture. And, sure arbitrarily labeling packages as "system" is not a
winning strategy. But, some way to tag/classify depth and branches in the
tree sure sounds like a winner to me.

Why doesn't somebody do a dependency analysis of packages in our existing
repo? How many times does each package occur as a dependency (directly or
transitively)? That type of automated analysis could very well provide a
good sense of what "system" packages are.

Next, using @adamjstewart example of flex/bison issues with HDF5
installation...is it acceptable to adopt a policy where we do not then make
HDF5 directly and permanently dependent on flex/bison but instead allow for
some kind of a mechanism/approach that enables that dependency when a
failed install suggests its necessary but averts it otherwise?

I don't see how Spack could know that such-and-such obscure build error
means flex/bison is missing. More often than not, I discover the answer to
these kinds of problems by throwing my build errors into Google.

Its just the only time I really like extra dependencies is when I am
filing my taxes.

Keep in mind that Spack is building those extra dependencies, not you. As
long as it's not too extreme, you're still saving time. But if saving disk
spack or inodes is important (and it is in some cases), you can engineer
away extra dependencies.

When we don't need the features, I prefer the ability to turn them off.
Currently -- that may require a journey to understand why its there in the
first place, and some crazy variant forwarding chain.

I have consistenlty advocated against variant forwarding, for exactly this
reason. With the ability to set variants in packages.yaml, as well as
global variants, forwarded variants are almost NEVER the right approach.
I'd love to see them eliminated from Spack.

I've also advocated for other principles along similar lines: (a) The
+doc variant should always default to False, (b) Turn off "extras" not
needed to make the package work, (c) turn off extra language bindings, etc.

I was thinking about packages that include HTML documentation build as a
default part of a make install.

Do you know a specific package that builds docs in make install, and that
cannot be turned off? We do everything we can to turn off doc-building by
default. Remember that default Spack variants don't have to be the same as
defaults for the upstream package.

I also view CMake as must-be-built, b/c different release CMake vary
quite a bit in bugs, and features.

In fact, projects specify which version of CMake they work with. If the
CMake you're using is too old, it won't even run on a newer CMake-based
project. In practice, that means that the latest CMake must be built when
you're running on any real (i.e. released and installed) OS.

I am not asking for the ability to trim away required dependencies. At
least I don't think I am. I am asking for the abiliity for Spack to trim
building required dependencies and just use already available installations
(call them system, call them external, call them pre-installed,
whatever)...and yet still keep track of the dependency provenance

Yes, Spack already has that functionality with packages.yaml. Given 20
minutes of your time, you can use spack spec to determine which
dependencies you don't really want to build, and add them to your
packages.yaml. This is far less time than it takes you to build
manually. Maybe sometime in the future, that process will be more
automated.

When/how would the findme method get triggered in place of the standard
package install pipeline?

You would run a spack findme kind of command, and it would generate a
packages.yaml. You then delete the entires you don't want and use the
resulting packages.yaml for your builds.

Its probably represents a lot of work but I am not sure I see any other
choice.

I don't think it's very much work. Or at least, the work is proportional
to the number of packages with a findme() method. Building the basic
machinery and demonstrating for a single package would not be hard.

Hmm. I like this notion of variants that incur dependencies. That seems
like a useful way to classify a variant. That would mean any package where
we have variant('+foo'...) and a depends_on('bar', when='+foo'), we'd want
the variant to be off by default, right? That makes sense. Any way to check
how well the 1,400 packages currently obey this practice?

Yes. Load up all 1000+ packages in Python (which Spack does already), and
then search through the resulting Package data structures based on the
variants and conditional dependencies.

Any way to maybe encourage developers to do that?

Yes. Package up your checker and submit to Spack as an additional Travis
check. Then it will flag any new default=True variants that incur a
dependency.

[tgamblin]

I think GitHub is really bad at parsing quoting from email. But I cleaned up @citibeth's comments for consistent quoting and posterity.

[tgamblin]

@adamjstewart @citibeth: if you want to take a stab at a generic find framework for build dependencies (i.e., things that are called like commands), you could model it off the one in spack/compiler.py:

1. Each compiler class (in spack/compilers) subclasses spack.compiler.Compiler and defines regexes for executable names as well as possible prefixes and suffixes.
2. It also provides a method (default_version, which is called by cc_version, cxx_version, etc.) to get the version of a compiler, given an executable.
3. spack compiler find calls Compiler._find_matches_in_path, which searches your path for executables that match any of the possible (prefix X name X suffix) combinations.
4. All the command invocations (which can take quite a while if you run them sequentially) are farmed out using spack.util.multiproc.parmap.

For compilers, it does some grouping to get the different language compilers in 4-tuples with the same version and most similar possible name format (e.g., we prefer to pair gcc-5.5 with gfortran-5.5), but that's about it.

That gets us a compiler and a version, and if you repurpose it to work with Package instead of Compiler, it's enough to get you a minimal spec for use in packages.yaml, along with the path, which you know from the search. You can set the arch to the default platform-os-target. Spack marks external specs concrete by fiat. That is, even if they don't have all metadata filled in, they're still considered concrete because they're installed and the provided spec is all we know about them. When external specs are included in Spack builds, we truncate them and assume the system or the user knows how to find their dependencies; we don't do anything special to find dependencies.

I think this would at least be a start. It doesn't handle libraries (which is harder) but we likely don't need to find many dependency libraries since we prune most external specs. I think this would handle most of the common build dependencies. It also gets you a version, so if a package depends_on('cmake@3:'), and this found [email protected], Spack could still know to build a new cmake b/c the system dep won't work.

The interface should probably be similar to the interface we're using for compilers -- Packages could have attributes for executable names, prefixes, and suffixes, and a version detect method (eg. get_version(self, executable)). I think you can reuse the parallel spawn stuff from Compiler.

I think we could also repurpose this to do something I've wanted to do for a while -- we could add a spec parameter to which in the various places it's used in Spack. This:

which('curl', spec='curl')

Could mean to find curl, but build the curl package and try again if curl isn't available. That would allow Spack to lazily bootstrap the various implicit fetch dependencies (curl, git,svn,hg`, etc.) if it needs to.

This will not handle variants, but it won't link against them erroneously either. If you depend on cmake+something, Spack will be forced to build that because a concrete spec without a variant can never satisfy either +variant or ~variant. You could extend this approach to do per-variant checks later. I consider that a harder and less frequently necessary problem.

Finally, we need to do this anyway to unify the compiler support and packages like we've been talking about. It'll be a step towards making compilers first-class package citizens.

[citibeth]

if you want to take a stab at a generic find framework for build dependencies (i.e., things that are called like commands), you could model it off the one in spack/compiler.py

This will be needed if we want to one day turn compilers into build dependencies. So definitely, I like this proposal, in that it makes existing techniques more general.

I think we could also repurpose this to do something I've wanted to do for a while -- we could add a spec parameter to which in the various places it's used in Spack. This:

which('curl', spec='curl')

I really like this idea of auto-bootstrapping as well. We've thrown around bootstrapping ideas before, and this would work better than anything we've thought of so far. However, it needs a version associated with it. Most frequently, git exists but it's too ancient. So we need: which('[email protected]:', spec='git')

The other problem is that usually, curl is not the problem; but actually openssl. So we would need which('curl^[email protected]:', spec='curl'). Do you think that would be possible/easy?

[adamjstewart]

if you want to take a stab at a generic find framework for build dependencies (i.e., things that are called like commands), you could model it off the one in spack/compiler.py

Let's wait until #2507 is merged first. That will provide a basic interface for modifying packages.yaml:

$ spack external add <spec> <path>
$ spack external add <spec> <module>
$ spack external rm <spec>
$ spack external list

It won't be hard to add a spack external find command to that interface. Like you said, most of the work will be implementing package-specific find() methods.

[cyrush]

@adamjstewart @tgamblin It sounds like need to master using packages.yaml files. For our projects, I am happy to craft these by hand for the platforms we are building on.

[svenevs]

most of the work will be implementing package-specific find() methods

I'm curious...what's the python game gonna look like here? Are py-X packages going to be allowed to be external? If not, what happens when a user has an "external" python but wants to spack install a specific library?

[citibeth]

That is no problem, since libraries all install in their own root. And you
can use it all together by loading the appropriate set of modules. You
won't be able to activate spack-installed libraries into external python.
But upcoming spack environments might allow you to link both into a single
view.
On May 6, 2017 6:58 AM, "Stephen McDowell" [e-mail] wrote:

most of the work will be implementing package-specific find() methods

I'm curious...what's the python game gonna look like here? Are py-X
packages going to be allowed to be external? If not, what happens when a
user has an "external" python but wants to spack install a specific library?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LLNL/spack/issues/3181#issuecomment-299632017, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AB1cdwuJzOjNndqg5UYzfupRh_gGP4Etks5r3FJNgaJpZM4ME538
.

[svenevs]

This is great news! But for example, say I have specified an external py-numpy for some reason, and then build py-matplotlib (or anything else that uses py-numpy). Then I go and remove my external py-numpy (either intentionally or not). What should happen? I guess this holds more generally for any external runtime dependencies. Are things going to be marked as "Y was installed with package X which was provided externally", and then when I try and spack load Y, are there going to be any checks in place?

The concern here is that say I'm using my system package manager (dnf). If I'm not paying very close attention, an upgrade or remove may change the external dependency without me really being conscious of what happened. It seems like it might be a good idea to have spack load verify things are all still in place using the file that specified them in the first place.

On the other hand, if it is too picky (say my system py-numpy updated a minor version but is for all intensive purposes the same thing, same for cmake or pkg-config), then these "sanity checks" basically put us back to square 1 where I'm battling spack to let it know it's "ok".

[amklinv]

I am also interested in this feature. Has there been any progress toward this?

[alalazo]

Looking for other things I found this old issue. Just want to leave a note that either #15158 solved this or #16526 will probably do. Comments on #16526 and on the procedure it adds to make a package discoverable using:

$ spack external find ...

are more than welcome.

[adamjstewart]

I vote we close this issue now that #15158 has been merged.

[alalazo]

Closing following @adamjstewart comment