Spack: Easy installation of a git repo with a package.py in the root

[DavidPoliakoff]

Hey @citibeth @trws @adamjstewart

We were talking about this yesterday (#1108) . Bower lets you do the following

bower install [github link]

It pulls a package.json out of the given repository and installs the dependencies and the package itself. This leads to a turnaround from "I see a project that interests me" to "I have this package installed for my project" on the order of five minutes. The spack equivalent is to... spack edit -f [desiredPackageName] and copy and paste from the github repo? In the absence of this feature, I'll be polishing up a bash script for this purpose (do not run this under any circumstances, it hasn't been vetted and I can think of ways to make it delete things you wouldn't want deleted)

#!/bin/env bash
echo $tarball
curl -L $1/tarball/master  | tar -xz */package.py
#name of the downloaded package
packagename=$(grep "class.*Package" */package.py | sed "s,class ,," | sed "s,[(].*,,")
#name of the folder I get from github
foldername=$(ls -d */)
mv -f $foldername $packagename
#make a package folder
mkdir ../var/spack/repos/builtin/packages/$packagename
#move the package into that folder
mv -f $packagename/* ../var/spack/repos/builtin/packages/$packagename
rm -r $packagename
#install it
spack install $packagename

(Run from a specific directory) I can point this at my test repo (https://github.com/DavidPoliakoff/trivial), whose only interesting feature is having a package.py in the root, and it will pull the package file out of the repo, make a directory for it in my spack repo, move the package.py into that directory, then install it.

This is a horrible way of achieving the goal, but it's what I'll use in projects I don't feel like putting up package PR's for until

spack [verb] [github link]

lets me install the contents of that repo using spack, using a package.py provided by that github repo. Essentially, this lets me turn the spack model on its head, instead of $SPACK_ROOT/var/spack/repos/builtin/packages/ being the place that says what packages exist, software packages can provide a description of how they should be packaged by spack, and commands can be run to pull them down.

TODO's would be

1) Pythonic, spacky implementation of this bash script (perhaps through a new verb, spack clone)
2) Dependency handling. How can my repo depend on another repo? Perhaps syntax like (depends_on(get_github_package("[github URL]"))
3) Doing the same for hg, svn, tarballs

Basically, Spack wants to have a repo (or set of repos) that tell it where software exists, and how to pull it down. This is a great model, but I also want software to be able to say "here's how Spack can install me" and have Spack be able to respond

[adamjstewart]

I would also love it if developers wrote package.py files for us, but how likely is this to happen? I don't see what the benefit would be for developers to write their own package.py files and not add them to Spack as opposed to just adding them to Spack. Once they are in Spack, we can vet them and confirm that they work as intended. We can also patch them when dependencies get renamed or the package.py structure changes. I have a feeling the developers for popular packages have never heard of Spack and certainly don't want to maintain a package.py-like file for every package manager out there.

[DavidPoliakoff]

@adamjstewart

1) I'm going to write them for my own software, much of which won't be able to be put on github. I also find it awkward that every time a package developer modifies or creates a package, @tgamblin or somebody has to approve the action through a PR. I don't like the state of every spack package being something the spack repository has to monitor.

2) Installation. "bower install [github repo]" is the cleanest interface I have used in package management. How do I verify that the repo is installable by bower? It had bower.json in the root. How do I install it? "Copy" "bower install " "Paste".

I don't know that we want to plan on spack not growing to the point where Todd can't manage a PR every time a package is updated, having some mechanism for distributing our packages in places people can maintain them themselves is valuable, as is the bower-like interface.

[DavidPoliakoff]

As a fun strawman, I searched for the number of files with "devDependencies," a bower spec keyword, on github, and came up with 3 million files. Assuming 95 percent are false positives, if spack were to be a tenth as popular as bower that would still be 15,000 packages Todd would be responsible for reviewing PR's for, and if he shunts 1% of those to me I wouldn't have time to do my job. It _would_ be amusing if the HPC package manager ran into a scalability problem, though.

[trws]

On 28 Jun 2016, at 14:30, Adam J. Stewart wrote:

I would also love it if developers wrote package.py files for us, but
how likely is this to happen? I don't see what the benefit would be
for developers to write their own package.py files and not add them to
Spack as opposed to just adding them to Spack. Once they are in Spack,
we can vet them and confirm that they work as intended. We can also
patch them when dependencies get renamed or the package.py structure
changes. I have a feeling the developers for popular packages have
never heard of Spack and certainly don't want to maintain a
package.py-like file for every package manager out there.

See clibs, npm, and bower. They all do this, partly because it's a very
easy way to install the dependencies they require even if they don't
want to package their code as such. It's not something everyone will
want to do, but it's good for a project that wants to have something for
just them, or isn't public, or just to develop a new version of the
package file for new project requirements as well. Whether that's all
sufficient to add it is another question, but I would see myself using
it if only for the dependency installation part of the story.

[tgamblin]

I don't think this is a bad feature, but it's also not at the top of my priority list.

There are a few issues I can think of:

1. Where does the package name come from? If the git repo contains package.py, do we just reverse-map the class name to a package name? I would assume so. I would also assume that these packages would be namespaces.
2. How do I even begin to get a package version out of this? This is super for installing something once. How do I install it twice? Do I need the as-yet-unimplemented feature where we auto-version every git version by commit SHA and every svn version by its revision? I would really like it if spack figured out that master is actually a particular SHA and canonicalized the installed version somehow.
3. This is not secure unless YOU control the repo and maybe not even then. You're trusting a github repo to contain what you expect, especially if all you ask for is a URL. If the repo gets hacked, or worse, if github does, you're screwed. Checksums make me feel warmer and safer.

[citibeth]

1. How do I even begin to get a package version out of this?

The git repo would have a branch called "spack," which would contain just
the Spack package.py (for all versions).

Of course... how would Spack FIND this URL? I suppose you could specify
certain URL patterns as secondary repositories. If Spack can't find
"mypackage" in the main repos on-disk, it would look for "
http://github.com/.../mypackage.git" to see if it can find the package
there.

Right now, Spack wants to read all the package.py files before it starts.
This works OK for now, but would no longer work for remote repository
collections such as GitHub.

1. This is not secure unless YOU control the repo and maybe not even
   then.

Absolutely not secure. It would be prohibited unless running in the
proposed "unverified" mode.

-- Elizabeth

[alalazo]

If I put this ( @citibeth ) :

Right now, Spack wants to read all the package.py files before it starts.
This works OK for now, but would no longer work for remote repository
collections such as GitHub.

together with @DavidPoliakoff wishlist :

2) Dependency handling. How can my repo depend on another repo? Perhaps syntax like (depends_on(get_github_package("[github URL]"))

I see another design issue which could be quite difficult to sort out. How would you deal with _distributed virtual dependencies_ ?

[citibeth]

together with @DavidPoliakoff https://github.com/DavidPoliakoff
wishlist :

2) Dependency handling. How can my repo depend on another repo? Perhaps
syntax like (depends_on(get_github_package("[github URL]"))

I don't think this is practical. I think a more practical way would be to:

1. Configure Spack:

spack_repo1 = spack/var/../repos # The standard repo location spack_repo2 = http://github.com/%s.git spack_repo3 = http://bitbucket.com/%s.git

1. Don't change the syntax of depends_on().

I see another design issue which could be quite difficult to sort out. How
would you deal with _distributed virtual dependencies_ ?

NOTE on terminology: A _spack repo_ is a set of Spack pacakges
(package.py plus affiliated stuff), each associated with a _package
name_. A _git repo_ is a set of software that can be downloaded with one
git clone command. In this proposal, git repos may sometimes function by
providing an individual packge to a spack repo. GitHub as a whole would be
considered a spack repo, not a git repo. Confusing...

I don't know if the above (1) would change this issue. But let me suggest
that Spack would probably need to cache external spack repos by copying the
appropriate package.py files into an internal cache directory (one per
external repo). This is not without precedent: when you use yum or
apt-get, it caches various software repos too. If you haven't run for a
while (maybe a day), it re-reads them.

Of course, Spack cannot rifle through all zillion git repos on GitHub to
find which ones provide a package.py. So how does it figure out that a
package repo exists on GitHub? When Spack searches for dependencies during
concretization, it can be told to look in each configured spack_repo. This
could be slow, of course, since it will require one query to GitHub for
every depends_on(). However, it would only need to be done once (per
day/week/whatever timeout). After that, it will cache information on
whether that Spack repo contains the given package. There would have to be
a command-line flag ignore what's in the cache(s) on any given spack install.

When you "refresh" an external Spack repo, it would only refresh packages
that have already been "discovered" and therefore cached. (These
discovered packages would be marked as either existing or not existing in
the Spack repo).

With a cache in place, Spack would only read cached packages upon startup.

I haven't thought through all the ramifications that a system like this
might have. The lack of discovery on the set of packages in the external
repo could be an issue. Maybe it would best to prohibit virtual packages
in the external repos. I don't think this would be a huge problem.

[DavidPoliakoff]

Just wanted to highlight in @alalazo (there was no "quake in fear" reaction, so I settled for :+1: ) and @citibeth on the virtual dependency idea, I think that

I haven't thought through all the ramifications that a system like this
might have. The lack of discovery on the set of packages in the external
repo could be an issue. Maybe it would best to prohibit virtual packages
in the external repos. I don't think this would be a huge problem.

Is a fair constraint on this. I think "require graph searches over all of github" isn't what they were looking for when they asked us to write an HPC package manager. I'm looking at downsizing this request. Perhaps "be able to look up packages which reference a given github link" plus "be able to have separate places to store repos (as per https://github.com/LLNL/spack/pull/965)" would solve my constraints while respecting Spack's "I have directories, and directories are where I look for packages" mentality.

Roughly, the sketch of this new version of spack-clone would be one that looks at a github repo link, looks through your spack repos for a package which references that github repo, and installs that package. I still _want_ packages to be able to be distributed with the software (I particularly appreciated the "have it in a branch a la gh-pages" mechanism, and think that's more generally interesting in the realm of package management as it also makes searching for packages more tenable, you search for repos providing that branch) I just think it's more than I should ask for right now

[citibeth]

Would you be cble to share the requirements and constraints you are working
under? I have a feeling you can probably do what you need, if not exactly
how you want to do it, without adding spack features.
On Jun 29, 2016 10:13 AM, "David Poliakoff" [email protected]
wrote:

Just wanted to highlight in @alalazo https://github.com/alalazo (there
was no "quake in fear" reaction, so I settled for 👍 ) and @citibeth
https://github.com/citibeth on the virtual dependency idea, I think that

I haven't thought through all the ramifications that a system like this
might have. The lack of discovery on the set of packages in the external
repo could be an issue. Maybe it would best to prohibit virtual packages
in the external repos. I don't think this would be a huge problem.

Is a fair constraint on this. I think "require graph searches over all of
github" isn't what they were looking for when they asked us to write an HPC
package manager. I'm looking at downsizing this request. Perhaps "be able
to look up packages which reference a given github link" plus "be able to
have separate places to store repos (as per #965
https://github.com/LLNL/spack/pull/965)" would solve my constraints
while respecting Spack's "I have directories, and directories are where I
look for packages" mentality.

Roughly, the sketch of this new version of spack-clone would be one that
looks at a github repo link, looks through your spack repos for a package
which references that github repo, and installs that package. I still
_want_ packages to be able to be distributed with the software (I
particularly appreciated the "have it in a branch a la gh-pages" mechanism,
and think that's more generally interesting in the realm of package
management as it also makes searching for packages more tenable, you search
for repos providing that branch) I just think it's more than I should ask
for right now

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LLNL/spack/issues/1136#issuecomment-229368453, or mute
the thread
https://github.com/notifications/unsubscribe/AB1cd4Kq0vtJgrougyb5Sc6TXW0wWj9cks5qQn17gaJpZM4JAjJM
.

[tgamblin]

Introducing URLs as package identifiers seems like a poor choice. Packages refer to each other by unqualified names right now, and they can optionally use a namespace to be very specific about which repo they come from.

I sort of thought this was about having one project build is dependencies automatically in a development environment. Is that correct? In that case can't a repo contain a regular package file that refers to other packages by name? And would it need to depend on _other_ GitHub repos? There is a mess of versioning horribleness here.

There are a couple of soon to be merged PRs that might also help a dev environment. #543 and #721 might be worth looking at, and @gartung is still working on the Fermi "SpackDev" layer but I don't have a link to a presentation handy.

[gartung]

@tgamblin "SpackDev" is smoke and mirrors at this point.

[trws]

If I'm understanding this, which perhaps I'm not, it's partly to get dependencies and partly to have an easy way to build something that hasn't made it into the main package repo. There is something nice about the way bower/npm/cargo/go/ tend to do this, in that it allows you to make your deps part of your repository for developers and intrepid users.

Consider an npm package.json like this:

{ "name": "example-local-package",
  "description": "a delightfully fruity coffee varietal",
  "version": "1.2.3",
  "devDependencies": {
    "coffee-script": "~1.6.3" # dependencies for dev-env only
  },
  { "dependencies" : # note the various version types
      { "foo" : "1.0.0 - 2.9999.9999"
      , "bar" : ">=1.0.2 <2.1.2"
      , "baz" : ">1.0.2 <=2.3.4"
      , "boo" : "2.0.1"
      , "qux" : "<1.0.0 || >=2.3.1 <2.4.5 || >=2.5.2 <3.0.0"
      , "asd" : "http://asdf.com/asdf.tar.gz" #this is an explicit version
      #note the explicit version by tag
      , "caliper"  : "git+https://github.com/llnl/caliper#v1.01"
      #note the explicit version by sha
      , "caliper"  : "git+https://github.com/llnl/caliper#ahf9h23h9rha"
      #note the explicit version by branch
      , "caliper"  : "git+https://github.com/llnl/caliper#develop"
      }
    },
  "scripts": {
    "prepublish": "coffee -o lib/ -c src/waza.coffee"
  },
  "main": "lib/waza.js"
}

Note that the package names for vcs-based dependencies are explicit, the version for the package being built is explicit, the _name_ of the current package is explicit, as is the version. For CMake-based projects that could also be automatic, but lets keep this simple. It's completely reasonable to require that these things be explicitly set for a mechanism like this. Further, the versioning issue is pretty trivial, since the whole point is that you just list the dependency with an explicit revision, tag, branch whatever to get what you need. If you don't specify it, more like go where you can't, then you get whatever the latest is on the master branch or trunk or what-have-you at the target.

Unfortunately, this is not presently how spack packages work, and it would mean adding new attributes or metadata somewhere.

Where this would be most useful, at least for me, is to go into a directory, use spack diy and have it pick up the local package and get me everything I need for a little personal or unreleased or sensitive project. It would be great if it would be yet more npmish and put a view for those deps in a known local directory too, but the functionality in question is really just to have a way to give spack package information whithin the repository of a project, rather than having to cram it into the spack repo, where it's more practical to do it that way.

All of this said, it's not how spack is meant to be used right now, but many of the arguments I've seen on this thread have been attacking assumed implementation details rather than whether or not having a way to do this would be appropriate or practical.

By the way, while the security argument is valid on the surface, I really don't buy it for this use case. If a user does spack create ... it will create an MD5 based on whatever man-in-the-middle attack is currently interposing itself between them and the actual source just as much as using a repo url would. Also, if spack doesn't do it, they would just clone the darn thing. This is not to say that packages in the repository shouldn't have md5s, they should, but for one-offs it's not a win unless the md5 can be retrieved from the remote, which could also be replaced by the same malicious interposer. This mechanism shouldn't be our preferred way to do things in the repo, but this is a different mechanism than that.

[DavidPoliakoff]

Okay, I think out of this, I've isolated what I want, and how I want to make it happen. These are separate conversations.

What I want:

1) "spack [verb] [github link]" to be like bower (thanks @trws for a good description of "Bower-like")
2) The ability to maintain my own packages without sending Todd and Greg a PR every time I change my mind about something.

(2), specifically, seems like it's addressing a preposterous claim, that the Spack repo should be involved any time I decide to update how I want my software packaged. "How this software should be packaged" feels like something that is entirely a decision of the software developer, and that Todd shouldn't have (or want) any say. Further, this would let me change how the software is packaged based on the repo, so if I have something that is specific to the LC site, I could have that in a package.py in an LC repo. I don't know, "the spack git repo is the repo containing all spack packages" feels obnoxious to me.

Proposed implementation

1) Github repositories may supply a branch (thanks @citibeth , this might make "find github repos containing spack packages feasible, though I'm not proposing it here) named "spack-package" containing a package.py and any other contents you might put into the directory package.py
2) A new verb, spack clone. The "spack clone [git URL]" command should take in the main github repo link (https://github.com/LLNL/spack/), transform it into one containing the branch we want (https://github.com/LLNL/spack/tree/spack-package (does not exist in this case)), inspect the package for the class name, transform the class name into a package name, make a folder in a specified repo (default: builtin) of that name, then spack install that package
3) (optional, I understand people's discomfort). I want to extend install to have depends_on(github_package([git URL])). github_package([git URL]) does essentially a spack clone (sans install), then depends on the resulting package.

Immediate problems

1) Todd's question 2 of the version of the installed package
2) "I store my github repositories at [random site].dog/[random name].tar, but it's a git repository I swear." Spiral 0 will probably work on things in "[github,bitbucket,stash].[3 letters]/stuff" links
3) Storing this on a branch means we have to transform URLs differently if bitbucket gives you a branch URL differently from github (update: surprise, it does! Who could have guessed)

Possible extensions (why not go pie in the sky)

1) Steal @trws ideas (rarely steered me wrong in the past). Having a bower-like system where you get those packages in a local directory would be sweet. If I can have a CMake project, do something like "spack import [package name]", and get links to the install directory of "package" and its dependencies, I probably stop writing smart build systems and offload that work onto spack import. "spack clone --local [package_name]" doing something similar would be swell.
2) More broadly enhancing spack's interaction with git, I've talked about spack raising issues on failed builds, spack feature requests, this feels like a space to mention that
3) Leveraging @citibeth 's suggestion of this package being on a branch to look things up on github. I feel like this quickly becomes untenable, but the idea appeals to me

Thanks @alalazo for pointing out the nightmare case for this (virtual dependencies).

[trws]

For my part, I like all of what you said @DavidPoliakoff, except using a branch instead of a file in the repo. It complicates the design for non-git targets, and divorces the package file from the software version being grabbed. I'm not necessarily against supporting it, but it seems like it would cause more problems than it would solve unless it's required to be able to support back versions of the package and supply versions for them rather than embedding that information in the commits.

[tgamblin]

RE: what you want:

1. @DavidPoliakoff: I agree with Tom, I like the package.py at the root level. It then gets versioned with the software kind of like travis.yml.
2. spack clone is not my favorite name, if only b/c it seems ambiguous. I am not sure what is better but I'll think on it.

RE: proposed implementation:

1. (also 2) I think the branch stuff really complicates the implementation here and this would be simpler if you just used a URL. I think making this take a valid git clone URL like Tom's "delightful coffee varietal" is fine.
2. Why not something like depends_on('foo', git='https://github.com/LLNL/spack.git')? This has the advantage of supplying a name for the package, AND you could leverage the fetcher/keyword args identification logic in spack.fetch_strategy. I claim the version installed should be something like git.<SHA> where SHA is the commit hash. I would really like it if depends_on('foo', git='https://github.com/LLNL/spack.git', branch='master') ended up installing a proper git.<SHA> version (as I could keep installing newer versions).

RE: Immediate problems

1. See above for my version suggestion. I think it would actually be nice if all git/svn versions (not just ones for this PR) were auto-canonicalized by Spack so that fetching and installing, e.g. master results in reliable provenance.
2. Using a git clone URL instead of a branch simplifies this too, and makes it more versatile.
3. Do you get the idea I don't like the branch thing?

RE: Possible Extensions

1. Yes but save if for a later PR. Also look at spack view.
2. See my suggestion above about canonical git versions. That would be my first git fix.
3. I'll wait til you have more concrete suggestions to comment on that.

Concerns/suggestions

• Packages fetched this way, since they are not associated with a particular spack repository, should be given their own namespace, I just dunno what to call it. I think there should be a name for this feature and the namespace should be the name, but just as I dunno what would be better than spack clone, I don't have a name for this feature off the top of my head. "Instant packages" and "hack packages" (or "spack hack packages") come to mind, so maybe instant.* or hack.*. This at least lets you know the package didn't come out of a known repo.

Two other points:

1. I like this because I think it's useful for developers. What I do not want to do is allow any hack packages in the main repos. If a package goes into builtin, it needs to depend on other packages by name, not with this. Note that this has ramifications on being able to build a decent, reproducible mirror as well. I think this feature should only be used to build one-off DAGs where the root comes is a hack package, so that we can keep builtin sane.
2. You say "How this software should be packaged" feels like something that is entirely a decision of the software developer. I know what you mean but remember that Spack customizes a number of build options for packages and uses things like compiler wrappers that do take away some of the say of the developer in how their software is packaged and installed, and that's by design. If you do a little *reductio ad absurdum` on your statement, you end up with the traditional build chaos of HPC, which is what Spack is supposed to avoid. So I think there are limits here. You have to adopt some structure if you want to handle combinatorial DAG versioning, which is what Spack needs to be able to do. The implementation of this should keep that in mind and not kill the build model.

[citibeth]

What I want:
2) The ability to maintain my own packages without sending Todd and Greg a PR every time I change my mind about something.

Clone the main Spack git repo, and add whatever packages you like to it without telling Todd or Greg. Many of us work that way. Alternately, add a second Spack repo to your Spack configuration, and put whatever you like in it without telling Todd or Greg. With two ways to accomplish this already, I don't see why we should be implementing yet another.

Maybe there should be an easier way to submit new packages: cloning Spack and submitting PRs could be discouraging contribution from users who just want to add new packages. Maybe we should allow people to create a new issue and post a package.py file in it, and then someone else (or a script) would convert those issues into PRs. Anything more complex, I think people should learn and apply the standard ways of doing things with GitHub.

1) "spack [verb] [github link]" to be like bower (thanks @trws for a good description of "Bower-like")

We've been discussing how this might work. Many on this list agree the idea has merit. However, I must point out that this is a want, not a need. I don't see even one thing that this feature would enable, that can't already be done. If someone who really wants it implements it and submits a PR, and it is well designed, it is likely to make it into Spack.

Why not something like depends_on('foo', git='https://github.com/LLNL/spack.git')?

Because this is mixing up the repo in which a package is found, vs. the identity of the package itself. Packages and source code can and do change location.

For my part, I like all of what you said @DavidPoliakoff, except using a branch instead of a file in the repo. divorces the package file from the software version being grabbed

Central to Spack is an ability to deal with combinatorial complexity of many different versions of the same package. From this need springs a core Spack design decision: one package tells how to install many different versions of the package. Changing to a system where there are many packages for the same piece of software, and each package only installs one version, would be fundamentally different. That's just now how Spack works; and without a LOT of thought and re-working of Spack fundamentals, it would likely break Spack. I don't care to put further thought into this issue. But definitely... coherent, complete designs are welcome, especially if they come with an implementation in a PR.

Maybe these Bower-like ideas would fit better into EasyBuild, where package files are version-specific.

It complicates the design for non-git targets, and

It could be done with non-git systems. Other VC systems have branches as well. Or it could go in a separate repo with similar name (eg: mypackage-spack.svn)

[DavidPoliakoff]

@citibeth

Clone the main Spack git repo, and add whatever packages you like to it without telling Todd or Greg. Many of us work that way. Alternately, add a second Spack repo to your Spack configuration, and put whatever you like in it without telling Todd or Greg. With two ways to accomplish this already, I don't see why we should be implementing yet another.

This can't be posed as a serious solution for a package manager used by people not on a first-name basis with one another. The point of a package manager is to allow us to package and distribute software. We currently can't iterate on that without a PR or a fork. All the rest derives from that central point.

I think the point of this being a want and not a need is valid, but I personally see myself iterating on the spack package I want to develop, pushing it out to users, taking feedback, and reiterating (no Spack PR). I really don't want to have this work by my user git pulling in spack on a branch or using mechanisms other than Spack (including downloading the package file and copying it into a directory repeatedly). If I could push my spack package to my git repo and just have my user "spack clean raja; spack [verb] github.com/LLNL/RAJA" until the package is good I would be happy.

Posing it as a design challenge: how can a developer and friendly user do ten iterations of a spack package for a piece of software, using the friendly user's currently existing spack install (based on develop), only using spack commands? My user (who isn't hypothetical, this is a real problem) _hates_ git, and if I tell him to merge a branch into develop or copy files deep into the spack repo he's going to tell me to do things that aren't anatomically possible.

@tgamblin I agree with all of this. In past github hacking, I've found using their REST API to pull out a lot of information about the state of the git repo to be helpful (https://api.github.com/repos/LLNL/spack, https://api.github.com/repos/LLNL/spack/branches), perhaps similar ideas could be added to GitFetchStrategy.

[citibeth]

On Wed, Jun 29, 2016 at 6:01 PM, David Poliakoff [email protected]
wrote:

@citibeth https://github.com/citibeth

Clone the main Spack git repo, and add whatever packages you like to it
without telling Todd or Greg. Many of us work that way. Alternately, add a
second Spack repo to your Spack configuration, and put whatever you like in
it without telling Todd or Greg. With two ways to accomplish this already,
I don't see why we should be implementing yet another.

This can't be posed as a serious solution for a package manager used by
people not on a first-name basis with one another. The point of a package
manager is to allow us to package _and distribute_ software. We currently
can't iterate on that without a PR or a fork. All the rest derives from
that central point.

I mentioned two existing solutions to your problem. Here is the second
one: Spack already provides for separate repos apart from the built-in
Spack. See the file repos.yaml. (@tgamblin this needs to be documented,
do you know who could do that?) You should be able to set up a Spack repo
using a VCS system that is NOT git, and share that with your colleagues?
They just have to point their Spack to it, and then you are good to go.
OK... so users will have to say "git pull", "svn update" or whatever every
time you iterate. But is that so hard?

The first solution (that you rejected above) is used for now because Spack
is still Alpha software. Things can change, things can break. Even if
core Spack doesn't change/break, people update packages. The result is
that YOU might find your software installs with YOUR preferred version of
Spack --- but it breaks for others. Because some package 6 levels down in
your DAG was updated between your version and their version, and there's an
error somewhere, and it breaks. This is not a problem just with Spack:
MacPorts is notorious for builds breaking all the time --- things that used
to work suddenly stop working (in their defence; usually it has to do with
an Apple-imposed upgrade).

My point is...

1. If you're assembling a large software stack, then the best way (today)
   to ensure your users can install it is to completely control the version of
   Spack (and its repos) that you use to install --- and then test the h*ll
   out of that version. I know that this breaks a core advantage of Spack,
   the idea that it can be used to simultaneously install two large software
   stacks. One idea that's been kicked around is to separate Spack core from
   the Spack repo, so you can update one without the other. I think that will
   be practical in the future. But not now, not while Spack is still in
   Alpha. At least one recent PR, for example, has resulted in all packages
   being touched.
2. Even if we had things the way you envision, chances are random
   `package.py

or using mechanisms other than Spack (including downloading the package
file and copying it into a directory repeatedly).

VCS systems are really good at this stuff. Why should Spack re-invent that
wheel? Is it too much to ask your users to "git pull" or "svn update" or
whatever when you change your package.py files?

Posing it as a design challenge: how can a developer and friendly user do
ten iterations of a spack package for a piece of software, using the
friendly user's currently existing spack install (based on develop), only
using spack commands?

It would be by having Spack call the git commands, instead of you doing
that directly. Maybe there is some value to this.

My user (who isn't hypothetical, this is a real problem) _hates_ git, and
if I tell him to merge a branch into develop or copy files deep into the
spack repo he's going to tell me to do things that aren't anatomically
possible.

IMHO, there needs to be a balance between making things easy for naive
users, and catering to their every whim. The professional tools are there
for a reason.

Anyway... see here for the kind of instructions I've put together than have
worked for others using my software (scroll down):
https://github.com/citibeth/icebin

@tgamblin https://github.com/tgamblin I agree with all of this. In past
github hacking, I've found using their REST API to pull out a lot of
information about the state of the git repo to be helpful (
https://api.github.com/repos/LLNL/spack,
https://api.github.com/repos/LLNL/spack/branches), perhaps similar ideas
could be added to GitFetchStrategy.

I think the best approach from Spack core would be to create a Spack repo
API by abstracting away the details of a filesystem, GitHub, etc. If we
can do that and get it implemented, then it should not be too hard to
implement a new Spack repo class that treats all of GitHub as a Spack repo.

--- Elizabeth

[alalazo]

@citibeth OT : I was reading your instructions. Concerning this :

The ``openssl`` section tells Spack to use the OS version of the
   OpenSSL library, rather than building one itself.  This is for
   security reasons.

   If you choose this route, Spack will later give you
   spurious warnings that look like::

        ==> Warning: This installation depends on an old version of OpenSSL,
                     which may have known security issues.
        ==> Warning: Consider updating to the latest version of this package.
        ==> Warning: More details at http://www.openssl.org

   You can safely ignore these warnings because they are false.

In develop the reserved version tag is external not system. I think that if you change it you should not receive a warning, just a message saying that you are using an OpenSSL version externally installed at <folder>.

[DavidPoliakoff]

I mentioned two existing solutions to your problem. Here is the second
one: Spack already provides for separate repos apart from the built-in
Spack. See the file repos.yaml. (@tgamblin this needs to be documented,
do you know who could do that?) You should be able to set up a Spack repo
using a VCS system that is NOT git, and share that with your colleagues?
They just have to point their Spack to it, and then you are good to go.
OK... so users will have to say "git pull", "svn update" or whatever every
time you iterate. But is that so hard?

I'd make the claim that this is so hard. I have to git clone a (git) repo containing the (spack) repo containing the package I want, spack repo add it, then every time that _single_ package updates, I have to spack repo list, find the directory in which I stored that repo, and do a git pull. This is bananas. The way I update a package in spack is to use git? Bare minimum I need "spack repo create [github link]" to do something intelligent, and a "spack update remote repos" command to do the git pull.

Two comments you make that I'd like to bold face

One idea that's been kicked around is to separate Spack core from
the Spack repo, so you can update one without the other. I think that will
be practical in the future.

Absolutely necessary. If you introduce a smart package in Spack 1.2, and I introduce a bug that makes everybody hate Spack, we need to let people get the package without the bug. Obviously, Spack being alpha means the ideal end state isn't what we want to implement tomorrow, but :+1:

I think the best approach from Spack core would be to create a Spack repo
API by abstracting away the details of a filesystem, GitHub, etc. If we
can do that and get it implemented, then it should not be too hard to
implement a new Spack repo class that treats all of GitHub as a Spack repo

I don't know that treating all of GitHub as a Spack repo is doable (how do you find everything that is a spack package in a local spack repo? "ls." How do you find everything on Github that is a Spack repo? No, seriously, how?), but removing some of the friction between "this resource exists at a git link somewhere" and "this resource has been faithfully downloaded and registered with spack on the local file system" would remove much of my angst over how we currently use Spack. I seriously don't know whether I can recommend to my users a a system in which pulling my updates to a package.py means finding where spack stores the package and running a git pull.

I want to explore the points @alalazo raised on virtual dependencies in Github located package, and that everybody is raising on some of the "core" packages (how do I store a compiler package which spack should be able to recognize as satisfying something), in each case I think the answer might be "don't, those go through the PR process and get it in builtin" but am pretty certain that storing the package with the software is something I want to explore.

As a pure upside: this should involve improvements to the GitFetchStrategy which lead to storage of details about the downloaded git packages, something useful regardless of the rest of this PR, but I remain convinced that "spack [verb] [github URL]" should be a way to install software somehow

[tgamblin]

@DavidPoliakoff:

I'd make the claim that this is so hard. I have to git clone a (git) repo containing the (spack) repo containing the package I want, spack repo add it, then every time that single package updates, I have to spack repo list, find the directory in which I stored that repo, and do a git pull. This is bananas. The way I update a package in spack is to use git? Bare minimum I need "spack repo create [github link]" to do something intelligent, and a "spack update remote repos" command to do the git pull.

Is this really just about the fact that you can’t have spack repo add <git URL> and have it automatically pull? If that's all you need, implementing that _is_ easier than implementing what you’re proposing. All you really need is some of the functionality from homebrew taps, which would be simple to add.

[tgamblin]

@DavidPoliakoff:

Posing it as a design challenge: how can a developer and friendly user do ten iterations of a spack package for a piece of software, using the friendly user's currently existing spack install (based on develop), only using spack commands?

So, this actually _is_ fairly complicated at the moment. Spack is designed to work out of the box for "fairly recent" stuff that's already been packaged. Keep in mind that for most people and for most software, _that_ is the common use case. People want a stack they can trust, and something that's gone through PR and integration helps with that, but I agree we need something better for the use case you're describing.

This is basically a user support use case. We do this all the time in the DEG (tier-3 user support group) at LLNL. spack diy doesn't handle this well because it's designed for when _you_ are tweaking _your own_ software. You're asking for a way to quickly iterate with _someone else_ on _your_ software. We _do_ encounter this use case all the time.

@citibeth:

Why not something like depends_on('foo', git='https://github.com/LLNL/spack.git')?

Because this is mixing up the repo in which a package is found, vs. the identity of the package itself. Packages and source code can and do change location.

I agree, sort of -- this is why I said above that I wouldn't allow packages like this in the mainline. I see a use for it for rapid iteration, which is what @trws and @DavidPoliakoff are talking about. Here is some rationale:

Spack currently uses the syntax [email protected] to describe a package at a particular version. The package name gets translated to a package file that tells you how to build, and the version tells you an archive to fetch. If you don't specify the version, Spack picks one, but this is basically an identifier for some source code and a build methodology. There are other parameters (like the compiler and build options), but that is basically what [email protected] boils down to.

<arbitrary VCS URL> is _also_ a way to refer to some software at a (probably moving) version. What it lacks is a build description that Spack understands. To get back to @DavidPoliakoff's challenge above, I think you want:

``````

$ spack install some-package ^https://github.com/LLNL/RAJA.git

``````

That means I want to build some-package with the latest snapshot version of RAJA from the provided URL. Not _actually_ so different from what we do now. What we do now implies a search of some local package repos; here there is no search; you're just saying build this. It's more specific.

Alternately if i just want to build the latest RAJA snapshot, I could type:
shell $ spack install https://github.com/LLNL/RAJA.git

I'm actually _not_ opposed to this and don' t think it breaks the build model, as long as it does a few things right.

To preserve the build model, spack would _need_ to check the package name and version _before_ concretizing. i.e.:

1. Query the VCS repo for commit id
2. Pull down the root package.py
3. Figure out the package name from that.

Once you have done that, you basically have a valid raja@version spec. This would be an _unsafe_ version in that you're trusting what is at that URL. As @trws said above, you knew that, _caveat emptor_. Per my earlier message, I would require one more thing here, which is that the spec is not just raja@version but hack.raja@version, using the namespace mechanism to tell you that this did _not_ come from the builtin repo or any other. And Spack should store the build provenance for this thing, including all package.py files, like it does for anything else.

So again, in my mind, the URL isn't really violating the build model. You just have to remember that it's a handle on a _moving_ version of a piece of software, and you need to lazily evaluate the version. If you do that, it actually _makes sense_ in a Spack spec.

I remain very unconvinced that these one-off specs should go in the builtin repo, or really any Spack package repo, as @citibeth rightly points out that this removes some useful indirection between a package's identity and its location. In the mainline, or in any repo that is used to manage a software suite, _you should be able to maintain URLs for one package in one place_. That is what the indirection buys you.

However, both of the commands above are _one line long_ and I think they could help support teams help users. You can paste one of the commands above in an email to a grumpy person who just needs a build right now. The same command can be re-executed without even another email, after the first attempt turns out to be broken. And Spack will happily install each successive invocation as a new, but _concrete_ version of the thing in the repo you referenced. Honestly, I kind of like it.

[DavidPoliakoff]

Cool. For now I have this in a branch: https://github.com/DavidPoliakoff/spack/tree/feature/spack-clone

I have (1) and (2) working, and will try to make (3) happen as well. I also really think the GitFetcherStrategy should be aware of the Github API, but I'll implement that and show that it's useful rather than talk about it. Spiral 0 will only work with github, but I'll extend it to bitbucket after proving the concept there

Thanks all for the spirited discussion!

edit: yes, the verb is "clone." When I put up a PR, issue one will be renaming it

[citibeth]

There's a hierarchy here. Git=git. GitHub and Bitbucket subclass Git.
When implementing GitHub-only features, they should not be put in general
git stuff, "subclass" a github fetcher strategy.

On Thu, Jun 30, 2016 at 9:40 AM, David Poliakoff [email protected]
wrote:

Cool. For now I have this in a branch:
https://github.com/DavidPoliakoff/spack/tree/feature/spack-clone

I have (1) and (2) working, and will try to make (3) happen as well. I
also really think the GitFetcherStrategy should be aware of the Github API,
but I'll implement that and show that it's useful rather than talk about
it. Spiral 0 will only work with github, but I'll extend it to bitbucket
after proving the concept there

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LLNL/spack/issues/1136#issuecomment-229661005, or mute
the thread
https://github.com/notifications/unsubscribe/AB1cdyzmpjhLhtgM1oZ4KCxovg3Zy0k9ks5qQ8dFgaJpZM4JAjJM
.

[tgamblin]

@DavidPoliakoff: why only GitHub? I believe you can do hear things with just git, or does the protocol not support just asking for the tip of a remote branch without a whole fetch? Do you _need_ the REST API?

[DavidPoliakoff]

@trws (and anybody else who is a git geek): I'm specializing around the github API for the following reasons

1) Using the raw.githubusercontent domain, I can pull down exactly the package.py file and nothing else. I don't know how to do git clone --branch --get-only-this-one-file, I've seen "shallow cloning" but not "just give me a file"
2) The REST API gives me a JSON file. json.load(serverResponse) gives me a python dictionary, and as I am _profoundly_ lazy, the ease of use there is appealing when compared to writing some set of parsers for subprocess output grossness to extract the same information. Unless git has a "git [verb] [miraculous json flag]" this buys us ease of use.

Any ideas on features I'm missing that make the REST API a bad idea, that let me just use git directly instead?

Example snippet

    repoStatusFetcher = URLFetchStrategy(url=descriptorURL)
    repoStatusDict = {}
    branchStatusDict = {}
    branchName = ""
    with Stage(repoStatusFetcher) as stage:
        repoStatusFetcher.fetch()
        descriptor = stage.path + "/" + descriptorURL.split("/")[-1]
        with open(descriptor,"r") as statusFile:
            repoStatusDict = json.load(statusFile)
            branchName = repoStatusDict["default_branch"]

At the end of this, repoStatusDict is https://api.github.com/repos/LLNL/spack as a Python dictionary, and I do something similar for the current branch, getting a rather comprehensive view of the state of the repository. Favorite entry for Spack

  "has_issues": true

Which seems a bit passive-aggressive of github, but oh well

[tgamblin]

@DavidPoliakoff: this should work with plain git, not just github. The whole point is so users can be lazy, not us.

[DavidPoliakoff]

The whole point is so users can be lazy, not us.

See, this is the kind of thing that makes me not want to implement things. The effort, the actual thinking...

More seriously, I'm still looking for how to pull down package.py from a git repo. I could do a shallow clone of the root directory and just ignore a lot of it, but that still seems iffy

[tgamblin]

Would prefetching with a shallow clone of the git repo be so bad? You're going to build it anyway, much of the time, and the fetcher already does a shallow clone... Also there's always googling...

http://stackoverflow.com/questions/2466735/how-to-checkout-only-one-file-from-git-repository

[DavidPoliakoff]

Thanks, Todd, hadn't considered Google, having been born yesterday (early versions of my bash script used that methodology on SO, and ran into problems)

The problem with that, if somebody tries to spack clone [github URL], I'm not yet positive that respects default branches. Is HEAD the default branch on the github repo, or is it master? Above that, there's also the problem's pointed out in the answer, the git server has to respect quite a few options (which, again, I ran into issues with when trying to write this as a script)

I'll do it with a shallow clone, but this isn't as trivial a decision as it's being made out to be.

[citibeth]

People will want to use this with non-GitHub repos. If you do it GitHub-only, then someone will have to do it again for plain vanilla git. Even if plain vanilla git is more work than GitHub, it is still less work in the end --- because one implementation can work for both. If the plain vanillia git implementation is lacking in usability, it can be optimized for the GitHub case.

Here is info on default branches and HEAD. It would be nice to assume that people all use git >=1.8. Unfortunately, that is not true --- and it's not easy to get the people administering these systems to upgrade either.

http://stackoverflow.com/questions/18726037/what-determines-default-branch-after-git-clone

[tgamblin]

I'm not necessarily opposed to requiring a certain git version for the feature. I think requiring 1.8 would be ok. RHEL6 comes with git 1.7 FWIW.

[DavidPoliakoff]

Cool, I'll do it with a git shallow clone _for this specific thing_. I think we do want github-specific features, the "if this is broken, raise an issue" thing is tightly connected with github/bitbucket, but for the first pass of just cloning repos, I'll do shallow clone

[citibeth]

Yes, RHEL6 comes with git 1.7. At our lab, we are stuck with a git server running on RHEL6. Why doesn't it get upgraded? Because people are (a) paranoid about security and (b) lazy. Combine the two, and upgrades will only be applied through Yum and the default RHEL repo. Why don't they use Spack to produce a newer version of git? I'm working on that...

Realistically, you have to assume that any LTS/Enterprise system that has not gone end-of-life will still be out there. I'm not saying this features has to rely on git 1.7 capabilities only. But it should at least detect the version of git in use, provide that to the Spack hacker, and allow for alternate procedures in the case of old gits.

[DavidPoliakoff]

For now, I'm going to write things that use the GitFetchStrategy class, say "I faithfully executed my duties as a developer of Spack by using its provided classes," and file issues if it breaks, as I'm looking to not spend terribly much time hacking on this until the hackathon/spackathon after next

[trws]

Seems you got here anyway, but shallow cloning is the way to go here. The remote archive feature @tgamblin linked to isn't supported by github, or the https protocol at all actually, so it's not reliable regardless of git version available. There are ways to filter down to just part of a repository, including porcelain for sub-trees, but they all require a local repository to work on.

Ironic note of the day, the easiest way to get this for github, or github-like services, is actually to use their subversion support and just check out the one file... Really not promoting that as a preferred option, but if the feature supports SVN repos it will give users the option to use that link instead if one is available.

[DavidPoliakoff]

@trws I know (re: SVN). I just can't bring myself to. And Jeff and Rich and Dave can never know we had this conversation

[citibeth]

Sounds good to me
On Jun 30, 2016 1:27 PM, "David Poliakoff" [email protected] wrote:

For now, I'm going to write things that use the GitFetchStrategy class,
say "I faithfully executed my duties as a developer of Spack by using its
provided classes," and file issues if it breaks, as I'm looking to not
spend terribly much time hacking on this until the hackathon/spackathon
after next

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LLNL/spack/issues/1136#issuecomment-229729686, or mute
the thread
https://github.com/notifications/unsubscribe/AB1cd6P-Kd3_I3fWsECyQ7eXpPH_bSVEks5qQ_xdgaJpZM4JAjJM
.

[trws]

@trws I know (re: SVN). I just can't bring myself to. And Jeff and Rich and Dave can never know we had this conversation

Is there anything about this that needs to tie it to git? I think there's a fetch strategy for svn already if not mercurial, and it would be cool for each of them to just work (given an appropriately formatted URL of course).

Completely understand wanting to prototype with just one, but something to think about.

[DavidPoliakoff]

@trws The only thing standing in the way is that I'm not excited about hg or svn (or tarballs, for that matter) and have things I want to do once this is in spack. Technically speaking though you're absolutely right

[DavidPoliakoff]

Also @tgamblin I'm missing some blanks in Spack terminology (or spack is missing features)

packageInfo = insertMagicHere(package_dir+"/package.py")
packageName = packageInfo.get_name()

Does Spack have insertMagicHere available?

[DavidPoliakoff]

Don't know if that sends a notification, but the PR is up

[tgamblin]

@DavidPoliakoff: At the moment all the loading stuff is in repository.py, and it's set up to load a package given a spec. We should talk about how that should look for packages loaded this way.

[citibeth]

See #1149 ?
On Jun 30, 2016 2:38 PM, "Todd Gamblin" [email protected] wrote:

@DavidPoliakoff https://github.com/DavidPoliakoff: At the moment all
the loading stuff is in repository.py, and it's set up to load a package
given a spec. We should talk about how that should look for packages loaded
this way.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LLNL/spack/issues/1136#issuecomment-229749922, or mute
the thread
https://github.com/notifications/unsubscribe/AB1cd2GnSrPvyTv3OOVVazsHnr2ez8gkks5qRAzngaJpZM4JAjJM
.

[DavidPoliakoff]

@tgamblin : I see a few avenues without changes in Spack

One is to just open the file, scan for the line that fits "class.*(Package):", and sed script our way to victory. Another is to use a placeholder name, load that package, gets its name, then move the placeholder into the appropriately named space.

Preferences?

[citibeth]

One is to just open the file, scan for the line that fits "class.*(Package):", and sed script our way to victory.

This won't work because not all packages subclass directly from Package. See #543 I would encourage one of the other proposed approaches.

[DavidPoliakoff]

I have this sneaking suspicion nobody wants my eval-based solution (kidding)....

More seriously, refactoring the linked implementation so I can use it for

depends_on([something gitty]) 

shenanigans (and better code quality in the bargain)

[DavidPoliakoff]

@tgamblin @citibeth

Thoughts on the verb for "spack [verb] [github-url]" being "install"? spack install --url=[github URL]

The way it would work is that at the start of install, it would check to see if it got a url flag, if it does it would call "integrate_github_package," then call install as normal (adding the name of the spec to the list of specs to install)?

Also, in the PR we now have integrate_github_url, which is 99% of depends_on([something gitty]) being done. Anybody have a preference on how to say "[something gitty]" (I believe Todd's vote was "don't," but I still think this is reasonable).

[trws]

WRT name, thoughts on using the repository's default name? IE, llnl/raja would become "raja", or perhaps the path from root of url, so it would be "llnl_raja"?

[DavidPoliakoff]

Well, if you want the first option, I have good news for you, because it's currently "url.split("/")[-1]."

It's certainly _easy_, but I don't know that it's the right option

[trws]

I wasn't quite thinking url.split("/")[-1], but the equivalent of the directory that git would create from the repository if not given an argument, so I suppose it would be that but stripping ".git" off the end if it's there...

[citibeth]

Thoughts on the verb for "spack [verb] [github-url]" being "install"? spack install --url=[github URL]

Yes, I think that's perfect. It should work with exactly the same grammar as spack create. I believe with spack create it's --url [URL] not --url=[URL]. But I could be wrong. No matter... as long as you copy relevant parts of the arg parser from spack create, it will be consistent.

[tgamblin]

If you add a flag for URL, you haven't got a way to use a repo as a dependency, which I suspect will be a common use case. I'm kind of inclined to work it into the spec grammar where you could have a (potentially quoted) URL instead of a package. what about allowing identifiers like so:

spack install url:https://github.com/foo.git
spack  install foo ^ url:https://github.com/foo.git

[DavidPoliakoff]

@tgamblin hmmm, I'd need help if it's going into the spec grammar, but I like that idea, especially since one question I had was "how do I do 'spack install [raja github url] +cuda +omp'," or in other words "how do I install a variant from a URL?"

edit: and having it just fit into the existing grammar is a good implementation for that

[citibeth]

@tgamblin Oh, I see. Your suggestion sounds reasonable enough. The url:
shouldn't be required, since it's easy to tell a URL from a bare package
name.

On Thu, Jun 30, 2016 at 10:34 PM, Todd Gamblin [email protected]
wrote:

If you add a flag for URL, you haven't got a way to use a repo as a
dependency, which I suspect will be a common use case. I'm kind of inclined
to work it into the spec grammar where you could have a (potentially
quoted) URL instead of a package. what about allowing identifiers like so:

spack install url:https://github.com/foo.git
spack install foo ^ url:https://github.com/foo.git

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LLNL/spack/issues/1136#issuecomment-229840021, or mute
the thread
https://github.com/notifications/unsubscribe/AB1cdyFvup_mD9EmtyrDEnZETLVUiZlzks5qRHzEgaJpZM4JAjJM
.

[citibeth]

In @tgamblin's suggestion, I believe you would not have to change CMD line
parsing (i.e. it should be like the parsing for spack install --- hey,
why isn't this just an add-on to spack install?). Anyway, you should be
able to look at spack install to see how it works.

On Thu, Jun 30, 2016 at 10:50 PM, Elizabeth A. Fischer <
[email protected]> wrote:

@tgamblin Oh, I see. Your suggestion sounds reasonable enough. The
url: shouldn't be required, since it's easy to tell a URL from a bare
package name.

On Thu, Jun 30, 2016 at 10:34 PM, Todd Gamblin [email protected]
wrote:

If you add a flag for URL, you haven't got a way to use a repo as a
dependency, which I suspect will be a common use case. I'm kind of inclined
to work it into the spec grammar where you could have a (potentially
quoted) URL instead of a package. what about allowing identifiers like so:

spack install url:https://github.com/foo.git
spack install foo ^ url:https://github.com/foo.git

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LLNL/spack/issues/1136#issuecomment-229840021, or mute
the thread
https://github.com/notifications/unsubscribe/AB1cdyFvup_mD9EmtyrDEnZETLVUiZlzks5qRHzEgaJpZM4JAjJM
.

[tgamblin]

I like omitting url:, and I also think you could do this the spec grammar. Seems like you could:

1. find the URLs with a regex
2. Translate them into .@ specs.
3. concretize and run do_install on the specs like normal

I worry that inferring the name from the repository name is too inflexible. What if there are two forks in the same github org/project? You want A way to plug a URL in where other Spack packages refer to a specific name. So what if you could _optionally_ prepend a name to the URL? For example,

spack install raja:https://github.com/other-raja.git

That would force the name to raja, so Spack knows other thing that depend on raja can use it.

[tgamblin]

Finally w.r.t. (2) above, if you pick a reserved namespace you can use that to control what repo is used toget your custom Package from the concretized spec at install time. This happens when Spack traverses the spec passed to do_install -- it calls spack.repo.get() on each DAG node, which searches repos for a package with a name.

Most package names in specs start out unqualified, and they are assigned a namespace from the first repo containing the name. If you specify the namespace it forces a _particular_ repo to be used.

So, you can subclass Repo with a simple cache class that regurgitates the Package you got from loading the custom package.py. Also, when you load the package.py, you will want to set __file__ on it, as this is used to determine where to grab patches and other files the package might need. The need to grab patches is another good reason to just load the package straight out of the shallow clone directory instead of doing a fancy single-file fetch.

Hopefully that helps.

[tgamblin]

Apparently I hit close by accident. Oops.

[alalazo]

Do we want to keep this open? It seems #1151 has been closed by its author so I am wondering if somebody want to continue on this.

[citibeth]

I wouldn't mind closing.

On Tue, Jan 10, 2017 at 10:06 AM, Massimiliano Culpo <
[email protected]> wrote:

Do we want to keep this open? It seems #1151
https://github.com/LLNL/spack/pull/1151 has been closed by its author
so I am wondering if somebody want to continue on this.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LLNL/spack/issues/1136#issuecomment-271598911, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AB1cd-eVA-b3QPe8Zajf0VL8GOTyh6QSks5rQ55vgaJpZM4JAjJM
.

[adamjstewart]

Closing this as the discussion died a long time ago. The new Spack environments spack.yaml file provides some of the requested features (ability of project developers to explicitly list their dependencies) so maybe this is good enough.