Spack: `spack setup` is just plain broken

[citibeth]

spack setup does not work. For posterity, here is a relatively simple example of it not working. Looking inside setup.py, I barely recognize what I see:

1. spack setup was originally derived directly from the spack diy code; but now the two look nothing alike.

2. spack setup ends up concretizing the spec twice. When specs take about a minute to concretize, that can be a serious problem. It could also be a cause of the problem below.

3. It looks like changes were made to install.py without corresponding changes to setup.py (or diy.py).

I'll see what I can do; this Issue is posted to document that whatever fix I come up with actually solved a problem.

$ spack -d setup everytrace@develop
==> WRITE LOCK: /home2/rpfische/spack5/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> Reading config file /home2/rpfische/.spack/compilers.yaml
==> READ LOCK: /home2/rpfische/.spack/cache/providers/.builtin-index.yaml.lock[0:0] [Acquiring]
==> READ LOCK: /home2/rpfische/.spack/cache/providers/.builtin-index.yaml.lock[0:0] [Released]
==> WRITE LOCK: /home2/rpfische/spack5/opt/spack/.spack-db/lock[0:0] [Released]
==> Error: The spec 'm4' is configured as not buildable, and no matching external installs were found
Traceback (most recent call last):
  File "/home2/rpfische/spack5/bin/spack", line 175, in main
    return_val = command(parser, args)
  File "/home2/rpfische/spack5/lib/spack/spack/cmd/setup.py", line 178, in setup
    install.install(parser, inst_args)
  File "/home2/rpfische/spack5/lib/spack/spack/cmd/install.py", line 317, in install
    specs = spack.cmd.parse_specs(args.package, concretize=True)
  File "/home2/rpfische/spack5/lib/spack/spack/cmd/__init__.py", line 108, in parse_specs
    spec.concretize()  # implies normalize
  File "/home2/rpfische/spack5/lib/spack/spack/spec.py", line 1505, in concretize
    self._expand_virtual_packages(),
  File "/home2/rpfische/spack5/lib/spack/spack/spec.py", line 1422, in _expand_virtual_packages
    spec)
  File "/home2/rpfische/spack5/lib/spack/spack/concretize.py", line 122, in choose_virtual_or_external
    candidates = self._valid_virtuals_and_externals(spec)
  File "/home2/rpfische/spack5/lib/spack/spack/concretize.py", line 91, in _valid_virtuals_and_externals
    raise NoBuildError(spec)
spack.concretize.NoBuildError: The spec 'm4' is configured as not buildable, and no matching external installs were found

See here that spack install works on this package:

$ spack install everytrace@develop
==> Installing everytrace
==> openmpi is already installed in /home2/rpfische/spack5/opt/spack/linux-centos7-x86_64/gcc-4.9.3/openmpi-1.10.1-2ew2gg7hvoc7vj7fc4s6c3mdfeoblp76
==> cmake is already installed in /home2/rpfische/spack5/opt/spack/linux-centos7-x86_64/gcc-4.9.3/cmake-3.7.1-nscyuby65ynomqwybkldstqgcxqzxipp
==> Fetching file:///home/rpfische/.spack/manual_mirror/everytrace/everytrace-develop.tar.gz
curl: (37) Couldn't open file /home/rpfische/.spack/manual_mirror/everytrace/everytrace-develop.tar.gz
==> Fetching from file:///home/rpfische/.spack/manual_mirror/everytrace/everytrace-develop.tar.gz failed.
==> Trying to clone git repository:
  https://github.com/citibeth/everytrace.git
  on branch develop
Cloning into 'everytrace'...
remote: Counting objects: 34, done.
remote: Compressing objects: 100% (31/31), done.
remote: Total 34 (delta 2), reused 23 (delta 1), pack-reused 0
Unpacking objects: 100% (34/34), done.
==> No checksum needed when fetching with git
==> Already staged everytrace-develop-gyeos66xokjpztn4ez7cgmkzgftvtqfh in /home2/rpfische/spack5/var/spack/stage/everytrace-develop-gyeos66xokjpztn4ez7cgmkzgftvtqfh
==> No patches needed for everytrace
==> Building everytrace [CMakePackage]
==> Executing phase : 'cmake'
...

[tgamblin]

@citibeth: sounds like spack setup needs regression tests.

[citibeth]

I'm not sure if everytrace/package.py has made it into Spack. For those who wish to reproduce this behavior, it is attached here.
everytrace_package.py.txt

[citibeth]

@alalazo This issue is addressed by #2664. It can be closed, if having an open PR on the issue is sufficient.