Spack: Unable to build PETSc, PETSC_DIR already set

[adamjstewart]

configure fails immediately if the PETSC_DIR environment variable is already set. Spack should unset this.

===============================================================================
             Configuring PETSc to compile on your system                       
===============================================================================
TESTING: configureExternalPackagesDir from config.framework(/scratch/ajstewart/spack-stage/spack-stage-sFRYcv/petsc-3.7.4/config/BuildSystem/config/framework.py:834)
TESTING: configureDebuggers from config.utilities.debuggers(/scratch/ajstewart/spack-stage/spack-stage-sFRYcv/petsc-3.7.4/config/BuildSystem/config/utilities/debuggers.py:22)
TESTING: configureGit from config.sourceControl(/scratch/ajstewart/spack-stage/spack-stage-sFRYcv/petsc-3.7.4/config/BuildSystem/config/sourceControl.py:24)
TESTING: configureMercurial from config.sourceControl(/scratch/ajstewart/spack-stage/spack-stage-sFRYcv/petsc-3.7.4/config/BuildSystem/config/sourceControl.py:35)
TESTING: configureCLanguage from PETSc.options.languages(/scratch/ajstewart/spack-stage/spack-stage-sFRYcv/petsc-3.7.4/config/PETSc/options/languages.py:27)
TESTING: configureDirectories from PETSc.options.petscdir(/scratch/ajstewart/spack-stage/spack-stage-sFRYcv/petsc-3.7.4/config/PETSc/options/petscdir.py:23)
*******************************************************************************
         UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
-------------------------------------------------------------------------------
The environmental variable PETSC_DIR /home/ajstewart/petsc MUST be the current directory /scratch/ajstewart/spack-stage/spack-stage-sFRYcv/petsc-3.7.4
*******************************************************************************

[tgamblin]

@adamjstewart: Should we just set it to the value it says it has to be (os.getcwd()) in install(), and add it to setup_dependent_environment() for dependents?

[davydden]

there is already

    def setup_dependent_environment(self, spack_env, run_env, dependent_spec):
        # set up PETSC_DIR for everyone using PETSc package
        spack_env.set('PETSC_DIR', self.prefix)

It probably used to work because setup_dependent_environment() was executed for the package itself.
just realized that you have it set to something non-Spacky PETSC_DIR=/home/ajstewart/petsc, supposedly it works ok when PETSC_DIR is empty.

[davydden]

add this to install()

        # set PETSC_DIR for installation
        # Note that one should set the current (temporary) directory instead
        # its symlink in spack/stage/ !
        os.environ['PETSC_DIR'] = os.getcwd()

[adamjstewart]

Is this the difference between setup_environment and setup_dependent_environment? Should I add setup_environment?

[davydden]

@adamjstewart see https://github.com/LLNL/spack/issues/2016#issuecomment-258936647 Another documentation issue :-(

[davydden]

p.s. setup_dependent_environment() is for dependencies only, see http://spack.readthedocs.io/en/latest/packaging_guide.html#setup-dependent-environment

[tgamblin]

setup_environment set up this package's environment. setup_dependent_environment allows it to put things in the environment of its dependents. so if A depends on B, B can put things in setup_dependent_environment to affect A's environment.

I think you want this change:

diff --git a/var/spack/repos/builtin/packages/petsc/package.py b/var/spack/repos/builtin/packages/petsc/package.py
index 578349c..ce335ce 100644
--- a/var/spack/repos/builtin/packages/petsc/package.py
+++ b/var/spack/repos/builtin/packages/petsc/package.py
@@ -218,6 +218,7 @@ class Petsc(Package):
                         '-pc_type', 'hypre',
                         '-pc_hypre_type', 'boomeramg')

-    def setup_dependent_environment(self, spack_env, run_env, dependent_spec):
-        # set up PETSC_DIR for everyone using PETSc package
+    def setup_environment(self, spack_env, run_env):
+        # set up PETSC_DIR for everyone using PETSc package, and for the build.
         spack_env.set('PETSC_DIR', self.prefix)
+        run_env.set('PETSC_DIR', self.prefix)

As the default implementation of setup_dependent_environment just calls setup_environment.

[tgamblin]

For those interested, the code that does all this stuff is in build_environment.py, in setup_package(), which gets run after forking the build process after every build.

Note the post-oder traversal. That iterates over all the dependencies before setting up this package's environment.

def setup_package(pkg, dirty=False):                                                                                    
    """Execute all environment setup routines."""                                                                       
    spack_env = EnvironmentModifications()                                                                              
    run_env = EnvironmentModifications()                                                                                

    # Before proceeding, ensure that specs and packages are consistent                                                  
    #                                                                                                                   
    # This is a confusing behavior due to how packages are                                                              
    # constructed.  `setup_dependent_package` may set attributes on                                                     
    # specs in the DAG for use by other packages' install                                                               
    # method. However, spec.package will look up a package via                                                          
    # spack.repo, which defensively copies specs into packages.  This                                                   
    # code ensures that all packages in the DAG have pieces of the                                                      
    # same spec object at build time.                                                                                   
    #                                                                                                                   
    # This is safe for the build process, b/c the build process is a                                                    
    # throwaway environment, but it is kind of dirty.                                                                   
    #                                                                                                                   
    # TODO: Think about how to avoid this fix and do something cleaner.                                                 
    for s in pkg.spec.traverse():                                                                                       
        s.package.spec = s                                                                                              

    set_compiler_environment_variables(pkg, spack_env)                                                                  
    set_build_environment_variables(pkg, spack_env, dirty)                                                              
    pkg.architecture.platform.setup_platform_environment(pkg, spack_env)                                                
    load_external_modules(pkg)                                                                                          
    # traverse in postorder so package can use vars from its dependencies                                               
    spec = pkg.spec                                                                                                     
    for dspec in pkg.spec.traverse(order='post', root=False, deptype='build'):                                          
        # If a user makes their own package repo, e.g.                                                                  
        # spack.repos.mystuff.libelf.Libelf, and they inherit from                                                      
        # an existing class like spack.repos.original.libelf.Libelf,                                                    
        # then set the module variables for both classes so the                                                         
        # parent class can still use them if it gets called.                                                            
        spkg = dspec.package                                                                                            
        modules = parent_class_modules(spkg.__class__)                                                                  
        for mod in modules:                                                                                             
            set_module_variables_for_package(spkg, mod)                                                                 
        set_module_variables_for_package(spkg, spkg.module)                                                             

        # Allow dependencies to modify the module                                                                       
        dpkg = dspec.package                                                                                            
        dpkg.setup_dependent_package(pkg.module, spec)                                                                  
        dpkg.setup_dependent_environment(spack_env, run_env, spec)                                                      

    set_module_variables_for_package(pkg, pkg.module)                                                                   
    pkg.setup_environment(spack_env, run_env)                                                                           

    # Make sure nothing's strange about the Spack environment.                                                          
    validate(spack_env, tty.warn)                                                                                       
    spack_env.apply_modifications()    

[davydden]

As the default implementation of setup_dependent_environment just calls setup_environment.

did not know that.

[adamjstewart]

Unfortunately, none of the suggested changes work.

During installation, PETSC_DIR must be set to the current directory from which configure is run. This is not self.prefix. So we will need a separate setup_environment and setup_dependent_environment.

Getting the current directory isn't so easy though. PETSc is expecting /scratch/ajstewart/spack-stage/spack-stage-SnQEin/petsc-3.7.4. Here is what the following return:

os.getcwd() in setup_environment:
/blues/gpfs/home/software/spack-0.10.0

self.stage.path:
/blues/gpfs/home/software/spack-0.10.0/var/spack/stage/petsc-3.7.4-2wq3at7rf2z374pd6ppibyna2idltsee

So unless there is some other way, I think I'll just try unsetting it.

[davydden]

@adamjstewart try in install (see above https://github.com/LLNL/spack/issues/2767#issuecomment-271030405)

        # set PETSC_DIR for installation
        # Note that one should set the current (temporary) directory instead
        # its symlink in spack/stage/ !
        os.environ['PETSC_DIR'] = os.getcwd()

this/similar is already done in SLEPC and does work.

[adamjstewart]

Yes, that seems to work. Do we like this solution better than unsetting it? If it is already in use then I'm fine with it.

[davydden]

Do we like this solution better than unsetting it? If it is already in use then I'm fine with it.

i think there is no harm to set it to what it actually should be for all current versions of PETSc, IIRC.

[tgamblin]

@adamjstewart: you should probably set it in the run env within setup_environment, and in run and spack_envs in setup_dependent_environment. Those three affect what dependents see in spack and what users of this package and dependents see in modules.

spack_env in setup_environment is just there for consistency -- using using os.environ is effectively the same thing as calling methods on it.

Since you need the out-of-source build dir at build time, you could just set it as @davydden suggests inside install():

with working_dir(blah):
    env['PETSC_DIR'] = os.getcwd()

In this case, though, unsetting it in setup_environment might be clearer and easier... maybe @BarrySmith has a preference?

[tgamblin]

PS - if anyone has suggestions for how to make this clearer, e.g. names of routines, spack_env vs run_env, let me know. Each has a distinct purpose, but the differences are subtle.

[BarrySmith]

I would actually unset both PETSC_DIR and PETSC_ARCH during the install (in case the user had them previously set), they are not needed and if you do set PETSC_DIR it will look like there is a reason to set it confusing people who look at the code in the future.

For usage I would set the PETSC_DIR as it has always been done and unset PETSC_ARCH, again in case someone had it set before (PETSC_ARCH is only useful for installing without --prefix) and since spack always uses --prefix it should not be set.

[adamjstewart]

Ok, now this is getting confusing. Would this look like:

• install()
  
  
  
  • unset PETSC_DIR
  
  
  • unset PETSC_ARCH
  
  
• setup_environment()
  
  
  
  • set PETSC_DIR in run_env
  
  
  • unset PETSC_ARCH in run_env?
  
  
• setup_dependent_environment()
  
  
  
  • set PETSC_DIR in spack_env
  
  
  • unset PETSC_ARCH in spack_env?
  
  

Does run_env.unset work with module files as you would expect? Like does it add the unset command or does it just not include it in the module file?

P.S. What a pitifully fragile build system...

[adamjstewart]

PS - if anyone has suggestions for how to make this clearer, e.g. names of routines, spack_env vs run_env, let me know. Each has a distinct purpose, but the differences are subtle.

I don't have any great suggestions. It's pretty bad as it is now. I am starting to vaguely understand the difference between setup_environment and setup_dependent_environment, but don't quite yet understand the difference between run_env and spack_env. Is run_env what gets put into modules and spack_env what gets set when running Spack? I haven't read the documentation on these in a while, so maybe my lack of understanding is my own fault.

[tgamblin]

Is run_env what gets put into modules and spack_env what gets set when running Spack?

yes 😄

[tgamblin]

Maybe this is a better explanation: if A depends on B, C, and D, and D depends on E, then setup_dependent_environment gets run for E, D, C, and B before the build for A.

[adamjstewart]

Does this look good?

    def install(self, spec, prefix):                                            
        # configure fails if these env vars are set outside of Spack            
        os.environ.pop('PETSC_DIR')                                             
        os.environ.pop('PETSC_ARCH') 
...
    def setup_environment(self, spack_env, run_env):                            
        # Set PETSC_DIR in the module file                                      
        run_env.set('PETSC_DIR', self.prefix)                                   
        run_env.unset('PETSC_ARCH')                                             

    def setup_dependent_environment(self, spack_env, run_env, dependent_spec):  
        # set up PETSC_DIR for everyone using PETSc package                     
        spack_env.set('PETSC_DIR', self.prefix)                                 
        spack_env.unset('PETSC_ARCH')   

or should I run spack_env.unset in setup_environment instead?

[BarrySmith]

install()
unset PETSC_DIR
unset PETSC_ARCH
setup_environment()
set PETSC_DIR in run_env
unset PETSC_ARCH in run_env?
setup_dependent_environment()
set PETSC_DIR in spack_env
unset PETSC_ARCH in spack_env?

I think this is right. If spack completely cleans out the environmental variables before starting the different modes then the unset are not needed. The problem comes about from people who are or have worked in the PETSC_DIR/PETSC_ARCH model and switching to the Spack model but don't clean up their environment (which will be everyone including me).

[adamjstewart]

Ugh, my code crashes if the variable isn't set. Time for an if-statement.

EDIT: Nvm, using spack_env.unset in setup_environment because we already check for unset variables.

[adamjstewart]

run_env.unset doesn't seem to do anything:

==> Warning: Cannot handle command of type <class 'spack.environment.UnsetEnv'>: skipping request
==> Warning: run_env.unset('PETSC_ARCH') at /blues/gpfs/home/software/spack-0.10.0/var/spack/repos/builtin/packages/petsc/package.py:232

[tgamblin]

@alalazo

[adamjstewart]

I found the following in the module file though:

setenv PETSC_DIR "/blues/gpfs/home/software/spack-0.10.0/opt/spack/linux-centos6-x86_64/gcc-6.1.0/petsc-3.7.4-2wq3at7rf2z374pd6ppibyna2idltsee"
unsetenv PETSC_ARCH

doesn't that mean it worked correctly?

[adamjstewart]

Maybe it only works for modules? This is what I found in the dotkit file:

dk_setenv PETSC_DIR /blues/gpfs/home/software/spack-0.10.0/opt/spack/linux-centos6-x86_64/gcc-6.1.0/petsc-3.7.4-2wq3at7rf2z374pd6ppibyna2idltsee

No unsetting done here.

[tgamblin]

I suspect it's a warning from the dotkit hook. So safe to ignore.

[adamjstewart]

No other packages uses this, so it's uncharted territory 😄

[adamjstewart]

Ok, #2768 has been updated with the latest and greatest.

[python152]

please excuse me for opening this old issue - I am still having trouble with PETSC_ARCH settings after spack install petsc. Please see the following snipets

-- Checking PETSc ...
CMake Error at /home/ubuntu/GridPACK/cmake-jedbrown/FindPETSc.cmake:151 (message):
  The pair
  PETSC_DIR=/home/ubuntu/spack/opt/spack/linux-ubuntu20.04-haswell/gcc-9.3.0/petsc-3.13.4-2bjevskgval6putbionc5ehmxeidsvnn
  PETSC_ARCH= do not specify a valid PETSc installation
Call Stack (most recent call first):
  CMakeLists.txt:280 (find_package)


-- Using PETSc version 3.13.4
CMake Error at CMakeLists.txt:290 (message):
  PETSc found, but CMake configuration for PETSc installation not found?

There seem to be two issues:

• PETSC_ARCH is not set
• it is trying to locate ${PETSC_DIR}/${PETSC_ARCH}/conf/PETScConfig.cmake, and I am guessing this file was not generated due to missing PETSC_ARCH??

Thanks
Oliver

[adamjstewart]

^ @balay @BarrySmith @jedbrown can one of you take a look at this?

[balay]

For one this is unrelated to #2767

PETSC_DIR=/home/ubuntu/spack/opt/spack/linux-ubuntu20.04-haswell/gcc-9.3.0/petsc-3.13.4-2bjevskgval6putbionc5ehmxeidsvnn
PETSC_ARCH= do not specify a valid PETSc installation

This is correct for a spack install of petsc [which uses --prefix]. PETSC_ARCH is not used [and should be empty if set]

it is trying to locate ${PETSC_DIR}/${PETSC_ARCH}/conf/PETScConfig.cmake, and I am guessing this file was not generated due to missing PETSC_ARCH??

I don't see PETScConfig.cmake in current installs of PETSc. Our cmake support is limited. Current recommendation is to use pkgconfig file PETSc.pc

Wrt FindPETSc.cmake - @jedbrown needs to look at this.