Picongpu: First install

[cbontoiu]

Hello, for a first configuration of the PIConGPU code I purchased the RTX2060 graphics card and installed it on pretty old machine: Intel Xeon CPU X5660 @ 2.80 GHz x 6 running Linux Mint 19.2 with 16 GB of RAM. The idea is to learn the software, understand the limits of this machine and then run it on a cluster of graphics cards. After spending day of configuring the dependencies I ended up failing to source the picongpu.profile due to the module load commands and implicitly I couldn't build the examples, so I am restarting all the process from zero on a fresh OS install. So far I have only installed the graphics card driver and downloaded the Spack folder.
Would you please answer these questions:

1. What is the best approach to get all dependencies installed?
2. How should I edit my picongpu.profile? Should I start from the K20x GPUs (recommended) version?
3. Should I install CUDA 10 first of all?
4. Should I install CUDA aware opnempi or should I let Spack do it as shown here https://picongpu.readthedocs.io/en/0.4.3/install/instructions/spack.html ?

Many thanks for your attention and help. Best Regards! Cristian

[sbastrakov]

Hello @cbontoiu , thanks for your report. If I understood correctly, you tried two ways of installing PIConGPU: first manually from source and second via Spack. Sorry both did not work out smoothly for you, hopefully, the following explanations help.

With installing manually the unfortunate part is that the exact flags are system-dependent. (This is probably too late, but our documentation suggests reasonable flags to try. Please note that many dependencies are optional, and so very reasonable to skip for the first installation to save time; one can always install it later.). Our .profile files are just a way to load environment settings for PIConGPU, so that all our tools are available (pic-create, pic-build, etc.) and all dependencies are found during compilation. These files are also system-specific, so you would need to adapt an existing one, but we are willing to help with that. Most of .profile files in our repository rely on the Modules package since this is what most clusters use. However, this is not required. I think for you it is reasonable to just add paths to environment variables. For example, take a look at this profile. The system provides a module for Boost, so we just load it here. However, for blosc (note that this one is optional, this is just as an example) there was no module, so that library has been installed manually and in the profile we just add the paths to environment variables here. This is probably the simplest way for you to add Boost and all other dependencies this way in your machine. Please note that as part of the pic-build output it will say if Boost and other dependencies are found, and complain if something required is not found.

Installing CUDA 10: makes good sense to do early, since then other CUDA-aware software configuration might depend on whether CUDA was found. (PIConGPU can also run on CPU without CUDA, but I guess this is not what you want with the given hardware).

Spack ideally takes care of everything and so in the best case you do not need anything specific. There is also a way for you to tell it to use your existing package. In case you encounter issues, please feel free to report further here and/or on their github.

[cbontoiu]

Hi Sergei,

Thanks so much for your reply. I am trying to install the code again following your observations. So far I installed CUDA 10 and CUDA-aware opempi 3. the biggest issue that I have now is working with the module command and I need to fix this before proceeeding to PIConGPU again. Cuda comes with some samples to compile. Some of them rely on mpi and it seems that setting the path to the openmpi/bin installation is not enough. They ask users to type " module load mpi/openmpi-x86_64" which is common to what i have seen in the picongpu profile. So I followed this guidelines https://askubuntu.com/questions/343692/module-load-command-does-not-work/343721 and ended up with a modified bashrc file

case "$0" in
          -sh|sh|*/sh)  modules_shell=sh ;;
       -ksh|ksh|*/ksh)  modules_shell=ksh ;;
       -zsh|zsh|*/zsh)  modules_shell=zsh ;;
    -bash|bash|*/bash)  modules_shell=bash ;;
esac

#module() { eval `/usr/bin/tclsh /usr/lib/x86_64-linux-gnu/modulecmd.tcl $modules_shell $*`; }
module() { eval `/usr/bin/modulecmd $modules_shell $*`; }

and a modules.sh file places in etc/profile.d :

shell=$(/usr/bin/basename $(/bin/ps -p $$ -ocomm=))
if [ -f /usr/share/modules/init/$shell ]; then
   . /usr/share/modules/init/$shell
else
   . /usr/share/modules/init/sh
fi

There is an error however as shown below:

cristi@T7500:/usr/local/cuda-10.1/samples$ source ~/.bashrc 
cristi@T7500:/usr/local/cuda-10.1/samples$ module load mpi/openmpi-x86_64
/usr/bin/modulecmd: 27: exec: /usr/lib/x86_64-linux-gnu/modulecmd.tcl: not found

saying something about the /usr/bin/modulecmd file

if [ "$MODULES_USE_COMPAT_VERSION" = '1' ]; then
   cmd=/usr/lib/x86_64-linux-gnu/modulecmd-compat
else
   cmd=/usr/lib/x86_64-linux-gnu/modulecmd.tcl
fi
exec "$cmd" "$@"

My system is linux mint 19. I am trying hard to unsertands what do I have to do. Maybe you can help me please?
Best Reagrds,
Cristian

[sbastrakov]

Thanks @cbontoiu for providing more details.
Sorry for a super naive suggestion: can you please check that you have the modules package installed? I am asking since your system seems to miss /usr/lib/x86_64-linux-gnu/modulecmd.tcl, so either the package is not installed or maybe it is somewhere else and so not found.

@psychocoderHPC in case you have any idea, this is welcome.

[cbontoiu]

Hi Sergei,

I finished installing picongpu using spack and when loading I get this error

cristi@T7500:~$ spack load picongpu
ERROR: Unable to locate a modulefile for 'picongpu-0.4.3-gcc-7.4.0-4gnjuvs'

Do you have any idea on what happens here?
Many thanks and best regards,

Cristian

[ax3l]

Hi Christian,

awesome, you are nearly there! The module file for picongpu in spack is missing since it did not have access to environment modules when you first installed the picongpu package.
You can just run spack module tcl refresh to build the missing modules.

(Since you now have environment-modules available, newly installed packages will automatically install modules alongside them.)

Just for curiosity: did you run spack bootstrap on first setup of spack?
https://picongpu.readthedocs.io/en/latest/install/instructions/spack.html

[cbontoiu]

Hi Axel,

Thank you for your reply! Yes I followed all indications in the right order, including the bootstrap. With your advice, after getting a new terminal it works and I can see the output of info picongpu. I got the example myLWFA finished in 1 min 43 sec. I am not sure if this is reasonable time. Now i need to learn the postprocessing part. Cheers and I appreciate your help. Cristian

[sbastrakov]

This looks good. If you take a look into an output directory, there should be simOutput directory which (for our example LWFA setup) has an internal file e_macroparticlesCount.dat and subdirectories for png output

[cbontoiu]

Hello guys,

I tried to run the bremsstrahlung example following the same procedures as for the laser wakefield acceleration example. If in the latter case pic-build gave me a 1.cfg file wich I could successfully run, the same command give me now a 8.cfg file; I assume it is for 8 GPUs. Trying to run with
tbg -s bash -c etc/picongpu/8.cfg -t etc/picongpu/bash/mpiexec.tpl $SCRATCH/picRuns/brem_001
yields this error
ERROR: CUDA error: invalid device ordinal
I also got a reference to a log file which contains
Please let me know what can I do in this case.
Best Regards,
Cristian

[ax3l]

Hi,

are you running on a machine with only one GPU? In that case, just edit the .cfg file to use only one device:

TBG_devices_x=1
TBG_devices_y=1
TBG_devices_z=1

Looks like the cuda memory check that is run reports exactly that: it cannot access a GPU (likely GPUs 2 to 8 are not found on the machine :) ).
https://picongpu.readthedocs.io/en/latest/usage/tbg.html#cfg-file-macros

Potentially, you also have to shrink the domain size in case you are running out of memory (TBG_gridSize).

@PrometheusPi can you potentially provide a 1.cfg file for that example case as well?

[PrometheusPi]

@ax3l Yes, I will provide a 1.cfg for the Bremsstrahlung example.

[PrometheusPi]

@cbontoiu Please be aware that while writing the 1.cfg I discovered a miss-configuration in the 8.cfg that might effect you as well when you adjust the Bremsstrahlung example. (see #3096)

[cbontoiu]

Yes, this has happened already and I stopped trying to run the example for now. Thank you.

[ax3l]

fixed in #3097, thanks @PrometheusPi !
Here is the small change you will have to apply to the .cfg file, sorry for the hiccup:
https://github.com/ComputationalRadiationPhysics/picongpu/pull/3097/files

[PrometheusPi]

@cbontoiu a working (tested) 1.cfg is introduced with #3098.

[cbontoiu]

I try to visualize some h5 which are supposed to contain phase space data for example. Because I don't know how to open the h5 format directly in Paraview (Version 5.7) which I installed, I try to follow the tutorial

https://github.com/ComputationalRadiationPhysics/picongpu/wiki/ParaView

without connecting to the server indicated; I don't yet see the benefit of doing so.

With a different example, now asking for all species and all fields in my cfg file:

TBG_hdf5="--hdf5.period 100 --hdf5.file simData --hdf5.source 'species_all,fields_all'" 
TBG_plugins=" !TBG_e_pngYX !TBG_eBin !TBG_e_PSxpx !TBG_EField_slice !TBG_macroCount !TBG_hdf5" 

I succeed in getting a h5 folder with simData....h5 files.
So, I located pic2xdmf.py as export PIC_TOOLS=/home/cristi/src/spack/opt/spack/linux-linuxmint19-westmere/gcc-7.4.0/picongpu-0.4.3-4gnjuvssjgn5gjjmsbdzcf5zbc623hii/src/tools/bin
but running something like (Python version 2.7)

python pic2xdmf.py -t /home/cristi/picRuns/bunch_028/simOutput/h5

results in this error

Traceback (most recent call last):
  File "pic2xdmf.py", line 26, in <module>
    import splash2xdmf
  File "/home/cristi/src/spack/opt/spack/linux-linuxmint19-westmere/gcc-7.4.0/picongpu-0.4.3-4gnjuvssjgn5gjjmsbdzcf5zbc623hii/src/tools/bin/splash2xdmf.py", line 26, in <module>
ImportError: No module named h5py

What could it be? I assume I have to import a library but moving a copy of /home/cristi/src/spack/opt/spack/linux-linuxmint19-westmere/gcc-7.4.0/libsplash-1.7.0-qxwrzgnlvzxld7aubo2o4ady3oattycb/bin/splash2xdmf.py into the PIC_TOOLS did not help. Any suggestion is greatly appreciated. Thank you. Regards.

[PrometheusPi]

@cbontoiu Your python installation is missing the h5py module. If you are using conda, you can install it via e.g. conda install -c anaconda h5py or sudo apt-get install libhdf5-serial-dev and pip instal h5py if you installed python via the package manger.

[ax3l]

Using ParaView for the phase space data is a bit too complicated. ParaView is a complex program to set up and good for 3D rendering data while our phase space plugin creates 2D projected data.
In case you are already using the PIConGPU development branch, the easiest way to visualize the phase space plugin's data is by just using our python bindings for it.

The generally also just depend on h5py, matplotlib, numpy, scipy, pandas and the like:
all requirements, plugins only, plugin data reader only

conda install -c anaconda h5py numpy pandas scipy matplotlib will do this :)

[cbontoiu]

Hello,

I am trying to setup an independent install of picongpu aprat from the one used through spack. I followed all instructions and now i try to configure splash2txt. This tool looks for adios which I have installed both as sudo apt-get install libadios-dev libadios-bin and trough compilation from source. In the first case the library is placed automatically in /usr/lib/x86_64-linux-gnu and in the other case /opt/adios/lib as requested. Accordingly my .bashrc file contains
export ADIOS_ROOT=/opt/adios

export LD_LIBRARY_PATH=/opt/adios/lib:$LD_LIBRARY_PATH

export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/opt/adios/lib/"
However, the command
~/picongpu$ cmake -DCMAKE_INSTALL_PREFIX=$PICSRC/src/tools/bin $PICSRC/src/tools/splash2txt
returns:
-- Building Debug
-- HDF5: Using hdf5 compiler wrapper to determine C configuration
-- Found Splash: /opt/splash/lib/cmake/Splash
-- Found 'adios_config': /usr/bin/adios_config
-- ADIOS linker flags (unparsed): -L/usr/lib/x86_64-linux-gnu -ladios -L/usr/lib/x86_64-linux-gnu/hdf5/openmpi -L/usr/lib -L/usr/lib64 -L/usr/lib64 -L/usr/lib -lhdf5_hl -lhdf5 -lz -libverbs -lglib-2.0 -lz -lbz2 -lsz -Wl,-Bsymbolic-functions -Wl,-z,relro -lm -lpthread
-- ADIOS: Could NOT find library 'inux-gnu'
-- Found adios in /usr/lib/x86_64-linux-gnu/libadios.a
-- ADIOS: Could NOT find library 'inux-gnu'
-- Found hdf5_hl in /usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5_hl.so
-- Found hdf5 in /usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5.so
-- Found z in /usr/lib/x86_64-linux-gnu/libz.so
-- Found ibverbs in /usr/lib/x86_64-linux-gnu/libibverbs.so
-- Found glib-2.0 in /usr/lib/x86_64-linux-gnu/libglib-2.0.so
-- Found z in /usr/lib/x86_64-linux-gnu/libz.so
-- ADIOS: Could NOT find library 'bz2'
-- Found sz in /usr/lib/x86_64-linux-gnu/libsz.so
-- Found m in /usr/lib/x86_64-linux-gnu/libm.so
-- Found pthread in /usr/lib/x86_64-linux-gnu/libpthread.so
-- Could NOT find ADIOS: Found unsuitable version "", but required is at least "1.13.1" (found )
-- Boost version: 1.65.1
-- Found the following Boost libraries:
-- program_options
-- regex
-- Configuring done
-- Generating done
-- Build files have been written to: /home/cristi/picongpu
which means that Cmake takes by default the first adios location and fails. How can I force it to use my path to adios?

[ax3l]

The problem you see with ADIOS1 installed via apt-get is Ubuntu/Debian specific and a bug that is fixed in the latest version of those distributions (Debian: buster and newer; Ubuntu: disco and newer). What does lsb_release -a output for you?

You can manually fix this bug by editing /usr/bin/adios_config, e.g. via sudo nano /usr/bin/adios_config and exchange the typo in line 83:

-            CFLAGS=`pkg-config adiso --variable ADIOS_SEQ_INC`
+            CFLAGS=`pkg-config adios --variable ADIOS_SEQ_INC`

and adding around line 38

+VERSIONSTRING="1.13.1"

You also will have to install sudo apt-get install pkg-config libbz2-dev. Maybe there are more issues that are not in our control with those system packages.

This tool looks for adios which I have installed both as sudo apt-get install libadios-dev libadios-bin and trough compilation from source.

That said, we really recommend to rather install all dependencies manually, as described here (you can really just copy and paste what we wrote there) and here for general instructions, or all via spack as described here. Do not mix installation methods as they will be build with various different compiler versions and flags which will lead to incompatibilities and hard to debug issues down the road.

Best thing is probably to already clean up via sudo apt-get remove libhdf5* libadios* *openmpi* *mpich* which will avoid further confusion.

How can I force it to use my path to adios?

When you are building from source and install in non-system-path locations you have to somehow tell your build system where to find these dependencies. The instructions above list that you have to set environment hints by export-ing variables such as PATH, LD_LIBRARY_PATH and ADIOS_ROOT in the case of ADIOS1.
We usually group such hints into a single picongpu.profile file that we store in our $HOME directories and activate via source $HOME/picongpu.profile when we want to use PIConGPU. We list examples for many systems and hpc centers here and here is a block of self-installed software on one of those systems for illustration.

That said,

spack install libsplash
spack install adios
spack load -r libsplash
spack load -r adios

will provide the dependencies of splash2txt and automates these workflows :)

Last but not least: splash2txt is just a legacy data converter of ours, command line tools such as h5dump and bpdump do a similar job of converting files to text and come with hdf5 and adios. If you are a Python programmer, you can alternatively just use h5py, python-adios or openPMD-api to read our HDF5 & ADIOS files. That's what most of us do for post-processing.

[cbontoiu]

Hi Axel,

Thanks for your careful advices. I really appreciate. I will try to tidy up my system tomorrow. The lsb_release -a gives me this

No LSB modules are available.
Distributor ID: LinuxMint
Description: Linux Mint 19.2 Tina
Release: 19.2
Codename: tina

Spack is like an operating system by itself and what confuses me is that I have the same software installed in both the spack folder and on the operating system (I know this can be controlled with the .yaml file though). This is why I was trying to get picongpu running without spack and then simply delete the spack folder. I have completed all of the steps as indicated and this one with adios and spash2txt was the last one remaining.

Regards.
Cristian

[ax3l]

Hi Cristian,

don't worry, you are doing great here.

I have the same software installed in both the spack folder and on the operating system

That's actually quite common and the only thing we have to take care of, with any package manager, is to only depend on software from the same install method / package manager. We do that via the mentioned environment variable hints which will just select one over the other and all is fine :)

We often just install software directly in our home directories ("manual from source") but Spack can make your life easier by automating the install (also in your home dirs) and setting the required environment hints via spack load. Maybe the tutorial here also helps a bit to see what it does.

Best,
Axel

[cbontoiu]

Hello,

I succeeded in installing picongpu and all or most of the dependencies using Spack (only). The program runs fine the LWFA example. There are two issues for which I would be grateful if you could give me some help.

1) about ADIOS -----------------------------

ADIOS was also installed as:

spack install adios

Still when trying to optimize with pic-configure:

pic-configure -b "cuda:75" $HOME/PIC_INPUTS/myLWFA

I get an error about ADIOS and implicetly it is not used at runtime. This is how it looks. There must be a mistake somwhere:

-- Found 'adios_config': /home/cristi/src/spack/opt/spack/linux-linuxmint19-westmere/gcc-7.4.0/adios-1.13.1-nilxfbevrqqz4kda6iwzl53sc7pzvgml/bin/adios_config
-- ADIOS linker flags (unparsed): -L/home/cristi/src/spack/opt/spack/linux-linuxmint19-westmere/gcc-7.4.0/adios-1.13.1-nilxfbevrqqz4kda6iwzl53sc7pzvgml/lib -ladios -L/home/cristi/src/spack/opt/spack/linux-linuxmint19-westmere/gcc-7.4.0/zlib-1.2.11-tvzbhwk6syacjk3x3nnlrafzjcketwpn/lib64 -L/home/cristi/src/spack/opt/spack/linux-linuxmint19-westmere/gcc-7.4.0/lz4-1.9.2-bxzebyocmj3qzhrm3q4otvubsgl3xn2d/lib -L/home/cristi/src/spack/opt/spack/linux-linuxmint19-westmere/gcc-7.4.0/c-blosc-1.17.0-a7xszrmgden3kvgqa4korhcdeqv6znfx/lib -L/home/cristi/src/spack/opt/spack/linux-linuxmint19-westmere/gcc-7.4.0/zfp-0.5.5-kstv4llt3untcedqozlh363knna5me6o/lib -L/home/cristi/src/spack/opt/spack/linux-linuxmint19-westmere/gcc-7.4.0/sz-1.4.12.3-3icwnlen2gscvrrudjnnjsh65fxmurh7/lib -lz -llz4 -lblosc -lzfp -lSZ -lzlib -lpthread -lm
-- ADIOS: Could NOT find library 'inuxmint19-westmere'
-- Found adios in /home/cristi/src/spack/opt/spack/linux-linuxmint19-westmere/gcc-7.4.0/adios-1.13.1-nilxfbevrqqz4kda6iwzl53sc7pzvgml/lib/libadios.a
-- ADIOS: Could NOT find library 'inuxmint19-westmere'
-- ADIOS: Could NOT find library 'inuxmint19-westmere'
-- ADIOS: Could NOT find library 'inuxmint19-westmere'
-- ADIOS: Could NOT find library 'inuxmint19-westmere'
-- ADIOS: Could NOT find library 'inuxmint19-westmere'

2) about ISAAC -----------------------------------

I get this warning when trying to optimize with pic-configure:
-- ISAAC was found, but detected the following problems:missing dependencies:
-- IceT

Thank you
Cristian

[sbastrakov]

Hello @cbontoiu .

@ax3l definitely knows more, but I believe ISAAC is not part of the spack package. So I think in case you need it, would need to install it separately. It is an in-situ visualization tool, so in my opinion might be not the first priority thing, and also requires some effort to compile and configure. But of course you decide if it's worth your time and effort. The source code of ISAAC is here.

[sbastrakov]

Oh please ignore my previous message about ISSAC not in spack. spack install isaac should work.

[cbontoiu]

Hello,

Thanks for your reply. Both ADIOS and ISAAC are already installed with spack. My problem is that I don't know how to enable them such that running pic-configure overcomes the issues shown above.

Also I don't know how to activate ADIOS and ISAAC because running spack info picongpu give me this:

Variants:
Name [Default] Allowed values Description

adios [off]       True, False       Enable the ADIOS plugin
backend [cuda]    cuda, omp2b       Control the computing backend
cudacxx [nvcc]    nvcc, clang       Device compiler for the CUDA
                                    backend
hdf5 [on]         True, False       Enable multiple plugins
                                    requiring HDF5
isaac [off]       True, False       Enable the ISAAC plugin
png [on]          True, False       Enable the PNG plugin

[sbastrakov]

Ah, sorry, my misunderstanding. I suspect the issue is somehow related to the -linuxmint19-westmere part of the path, which is for some reason interpreted as -l inuxmint19-westmere and thus causes issues. Perhaps quotes are missing somewhere, but I am not sure where to make a fix.

[cbontoiu]

I am trying to compile the data_reader.py script and I get this error
Traceback (most recent call last):

  File "data_reader.py", line 9, in <module>
    ps_data.get_times(ps="xpx", species="e", species_filter='all')
AttributeError: 'PhaseSpaceData' object has no attribute 'get_times'

Is there something I need to do in my cfg file such that time is stored?

[PrometheusPi]

@cbontoiu Could you please tell me, where you found data_reader.py or if you wrote it yourself, what you used as reference.
The phase space module has no method .get_times only .get_iterations (which is time in units of time steps).
Thus replacing both method names should solve the issue you encountered.

Please be aware that we just recently fixed an inconsistency of the module documentation you are currently using (but you used the correct arguments already.) see #3110

[cbontoiu]

Thank you! I got the script from here
https://picongpu.readthedocs.io/en/0.4.3/usage/plugins/phaseSpace.html

[PrometheusPi]

I was a bit confused: I could not find get_times on the web page above (documentation for version 0.4.3). I only found it on the latest version (dev branch): https://picongpu.readthedocs.io/en/latest/usage/plugins/phaseSpace.html

(The method get_times was only introduced with #2779 and is currently only part of the dev branch.)

Might it be, that you accidentally accessed the latest documentation instead of the 0.4.3 version?

[ax3l]

ADIOS find issue: PIConGPU 0.4.3 (and dev) uses an outdated version of our FindADIOS.cmake module. Sorry for that, we fix that ASAP. In the meantime, please copy the content of this file to $PICSRC/thirdParty/cmake-modules/FindADIOS.cmake to fix this temporarily, until we update our dev branch with the changes in https://github.com/ComputationalRadiationPhysics/cmake-modules/pull/19
Update: PIConGPU dev (upcoming 0.5.0 release) will be fixed as soon as #3116 is merged.

ISAAC: as you already found out, this is also part of Spack mainline and part of our Spack PIConGPU package as well. It is not enabled by default since you will not need it unless you want to do 3D in situ visualization with it and it otherwise just increases your compile time. The note you see during CMake is just a note, not even a warning, and not an error. You can ignore it. It just tells you the dependency ISAAC was not found and as long as you do not use it in your .cfg file at runtime you are good to go.

Spack usage: you can install and load a different version of picongpu via spack by specifying more constrains, e.g.:

spack install picongpu +adios %[email protected] 
spack load picongpu +adios %[email protected]

says "set the variant ADIOS (default: off) to on" and "build with GCC 7.3.0".
The first examples that explain Spack demonstrate that in more detail and the corresponding section in the Spack manual can be found here. You already figured out spack info picongpu which tells you what variants you can set - and now you can change them to your liking :)

You can also uninstall other versions of PIConGPU, which makes typing your non-default variant easier during spack load, by running spack uninstall picongpu ~adios (this deinstalls the default picongpu variant which has no adios support on). Or spack uninstall --all picongpu which uninstalls all currently installed picongpu variants.

Likewise, if you want to switch to the latest development (dev) branch of picongpu instead of the current stable version 0.4.3, run:

spack install picongpu@develop +adios %[email protected] 
spack load picongpu@develop +adios %[email protected]

[cbontoiu]

Hello again

I am trying to get a fresh install with spack install picongpu +adios %[email protected] using the installed cuda 10 and openmpi (configured for cuda with ucx and gdrcopy). See below my packages.yaml file:

packages:
  openmpi:
    paths:
      [email protected]: /usr/local/openmpi
    buildable: False
  cuda:
    paths:
      [email protected]%[email protected] arch=linuxmint19-x86_64: /usr/local/cuda-10.1
    buildable: False

==> Error: The spec
    '[email protected]:9.2'
    is configured as not buildable, and no matching external installs were found

Spack acknowledges my openmpi installation but not the cuda one and starts to download a lower version. How can I force it to use the installed version? I spent hours on finding this without success. There is some discussion here
https://spack.readthedocs.io/en/latest/workflows.html
on CUDA at the end of the page and I even installed Docker and executed docker pull nvidia/cuda thinking that it will download a file which I can image into Spack folder, but I got lost there.
Any help will be appreciated.

regards
Cristian

[ax3l]

I did not check personally if newer CUDA releases work well and other maintainers can jump in.
You can just relax this line https://github.com/ComputationalRadiationPhysics/spack-repo/blob/gh-pages/packages/picongpu/package.py#L72 in your $HOME/src/spack-repo/packages/picongpu/package.py from

    depends_on('[email protected]:9.2', when='backend=cuda')

to

    depends_on('[email protected]:', when='backend=cuda')

to allow all cuda versions >=8.0

[cbontoiu]

Thanks for your help,
Eventually I noticed errors when trying pic-build with CUDA 10. There might be some incompatibility there. I switched back to CUDA 9.2 and the code runs fine so far; I can compile and execute LWFA for now. However, I just wanted to check the last version of Bremsstrahlung from the deveopment version, because I know you have worked on this model recently and it cannot be compiled. This is what I get:

CMake Error at picongpu_generated_main.cpp.o.Release.cmake:219 (message):
  Error generating
  /home/cristi/PIC_INPUT/myBremsstrahlung/.build/CMakeFiles/picongpu.dir//./picongpu_generated_main.cpp.o
CMakeFiles/picongpu.dir/build.make:63: recipe for target 'CMakeFiles/picongpu.dir/picongpu_generated_main.cpp.o' failed
make[2]: *** [CMakeFiles/picongpu.dir/picongpu_generated_main.cpp.o] Error 1
...
CMakeFiles/Makefile2:107: recipe for target 'CMakeFiles/picongpu.dir/all' failed
make[1]: *** [CMakeFiles/picongpu.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2

Using stable version of these model got me over the pic-build stage but failed at runtime.
I know that error was fixed but how can we fetch the correct mode, please?
Cheers

[sbastrakov]

@cbontoiu I quickly checked the changes in picongpu/include/picongpu/particles/bremsstrahlung in the dev branch since the last release. There should not be anything that changed the behavior. So while the version from dev needs some name changes to compile with the last release, it should not resolve any problems (nor introduce new ones). Please create another issue for that Bremsstrahlung runtime failure, as this seems unrelated to the installation questions, we will look into it. Please provide your setup (set of .param files) as well, or is it on the standard Bremsstrahlung example?

[cbontoiu]

Yes, it was the standard example from the gitHub dev distribution without any modification. I will give it another try later and if the problem persists I will open a new issue. Regards.

[ax3l]

Thank you, please also post a few more lines above the error posted in https://github.com/ComputationalRadiationPhysics/picongpu/issues/3092#issuecomment-557173548

There should be a more concrete error about what is failing, but it's probably missing here.

[sbastrakov]

The original installation issues seem to be resolved, so closing this one.