Picongpu: Long initialization time and no particles
Hello,
I have a problem which lasts already for a few days and I see no solution. The density.param file is expected to generate 9 thick, hollow tubes like in the image below.
As I have just installed a second GPU (and linked to the old one like in the image) below
the cfg file calls for 2 GPUs but I don't think this is the cause of the problem.
There are two problems:
- the initialization time is 27 mins!
- there is not charge density in the images (I thought that it might be the minimum weight, but no)
Here is the model if you could check it. Something strange might happen. And the output in the terminal starts with an error too:
(base) quasar@quasar:~/PIC_INPUT/PICONGPU/[M]-THICK-TUBES-3D-HEX-ARRAY/hex-array-thick-tubes-LX$ rm -r .build/ && pic-build &> out_log.txt && tbg -s bash -c etc/picongpu/runConfiguration.cfg -t etc/picongpu/bash/mpiexec.tpl /media/quasar/RawDataDisk/PICONGPU/[M]-THICK-TUBES-3D-HEX-ARRAY/HEX_ARRAY_THICK_TUBES_3D_PAR_ds[cm-3]_1.0e+23_RxC_2x2_inR[nm]_2.0_th[nm]_10.0_wl[nm]_177.0_I[Wcm-2]_1.00e+15_A0_4.79e-03_Dt[fs]_10_pol_LX_msh_200_480_200
Running program...
using default compiler
==> Error: Spec '[email protected]%[email protected]+adios+hdf5+isaac+png backend=cuda cudacxx=nvcc arch=linux-linuxmint19-skylake ^[email protected]%[email protected]+blosc~bzip2~fortran~hdf5~infiniband+lz4+mpi~netcdf+shared+sz~szip+zfp+zlib patches=01113e9efb929d71c28bf33cc8b7f215d85195ec700e99cb41164e2f8f830640,8ae17f655248e87cbab1d1ed794e15364a38d2f5f8d971b1086702f72d79bd42,d24b79b795f66e40ddcd331ea4be896ac9c393d6f68f4318616d23928b0694e9 staging=none arch=linux-linuxmint19-skylake ^[email protected]%[email protected] arch=linux-linuxmint19-skylake ^[email protected]%[email protected] arch=linux-linuxmint19-skylake ^[email protected]%[email protected] arch=linux-linuxmint19-skylake ^[email protected]%[email protected]+atomic+chrono~clanglibcpp~container~context~coroutine+date_time~debug+exception~fiber+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave cxxstd=11 visibility=hidden arch=linux-linuxmint19-skylake ^[email protected]%[email protected]+shared arch=linux-linuxmint19-skylake ^[email protected]%[email protected]+avx2~ipo build_type=RelWithDebInfo patches=cd40604a26157a0e018ea496cf3267e116e6ec5ff80a7d1cef11b841c154c388 arch=linux-linuxmint19-skylake ^[email protected]%[email protected]~doc+ncurses+openssl+ownlibs~qt patches=bf695e3febb222da2ed94b3beea600650e4318975da90e4a71d6f31a6d5d8c3d arch=linux-linuxmint19-skylake ^[email protected]%[email protected] arch=linux-linuxmint19-skylake ^[email protected]%[email protected] arch=linux-linuxmint19-skylake ^[email protected]%[email protected]+libbsd arch=linux-linuxmint19-skylake ^[email protected]%[email protected] arch=linux-linuxmint19-skylake ^[email protected]%[email protected] arch=linux-linuxmint19-skylake ^[email protected]%[email protected]+bzip2+curses+git~libunistring+libxml2+tar+xz arch=linux-linuxmint19-skylake ^[email protected]%[email protected]~cxx~debug~fortran~hl~java+mpi+pic+shared~szip~threadsafe api=none arch=linux-linuxmint19-skylake ^[email protected]%[email protected]~ipo build_type=RelWithDebInfo arch=linux-linuxmint19-skylake ^[email protected]%[email protected]+cuda~ipo build_type=RelWithDebInfo arch=linux-linuxmint19-skylake ^[email protected]%[email protected]~ipo build_type=RelWithDebInfo arch=linux-linuxmint19-skylake ^[email protected]%[email protected]~ipo+shared build_type=RelWithDebInfo arch=linux-linuxmint19-skylake ^[email protected]%[email protected] arch=linux-linuxmint19-skylake ^[email protected]%[email protected] patches=26f26c6f29a7ce9bf370ad3ab2610f99365b4bdd7b82e7c31df41a3370d685c0 arch=linux-linuxmint19-skylake ^[email protected]%[email protected] arch=linux-linuxmint19-skylake ^[email protected]%[email protected] arch=linux-linuxmint19-skylake ^[email protected]%[email protected] arch=linux-linuxmint19-skylake ^[email protected]%[email protected] arch=linux-linuxmint19-skylake ^[email protected]%[email protected]~ipo+mpi build_type=RelWithDebInfo patches=669608721dfce0ada7cef1ac84344352791a8916b7bb98ca8a0d4e6d4670e744 arch=linux-linuxmint19-skylake ^[email protected]%[email protected] arch=linux-linuxmint19-skylake ^[email protected]%[email protected] arch=linux-linuxmint19-skylake ^[email protected]%[email protected]~ipo build_type=RelWithDebInfo arch=linux-linuxmint19-skylake ^[email protected]%[email protected]~python arch=linux-linuxmint19-skylake ^[email protected]%[email protected] arch=linux-linuxmint19-skylake ^[email protected]%[email protected]+sigsegv patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 arch=linux-linuxmint19-skylake ^[email protected]%[email protected] arch=linux-linuxmint19-skylake ^[email protected]%[email protected]~symlinks+termlib arch=linux-linuxmint19-skylake ^[email protected]%[email protected]~atomics~cuda~cxx~cxx_exceptions+gpfs~java~legacylaunchers~lustre~memchecker~pmi~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=none arch=linux-linuxmint19-skylake ^[email protected]%[email protected]+systemcerts arch=linux-linuxmint19-skylake ^[email protected]%[email protected]+cpanm+shared+threads arch=linux-linuxmint19-skylake ^[email protected]%[email protected] arch=linux-linuxmint19-skylake ^[email protected]%[email protected]~ipo build_type=RelWithDebInfo arch=linux-linuxmint19-skylake ^[email protected]%[email protected] arch=linux-linuxmint19-skylake ^[email protected]%[email protected]+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87 arch=linux-linuxmint19-skylake ^[email protected]%[email protected] arch=linux-linuxmint19-skylake ^[email protected]%[email protected] arch=linux-linuxmint19-skylake ^[email protected]%[email protected]~ipo+pic+shared build_type=RelWithDebInfo patches=c9cfecb1f7a623418590cf4e00ae7d308d1c3faeb15046c2e5090e38221da7cd arch=linux-linuxmint19-skylake ^[email protected]%[email protected]+column_metadata+fts~functions~rtree arch=linux-linuxmint19-skylake ^[email protected]%[email protected]~fortran~hdf5~ipo~netcdf~pastri~python~random_access+shared~stats~time_compression build_type=RelWithDebInfo arch=linux-linuxmint19-skylake ^[email protected]%[email protected] arch=linux-linuxmint19-skylake ^[email protected]%[email protected] arch=linux-linuxmint19-skylake ^[email protected]%[email protected]~pic arch=linux-linuxmint19-skylake ^[email protected]%[email protected]~aligned~fasthash~ipo~profile+shared~strided~twoway bsws=64 build_type=RelWithDebInfo arch=linux-linuxmint19-skylake ^[email protected]%[email protected]+optimize+pic+shared arch=linux-linuxmint19-skylake ^[email protected]%[email protected]+pic arch=linux-linuxmint19-skylake' matches no installed packages.
PIConGPU: 0.5.0
Build-Type: Release
Third party:
OS: Linux-5.0.0-32-generic
arch: x86_64
CXX: GNU (7.5.0)
CMake: 3.18.4
CUDA: 9.2.148
mallocMC: 2.3.1
Boost: 1.70.0
MPI:
standard: 3.1
flavor: OpenMPI (3.1.5)
PNGwriter: 0.7.0
libSplash: 1.7.0 (Format 4.0)
ADIOS: 1.13.1
PIConGPUVerbose PHYSICS(1) | Sliding Window is OFF
PIConGPUVerbose PHYSICS(1) | used Random Number Generator: RNGProvider3XorMin seed: 42
PIConGPUVerbose PHYSICS(1) | Courant c*dt <= 1.74147 ? 1
PIConGPUVerbose PHYSICS(1) | Resolving plasma oscillations?
Estimates are based on DensityRatio to BASE_DENSITY of each species
(see: density.param, speciesDefinition.param).
It and does not cover other forms of initialization
PIConGPUVerbose PHYSICS(1) | species e: omega_p * dt <= 0.1 ? 0.0142719
PIConGPUVerbose PHYSICS(1) | species i: omega_p * dt <= 0.1 ? 0.000333063
PIConGPUVerbose PHYSICS(1) | macro particles per device: 38400000
PIConGPUVerbose PHYSICS(1) | typical macro particle weighting: 6955.06
PIConGPUVerbose PHYSICS(1) | UNIT_SPEED 2.99792e+08
PIConGPUVerbose PHYSICS(1) | UNIT_TIME 8e-17
PIConGPUVerbose PHYSICS(1) | UNIT_LENGTH 2.39834e-08
PIConGPUVerbose PHYSICS(1) | UNIT_MASS 6.33563e-27
PIConGPUVerbose PHYSICS(1) | UNIT_CHARGE 1.11432e-15
PIConGPUVerbose PHYSICS(1) | UNIT_EFIELD 2.13064e+13
PIConGPUVerbose PHYSICS(1) | UNIT_BFIELD 71070.4
PIConGPUVerbose PHYSICS(1) | UNIT_ENERGY 5.69418e-10
initialization time: 27min 31sec 610msec = 1651 sec
0 % = 0 | time elapsed: 2msec | avg time per step: 0msec
4 % = 1815 | time elapsed: 54sec 700msec | avg time per step: 30msec
9 % = 3630 | time elapsed: 1min 49sec 456msec | avg time per step: 30msec
14 % = 5445 | time elapsed: 2min 44sec 348msec | avg time per step: 30msec
19 % = 7260 | time elapsed: 3min 39sec 857msec | avg time per step: 30msec
24 % = 9075 | time elapsed: 4min 34sec 700msec | avg time per step: 30msec
29 % = 10890 | time elapsed: 5min 29sec 494msec | avg time per step: 30msec
34 % = 12705 | time elapsed: 6min 24sec 796msec | avg time per step: 30msec
39 % = 14520 | time elapsed: 7min 19sec 994msec | avg time per step: 30msec
44 % = 16335 | time elapsed: 8min 15sec 451msec | avg time per step: 30msec
49 % = 18150 | time elapsed: 9min 11sec 825msec | avg time per step: 31msec
54 % = 19965 | time elapsed: 10min 7sec 229msec | avg time per step: 30msec
59 % = 21780 | time elapsed: 11min 2sec 537msec | avg time per step: 30msec
64 % = 23595 | time elapsed: 11min 57sec 403msec | avg time per step: 30msec
69 % = 25410 | time elapsed: 12min 52sec 260msec | avg time per step: 30msec
74 % = 27225 | time elapsed: 13min 47sec 265msec | avg time per step: 30msec
79 % = 29040 | time elapsed: 14min 43sec 12msec | avg time per step: 30msec
84 % = 30855 | time elapsed: 15min 39sec 1msec | avg time per step: 30msec
89 % = 32670 | time elapsed: 16min 34sec 752msec | avg time per step: 30msec
94 % = 34485 | time elapsed: 17min 29sec 984msec | avg time per step: 30msec
99 % = 36300 | time elapsed: 18min 26sec 836msec | avg time per step: 31msec
calculation simulation time: 18min 27sec 167msec = 1107 sec
full simulation time: 45min 58sec 960msec = 2758 sec
cbontoiu
All 11 comments
Hello @cbontoiu , could you attach the .cfg file as well?
sbastrakov
on 30 Oct 2020
edit: nevermind, I see it is in the etc directory.
sbastrakov
on 30 Oct 2020
@cbontoiu the input set you attached seems to not compile. In density.param line 70 there is a wrong ; at the end of the line. Also, there is one include to a file that was removed in the current dev version. I assume you just attached some work-in-progress version of the model. not the one used for simulation (you could take it from the output directory)?
Meanwhile, I see in your .cfg file that particle counter plugin was enabled. You could use its output to see if there were any macroparticles generated (as a potential explanation for missing charge density).
sbastrakov
on 30 Oct 2020
Hello @sbastrakov
I attached the working model which compiled and has run without results (empty images and data files), even with the error on line 70 and this is strange. The error should have been caught and shown in the out_log.txt file in the top folder.
Do you mean I should delete #include <pmacc/preprocessor/struct.hpp> in the develop version. I think I am using the standard version, not the develop version.
I will try to run again activating the TBG_e_macroCount and check the for macroparticles. I only wanted to check images for this model as data files seemed empty and as the model solves very fast. It should take 10 hours and it took only 45 mins.
Any further suggestion is highly appreciated.
Thank you so much
cbontoiu
on 30 Oct 2020
Also in the terminal I see typical macro particle weighting: 6955.06 but I expect the weight to be CELL_VOLUME*BASE_DENSITY = 7.9 because the mesh size is about 0.43 nm and the plasma density 1e29.
cbontoiu
on 30 Oct 2020
I attached the working model which compiled and has run without results (empty images and data files), even with the error on line 70 and this is strange
I believe this strongly indicates something went wrong already at this point. For example, the build could have been made from a non-empty state and so it compiled not the model you were expecting. I would suggest investigating this first, as currently it;s not clear what model was compiled into the code at all and so barely possible to make conclusions about weird physics underneath.
In order to guarantee a clean build, please make sure that there exist no .build directory before you run pic-build. There will be none for setups created with pic-create from our examples, but may be in case you work on some derivative of an older directory. Then you can just remove it with all subdirectories.
sbastrakov
on 30 Oct 2020
I always remove, compile and run in one command like the one below
rm -r .build/ && pic-build &> out_log.txt && tbg -s bash -c etc/picongpu/runConfiguration.cfg -t etc/picongpu/bash/mpiexec.tpl /media/quasar/RawDataDisk/PICONGPU/[M]-THICK-TUBES-3D-HEX-ARRAY/HEX_ARRAY_THICK_TUBES_3D_PAR_ds[cm-3]_1.0e+23_RxC_2x2_inR[nm]_2.0_th[nm]_10.0_wl[nm]_177.0_I[Wcm-2]_1.00e+15_A0_4.79e-03_Dt[fs]_10_pol_LX_msh_200_480_200
so the .build folder was removed.
cbontoiu
on 30 Oct 2020
However, it is true that I am using an old directory for a more recent version of picongpu, but the problem might be somewhere else. Maybe the error in the beginning says something. I confirm that the LWFA model works (compiles, runs and produces data and images) with this picongpu version and the current setup (cuda, opnempi, gcc etc).
cbontoiu
on 30 Oct 2020
Okay, so it's currently not clear what exactly went wrong. However, I would be really surprised if that code with ; actually compiled, and in my understanding it cannot be explained with a difference in compiler versios and other system-specific things. So somehow either the files did not go to the compiler, or perhaps the corresponding templates were not instantiated and so this piece got effectively disabled.
To investigate, could you create a fresh setup with pic-create from one of the standard examples (of the currently used PIConGPU code version), and then manually put your density implementation there? To compile-check I think for that you only need to put your species from speciesDefinition.param, then the TubesFunctor and related variables from density.param, and finally activating it in speciesInitialization.param. For an actual simulation you of course would also need to modify grid.param and perhaps other files. But that three should be enough to make it compile (or as I suspect not-compile)
sbastrakov
on 30 Oct 2020
@cbontoiu You mentioned in #3421 that the problem was a missing customization of using Selected = .... Can this issue be closed?
steindev
on 10 Nov 2020
Yes, I will close this issue. Thank you.
cbontoiu
on 10 Nov 2020
Related issues
berceanu
路
4Comments
sbastrakov
路
3Comments
psychocoderHPC
路
4Comments
hightower8083
路
4Comments
cbontoiu
路
3Comments
Most helpful comment
Okay, so it's currently not clear what exactly went wrong. However, I would be really surprised if that code with
;actually compiled, and in my understanding it cannot be explained with a difference in compiler versios and other system-specific things. So somehow either the files did not go to the compiler, or perhaps the corresponding templates were not instantiated and so this piece got effectively disabled.To investigate, could you create a fresh setup with
pic-createfrom one of the standard examples (of the currently used PIConGPU code version), and then manually put your density implementation there? To compile-check I think for that you only need to put your species fromspeciesDefinition.param, then theTubesFunctorand related variables fromdensity.param, and finally activating it inspeciesInitialization.param. For an actual simulation you of course would also need to modifygrid.paramand perhaps other files. But that three should be enough to make it compile (or as I suspect not-compile)