Picongpu: Question: ISAAC is hanging

[sbastrakov]

Dear @steindev @n01r @PrometheusPi @FelixTUD,

I am trying to use isaac for visualization of modified lwfa-example. My .cfg file has

TBG_isaac="--isaac.period 1 --isaac.name TRY1 --isaac.url 127.0.0.2  --isaac.port 2500" 
TBG_plugins="!TBG_isaac"

and ../.build/picongpu -h gives :

[ajitup@kshitij-5-pre .build]$ ./picongpu -h
Usage picongpu [-d dx=1 dy=1 dz=1] -g width height depth [options]
:
IsaacPlugin:
  --isaac.period arg                    Enable IsaacPlugin [for each n-th   step].
  --isaac.name arg (=default)           The name of the simulation. Default is  "default".
  --isaac.url arg (=localhost)          The url of the isaac server to connect  to. Default is "localhost".
  --isaac.port arg (=2460)              The port of the isaac server to connect  to. Default is 2460.
  --isaac.width arg (=1024)             The width per isaac framebuffer.  Default is 1024.
  --isaac.height arg (=768)             The height per isaac framebuffer. Default is 768.
  --isaac.directPause arg (=0)          Direct pausing after starting  simulation. Default is false.
  --isaac.quality arg (=90)             JPEG quality. Default is 90.
  --isaac.reconnect arg (=1)            Trying to reconnect every time an image is rendered if the connection is lost 
                                        or could never established at all.

When I start isaac on the head node of my cluster it gives following message and hangs at that:

[ajitup@kshitij-5-pre lwfa_001_gpu_isaac]$ isaac --url 127.0.0.2 --web_port 2500
Using web_port=2500, tcp_port=2458 and sim_port=2460

lwsts[23367]: Initial logging level 7
lwsts[23367]: Libwebsockets version: 2.2.0 [email protected]
lwsts[23367]: IPV6 not compiled in
lwsts[23367]: libev support not compiled in
lwsts[23367]: libuv support not compiled in
lwsts[23367]:  Threads: 1 each 1024 fds
lwsts[23367]:  mem: platform fd map:  8192 bytes
lwsts[23367]:  Compiled with OpenSSL support
lwsts[23367]:  SSL disabled: no LWS_SERVER_OPTION_DO_SSL_GLOBAL_INIT
lwsts[23367]: Creating Vhost 'default' port 2500, 2 protocols, IPv6 off
lwsts[23367]:  Listening on port 2500
lwsts[23367]:  mem: per-conn:          568 bytes + protocol rx buf
lwsts[23367]:  canonical_hostname = kshitij-5-pre.rrcat.gov.in
Running ISAAC Master
Starting insitu plugin listener
Launching WebSocketDataConnector
Launching TCPDataConnector
Launching JPEG_URI_Stream

I keep on waiting but it does not return to shell even after half an hour.

I submitted the lwfa simulation from a different shell and it aborted giving following error:

running program...
==> Error: Spec '[email protected]%[email protected]+adios+hdf5+isaac+png backend=cuda cudacxx=nvcc arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected]+blosc~bzip2~fortran~hdf5~infiniband+lz4+mpi~netcdf+shared+sz~szip+zfp+zlib patches=01113e9efb929d71c28bf33cc8b7f215d85195ec700e99cb41164e2f8f830640,8ae17f655248e87cbab1d1ed794e15364a38d2f5f8d971b1086702f72d79bd42,d24b79b795f66e40ddcd331ea4be896ac9c393d6f68f4318616d23928b0694e9 staging=none arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected] arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected] arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected] arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected]+atomic+chrono~clanglibcpp~container~context~coroutine+date_time~debug+exception~fiber+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave cxxstd=11 visibility=hidden arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected]+shared arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected]+avx2 build_type=RelWithDebInfo patches=cd40604a26157a0e018ea496cf3267e116e6ec5ff80a7d1cef11b841c154c388 arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected]~doc+ncurses+openssl+ownlibs~qt arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected] arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected] arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected]+libbsd arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected] arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected] arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected]+bzip2+curses+git~libunistring+libxml2+tar+xz arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected]~cxx~debug~fortran~hl~java+mpi+pic+shared~szip~threadsafe api=none arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected]~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected] build_type=RelWithDebInfo arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected]+cuda build_type=RelWithDebInfo arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected] build_type=RelWithDebInfo arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected]+shared build_type=RelWithDebInfo arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected] arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected] patches=26f26c6f29a7ce9bf370ad3ab2610f99365b4bdd7b82e7c31df41a3370d685c0 arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected] arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected] arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected] arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected] arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected] arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected]+mpi build_type=RelWithDebInfo patches=669608721dfce0ada7cef1ac84344352791a8916b7bb98ca8a0d4e6d4670e744 arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected] arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected] build_type=RelWithDebInfo arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected]~python arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected] arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected]+sigsegv patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected] arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected]~symlinks+termlib arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected] arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected]~atomics~cuda~cxx~cxx_exceptions+gpfs~java~legacylaunchers~lustre~memchecker~pmi~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=none arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected]+systemcerts arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected]+cpanm+shared+threads arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected] arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected] build_type=RelWithDebInfo arch=linux-
rhel7-skylake_avx512 ^[email protected]%[email protected] arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected]+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4~uuid+zlib patches=0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87 arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected] arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected] arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected]+pic+shared build_type=RelWithDebInfo patches=c9cfecb1f7a623418590cf4e00ae7d308d1c3faeb15046c2e5090e38221da7cd arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected]+column_metadata+fts~functions~rtree arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected]~fortran~hdf5~netcdf~pastri~python~random_access+shared~time_compression build_type=RelWithDebInfo arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected] arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected] arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected] arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected]~pic arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected]~aligned~fasthash~profile+shared~strided~twoway bsws=64 build_type=RelWithDebInfo arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected]+optimize+pic+shared arch=linux-rhel7-skylake_avx512 ^[email protected]%[email protected]+pic arch=linux-rhel7-skylake_avx512' matches no installed packages.
PIConGPU: 0.4.3
  Build-Type: Release

Third party:
  OS:         Linux-3.10.0-957.el7.x86_64
  arch:       x86_64
  CXX:        GNU (7.3.0)
  CMake:      3.18.2
  CUDA:       10.0.130
  mallocMC:   2.3.1
  Boost:      1.70.0
  MPI:        
    standard: 3.1
    flavor:   OpenMPI (3.1.6)
  PNGwriter:  0.7.0
  libSplash:  1.7.0 (Format 4.0)
  ADIOS:      NOTFOUND
PIConGPUVerbose PHYSICS(1) | Sliding Window is ON
PIConGPUVerbose INPUT_OUTPUT(32) | ISAAC Init succeded
PIConGPUVerbose PHYSICS(1) | used Random Number Generator: RNGProvider3XorMin seed: 42
PIConGPUVerbose PHYSICS(1) | Courant c*dt <= 1.00229 ? 1
PIConGPUVerbose PHYSICS(1) | species e: omega_p * dt <= 0.1 ? 0.0247974
PIConGPUVerbose PHYSICS(1) | y-cells per wavelength: 18.0587
PIConGPUVerbose PHYSICS(1) | macro particles per device: 100663296
PIConGPUVerbose PHYSICS(1) | typical macro particle weighting: 6955.06
PIConGPUVerbose PHYSICS(1) | UNIT_SPEED 2.99792e+08
PIConGPUVerbose PHYSICS(1) | UNIT_TIME 1.39e-16
PIConGPUVerbose PHYSICS(1) | UNIT_LENGTH 4.16712e-08
PIConGPUVerbose PHYSICS(1) | UNIT_MASS 6.33563e-27
PIConGPUVerbose PHYSICS(1) | UNIT_CHARGE 1.11432e-15
PIConGPUVerbose PHYSICS(1) | UNIT_EFIELD 1.22627e+13
PIConGPUVerbose PHYSICS(1) | UNIT_BFIELD 40903.8
PIConGPUVerbose PHYSICS(1) | UNIT_ENERGY 5.69418e-10
initialization time:  4min 56sec 731msec = 296 sec
terminate called after throwing an instance of 'std::runtime_error'
  what():  /home2/ajitup/.local/spack/opt/spack/linux-rhel7-skylake_avx512/gcc-7.3.0/picongpu-0.4.3-4tebd4rs6otiq547yib44vfdloz2cyc5/thirdParty/alpaka/include/alpaka/stream/StreamCudaRtAsync.hpp(351) 'cudaStreamSynchronize( stream.m_spStreamImpl->m_CudaStream)' A previous CUDA call (not this one) set the error  : 'cudaErrorInvalidConfiguration': 'invalid configuration argument'!
[kshitij-5-gpu8:25837] *** Process received signal ***
[kshitij-5-gpu8:25837] Signal: Aborted (6)
[kshitij-5-gpu8:25837] Signal code:  (-6)
[kshitij-5-gpu8:25837] [ 0] /usr/lib64/libpthread.so.0(+0xf5d0)[0x7f2ca4b6d5d0]
[kshitij-5-gpu8:25837] [ 1] /usr/lib64/libc.so.6(gsignal+0x37)[0x7f2ca26e0207]
[kshitij-5-gpu8:25837] [ 2] /usr/lib64/libc.so.6(abort+0x148)[0x7f2ca26e18f8]
[kshitij-5-gpu8:25837] [ 3] /home2/ajitup/src/spack/opt/spack/linux-rhel7-haswell/gcc-4.8.5/gcc-7.3.0-xrfc4r3ue7v73fq4rqsqffw3o2dnvqlf/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125)[0x7f2ca2f4d185]
[kshitij-5-gpu8:25837] [ 4] /home2/ajitup/src/spack/opt/spack/linux-rhel7-haswell/gcc-4.8.5/gcc-7.3.0-xrfc4r3ue7v73fq4rqsqffw3o2dnvqlf/lib64/libstdc++.so.6(+0x8ef56)[0x7f2ca2f4af56]
[kshitij-5-gpu8:25837] [ 5] /home2/ajitup/src/spack/opt/spack/linux-rhel7-haswell/gcc-4.8.5/gcc-7.3.0-xrfc4r3ue7v73fq4rqsqffw3o2dnvqlf/lib64/libstdc++.so.6(+0x8efa1)[0x7f2ca2f4afa1]
[kshitij-5-gpu8:25837] [ 6] /home2/ajitup/src/spack/opt/spack/linux-rhel7-haswell/gcc-4.8.5/gcc-7.3.0-xrfc4r3ue7v73fq4rqsqffw3o2dnvqlf/lib64/libstdc++.so.6(+0x8f1e4)[0x7f2ca2f4b1e4]
[kshitij-5-gpu8:25837] [ 7] /home2/ajitup/run.picongpu/lwfa_001_gpu_isaac/runs/input/bin/picongpu[0x45a0ae]
[kshitij-5-gpu8:25837] [ 8] /home2/ajitup/run.picongpu/lwfa_001_gpu_isaac/runs/input/bin/picongpu[0x491dd2]
[kshitij-5-gpu8:25837] [ 9] /home2/ajitup/run.picongpu/lwfa_001_gpu_isaac/runs/input/bin/picongpu[0x4dd2ba]
[kshitij-5-gpu8:25837] [10] /home2/ajitup/run.picongpu/lwfa_001_gpu_isaac/runs/input/bin/picongpu[0x6efc8d]
[kshitij-5-gpu8:25837] [11] /home2/ajitup/run.picongpu/lwfa_001_gpu_isaac/runs/input/bin/picongpu[0x6f0ce4]
[kshitij-5-gpu8:25837] [12] /home2/ajitup/run.picongpu/lwfa_001_gpu_isaac/runs/input/bin/picongpu[0x6f6a05]
[kshitij-5-gpu8:25837] [13] /home2/ajitup/run.picongpu/lwfa_001_gpu_isaac/runs/input/bin/picongpu[0x6ed86d]
[kshitij-5-gpu8:25837] [14] /home2/ajitup/run.picongpu/lwfa_001_gpu_isaac/runs/input/bin/picongpu[0x5503f2]
[kshitij-5-gpu8:25837] [15] /home2/ajitup/run.picongpu/lwfa_001_gpu_isaac/runs/input/bin/picongpu[0x552e44]
[kshitij-5-gpu8:25837] [16] /home2/ajitup/run.picongpu/lwfa_001_gpu_isaac/runs/input/bin/picongpu[0x4a2d07]
[kshitij-5-gpu8:25837] [17] /home2/ajitup/run.picongpu/lwfa_001_gpu_isaac/runs/input/bin/picongpu[0x58fcb1]
[kshitij-5-gpu8:25837] [18] /home2/ajitup/run.picongpu/lwfa_001_gpu_isaac/runs/input/bin/picongpu[0x5902c0]
[kshitij-5-gpu8:25837] [19] /home2/ajitup/run.picongpu/lwfa_001_gpu_isaac/runs/input/bin/picongpu[0x414c3b]
[kshitij-5-gpu8:25837] [20] /usr/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f2ca26cc3d5]
[kshitij-5-gpu8:25837] [21] /home2/ajitup/run.picongpu/lwfa_001_gpu_isaac/runs/input/bin/picongpu[0x416eef]
[kshitij-5-gpu8:25837] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------

In the browser on the postprocessor node, do I have to use url: http://127.0.0.2:2500/ ??? as it shows: could not connect. When I used ../client/html/visualisation.html file and tried to connect, it did not connect.

When I cancel the issac-server it gives:

^C
Waiting for insitu Master thread to finish... Done
Asking WebSocketDataConnector to exit
Asking TCPDataConnector to exit
Asking JPEG_URI_Stream to exit
lwsts[23367]: lws_context_destroy: ctx 0x1f47020
lwsts[23367]: lws_context_destroy2: ctx 0x1f47020
WebSocketDataConnector finished
TCPDataConnector finished
JPEG_URI_Stream finished

Am I doing it right? Could you please help me in solving this issue and visualizing the output using isaac?

_Originally posted by @ajitup73 in https://github.com/ComputationalRadiationPhysics/picongpu/issues/3410#issuecomment-747870264_

[ajitup73]

@sbastrakov

Thanks.
To my understanding, it seems to be issue with issac-server but I don't know how to resolve it.

[psychocoderHPC]

The problem is that the ISAAC version used together with picongpu 0.4.3 has many bugs. Newer versions of ISAAC are incompatible with PIConGPU 0.4.3 :-(
Sry that this is not documented somewhere but currently you should use PIConGPU (dev branch) and the latest dev branch of ISAAC. This would require that you compile ISAAC develop with spack (command should be: spack install isaac@develop and spack install isaac-server@develop)

note tested but to compile the dev branch with dev Isaac it should be: spack install picongpu@develop +isaac ^isaac@develop

[FelixTUD]

When I start isaac on the head node of my cluster it gives following message and hangs at that:

[ajitup@kshitij-5-pre lwfa_001_gpu_isaac]$ isaac --url 127.0.0.2 --web_port 2500
Using web_port=2500, tcp_port=2458 and sim_port=2460

lwsts[23367]: Initial logging level 7
lwsts[23367]: Libwebsockets version: 2.2.0 [email protected]
lwsts[23367]: IPV6 not compiled in
lwsts[23367]: libev support not compiled in
lwsts[23367]: libuv support not compiled in
lwsts[23367]: Threads: 1 each 1024 fds
lwsts[23367]: mem: platform fd map: 8192 bytes
lwsts[23367]: Compiled with OpenSSL support
lwsts[23367]: SSL disabled: no LWS_SERVER_OPTION_DO_SSL_GLOBAL_INIT
lwsts[23367]: Creating Vhost 'default' port 2500, 2 protocols, IPv6 off
lwsts[23367]: Listening on port 2500
lwsts[23367]: mem: per-conn: 568 bytes + protocol rx buf
lwsts[23367]: canonical_hostname = kshitij-5-pre.rrcat.gov.in
Running ISAAC Master
Starting insitu plugin listener
Launching WebSocketDataConnector
Launching TCPDataConnector
Launching JPEG_URI_Stream

I keep on waiting but it does not return to shell even after half an hour.

I don't see a problem with the server, as it executes as expected, the server is just waiting for incoming connections of the ISAAC In-Situ Visualization and/or the html client interface. If anyone is connecting to the server, you get a new output informing you about that.
As your PIConGPU with ISAAC crashes on launch, there is never a connection attempt to the server and no new update on the server side.

[FelixTUD]

Expected server output if html client connects sucessfully:

callback_isaac: LWS_CALLBACK_FILTER_PROTOCOL_CONNECTION
ISAAC Connection from localhost (127.0.0.1)!
callback_isaac: LWS_CALLBACK_ESTABLISHED

Expected server output if In-Situ Visualization connects sucessfully:

New connection, giving id 0 (control)
Group complete, sending to connected interfaces

[FelixTUD]

In the browser on the postprocessor node, do I have to use url: http://127.0.0.2:2500/ ??? as it shows: could not connect. When I used ../client/html/visualisation.html file and tried to connect, it did not connect.

The current server doesn't serve html pages over http, you just open the html file from the cloned repo on your client and enter the IP and web_port of the server, if your port forwarding etc. is correctly set up you should be able to connect and select a running Simulation to watch

[ajitup73]

@FelixTUD, @sbastrakov,

yes, the server was alright. I could get the client access the server through "visualisation.html" and it resulted in :

Running ISAAC Master
Starting insitu plugin listener
Launching WebSocketDataConnector
Launching TCPDataConnector
Launching JPEG_URI_Stream
lwsts[11496]:  forbidding on uri sanitation
ISAAC Connection from kshitij-5-post.rrcat.gov.in (192.168.130.87)!
callback_isaac: LWS_CALLBACK_ESTABLISHED
lwsts[11496]: wsi 0x7faa740008c0: TIMEDOUT WAITING on 13 (did hdr 0, ah (nil), wl 0, pfd events 1) 1608541948 vs 1608541947

Also, the browser windows started showing:

ISAAC Visualization server
Name    ID  Nodes   Max Functors    Functors    Dimension   Sources     Meta data   Observe

BUT no further output, although I had submitted a simulation. The simulation exited with the same error reported earlier in this thread:

PIConGPU: 0.4.3
  Build-Type: Release
Third party:
  OS:         Linux-3.10.0-957.el7.x86_64
  arch:       x86_64
  CXX:        GNU (7.3.0)
  CMake:      3.18.2
  CUDA:       10.0.130
  mallocMC:   2.3.1
  Boost:      1.70.0
  MPI:        
    standard: 3.1
    flavor:   OpenMPI (3.1.6)
  PNGwriter:  0.7.0
  libSplash:  1.7.0 (Format 4.0)
  ADIOS:      NOTFOUND
PIConGPUVerbose PHYSICS(1) | Sliding Window is ON
PIConGPUVerbose INPUT_OUTPUT(32) | ISAAC Init succeded
PIConGPUVerbose PHYSICS(1) | used Random Number Generator: RNGProvider3XorMin seed: 42
PIConGPUVerbose PHYSICS(1) | Courant c*dt <= 1.00229 ? 1
PIConGPUVerbose PHYSICS(1) | species e: omega_p * dt <= 0.1 ? 0.0247974
PIConGPUVerbose PHYSICS(1) | y-cells per wavelength: 18.0587
PIConGPUVerbose PHYSICS(1) | macro particles per device: 100663296
PIConGPUVerbose PHYSICS(1) | typical macro particle weighting: 6955.06
PIConGPUVerbose PHYSICS(1) | UNIT_SPEED 2.99792e+08
PIConGPUVerbose PHYSICS(1) | UNIT_TIME 1.39e-16
PIConGPUVerbose PHYSICS(1) | UNIT_LENGTH 4.16712e-08
PIConGPUVerbose PHYSICS(1) | UNIT_MASS 6.33563e-27
PIConGPUVerbose PHYSICS(1) | UNIT_CHARGE 1.11432e-15
PIConGPUVerbose PHYSICS(1) | UNIT_EFIELD 1.22627e+13
PIConGPUVerbose PHYSICS(1) | UNIT_BFIELD 40903.8
PIConGPUVerbose PHYSICS(1) | UNIT_ENERGY 5.69418e-10
initialization time:  3min 44sec 667msec = 224 sec

terminate called after throwing an instance of 'std::runtime_error'
  what():  /home2/ajitup/.local/spack/opt/spack/linux-rhel7-skylake_avx512/gcc-7.3.0/picongpu-0.4.3-4tebd4rs6otiq547yib44vfdloz2cyc5/thirdParty/alpaka/include/alpaka/stream/StreamCudaRtAsync.hpp(351) 'cudaStreamSynchronize( stream.m_spStreamImpl->m_CudaStream)' A previous CUDA call (not this one) set the error  : 'cudaErrorInvalidConfiguration': 'invalid configuration argument'!
[kshitij-5-gpu8:03710] *** Process received signal ***
[kshitij-5-gpu8:03710] Signal: Aborted (6)
[kshitij-5-gpu8:03710] Signal code:  (-6)

@psychocoderHPC

What should I do? Shall I install development branches of isaac, isaac-server with [email protected] OR go for [email protected]?

When I tried [email protected] installation, it shows some compatibility error between cuda, picongpu and my hardware:-

 spack install picongpu backend=cuda %[email protected]
==> Error: An unsatisfiable version constraint has been detected for spec:

    [email protected]%[email protected] arch=linux-rhel7-skylake_avx512

while trying to concretize the partial spec:

    [email protected]%[email protected]+adios+hdf5+isaac+png backend=cuda cudacxx=nvcc arch=linux-rhel7-skylake_avx512
        ^[email protected]:
            ^autoconf
                ^[email protected]:
                ^[email protected]:
                    ^berkeley-db
                    ^gdbm
                        ^readline
                            ^ncurses
                                ^[email protected]:
            ^automake
            ^libtool
            ^python
        ^[email protected]%[email protected]~doc+ncurses+openssl+ownlibs~qt arch=linux-rhel7-skylake_avx512
            ^openssl
                ^[email protected]%[email protected]+optimize+pic+shared arch=linux-rhel7-skylake_avx512
        ^[email protected],develop
        ^[email protected]%[email protected]~atomics~cuda~cxx~cxx_exceptions+gpfs~java~legacylaunchers~lustre~memchecker~pmi~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=none arch=linux-rhel7-skylake_avx512
            ^hwloc@:1.999
            ^numactl
        ^[email protected],develop
            ^freetype
                ^bzip2
                    ^diffutils
                        ^iconv
                ^libpng
        ^[email protected]%[email protected] arch=linux-rhel7-skylake_avx512
            ^lz4
            ^[email protected]%[email protected] arch=linux-rhel7-skylake_avx512
            ^xxhash
            ^zstd


picongpu requires cuda version 9.2:10.2.89, but spec asked for 11.0.2

Which option should I choose:

spack install [email protected] [email protected] isaac=True %[email protected]

OR

spack install picongpu@develop +isaac ^isaac@develop

please advise.

[FelixTUD]

TBG_isaac="--isaac.period 1 --isaac.name TRY1 --isaac.url 127.0.0.2 --isaac.port 2500"

One more connection problem is that you defined the port 2500 in your picongpu .cfg, but you have set the web_port to 2500 and the sim_port is 2460, which is the one the simulation is supposed to connect to.
So either change the port in the simulation .cfg to the default 2460 or change them both (server and .cfg) to a different one.

[ajitup@kshitij-5-pre lwfa_001_gpu_isaac]$ isaac --url 127.0.0.2 --web_port 2500
Using web_port=2500, tcp_port=2458 and sim_port=2460

[ajitup73]

I changed the .cfg file and set

TBG_isaac="--isaac.period 1 --isaac.name TRY1 --isaac.url 127.0.0.2"

and

isaac --url 127.0.0.2

leaving all the ports to default. Now, I am able to see the results in browser.
Thanks for help in resolving this issue.