Picongpu: FoilLCT: runtime error

Created on 30 Jul 2019 · 9Comments · Source: ComputationalRadiationPhysics/picongpu

I pulled the latest version of the dev branch (1347ebb) and created a FoilLCT param set with pic-create on defq. When I ran the example with 4.cfg without any changes on k80it instantly crashes with the following error message:

stderr

Unloading module pngwriter/0.7.0 
Unloading module libpng/1.6.35 
Unloading module libsplash/1.7.0-cuda100 
Unloading module hdf5-parallel/1.8.20-cuda100 
Unloading module adios/1.13.1-cuda100 
Unloading module c-blosc/1.14.4 
Unloading module zlib/1.2.11 
Unloading module boost/1.68.0 
Unloading module openmpi/2.1.2-cuda100 
Unloading module cuda/10.0 
Unloading module gcc/7.3.0 
Loading module gcc/7.3.0 
Loading module cmake/3.11.3 
Loading module cuda/10.0 
Loading module openmpi/2.1.2-cuda100 
Loading module boost/1.68.0 
Loading module zlib/1.2.11 
Loading module c-blosc/1.14.4 
Loading module adios/1.13.1-cuda100 
Loading module hdf5-parallel/1.8.20-cuda100 
Loading module libsplash/1.7.0-cuda100 
Loading module libpng/1.6.35 
Loading module pngwriter/0.7.0 
no binary 'cuda_memtest' available or compute node is not exclusively allocated, skip GPU memory test
Unhandled exception of type 'St13runtime_error' with message '/home/carste06/src/picongpu/thirdParty/alpaka/include/alpaka/event/EventCudaRt.hpp(211) 'cudaEventRecord( event.m_spEventImpl->m_CudaEvent, queue.m_spQueueImpl->m_CudaQueue)' A previous CUDA call (not this one) set the error  : 'cudaErrorInvalidDeviceFunction': 'invalid device function'!', terminating
Unhandled exception of type 'St13runtime_error' with message '/home/carste06/src/picongpu/thirdParty/alpaka/include/alpaka/event/EventCudaRt.hpp(211) 'cudaEventRecord( event.m_spEventImpl->m_CudaEvent, queue.m_spQueueImpl->m_CudaQueue)' A previous CUDA call (not this one) set the error  : 'cudaErrorInvalidDeviceFunction': 'invalid device function'!', terminating
Unhandled exception of type 'St13runtime_error' with message '/home/carste06/src/picongpu/thirdParty/alpaka/include/alpaka/event/EventCudaRt.hpp(211) 'cudaEventRecord( event.m_spEventImpl->m_CudaEvent, queue.m_spQueueImpl->m_CudaQueue)' A previous CUDA call (not this one) set the error  : 'cudaErrorInvalidDeviceFunction': 'invalid device function'!', terminating
Unhandled exception of type 'St13runtime_error' with message '/home/carste06/src/picongpu/thirdParty/alpaka/include/alpaka/event/EventCudaRt.hpp(211) 'cudaEventRecord( event.m_spEventImpl->m_CudaEvent, queue.m_spQueueImpl->m_CudaQueue)' A previous CUDA call (not this one) set the error  : 'cudaErrorInvalidDeviceFunction': 'invalid device function'!', terminating
[kepler021:05428] *** Process received signal ***
[kepler021:05428] Signal: Segmentation fault (11)
[kepler021:05428] Signal code: Address not mapped (1)
[kepler021:05428] Failing at address: 0x10
[kepler021:05427] *** Process received signal ***
[kepler021:05427] Signal: Segmentation fault (11)
[kepler021:05427] Signal code:  (128)
[kepler021:05427] Failing at address: (nil)
[kepler021:05425] *** Process received signal ***
[kepler021:05425] Signal: Segmentation fault (11)
[kepler021:05425] Signal code:  (128)
[kepler021:05425] Failing at address: (nil)
[kepler021:05426] *** Process received signal ***
[kepler021:05426] Signal: Segmentation fault (11)
[kepler021:05426] Signal code: Address not mapped (1)
[kepler021:05426] Failing at address: 0x10
[kepler021:05427] [ 0] [kepler021:05425] [ 0] [kepler021:05428] [ 0] /usr/lib64/libpthread.so.0(+0xf6d0)[0x2aaaaacde6d0]
[kepler021:05427] [ 1] /usr/lib64/libpthread.so.0(+0xf6d0)[0x2aaaaacde6d0]
[kepler021:05425] [ 1] /usr/lib64/libpthread.so.0(+0xf6d0)[0x2aaaaacde6d0]
[kepler021:05428] [ 1] [kepler021:05426] [ 0] /usr/lib64/libpthread.so.0(+0xf6d0)[0x2aaaaacde6d0]
[kepler021:05426] [ 1] /trinity/shared/pkg/compiler/gcc/7.3.0/lib64/libstdc++.so.6(_ZSt28_Rb_tree_rebalance_for_erasePSt18_Rb_tree_node_baseRS_+0x21b)[0x2aaaada4198b]
[kepler021:05428] [ 2] /trinity/shared/pkg/compiler/gcc/7.3.0/lib64/libstdc++.so.6(_ZSt28_Rb_tree_rebalance_for_erasePSt18_Rb_tree_node_baseRS_+0x21b)[0x2aaaada4198b]
[kepler021:05426] [ 2] /home/carste06/SCRATCH/sims/pullrequest/test20/input/bin/picongpu(_ZN16cupla_cuda_async13cuplaFreeHostEPv+0x5c)[0x153b6cc]
[kepler021:05425] [ 2] /home/carste06/SCRATCH/sims/pullrequest/test20/input/bin/picongpu(_ZN16cupla_cuda_async13cuplaFreeHostEPv+0x5c)[0x153b6cc]
[kepler021:05427] [ 2] /home/carste06/SCRATCH/sims/pullrequest/test20/input/bin/picongpu(_ZN16cupla_cuda_async13cuplaFreeHostEPv+0x80)[0x153b6f0]
[kepler021:05428] [ 3] /home/carste06/SCRATCH/sims/pullrequest/test20/input/bin/picongpu(_ZN5pmacc16HostBufferInternINS_9SuperCellINS_5FrameINS_15ParticlesBufferINS_19ParticleDescriptionINS_4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEENS_4math2CT3IntILi16ELi16ELi2147483647EEEN5boost3mpl6v_itemIN8picongpu24placeholder_definition326probeEENSE_INSF_24placeholder_definition336probeBENSE_INSF_24placeholder_definition258positionINSF_24placeholder_definition2712position_picENS_24placeholder_definition2213pmacc_isAliasEEENSD_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENSE_INSF_24placeholder_definition4513interpolationINSF_28FieldToParticleInterpolationINSF_9particles6shapes3TSCENSF_30AssignedTrilinearInterpolationEEESP_EENSE_INSF_24placeholder_definition385shapeIS13_SP_EENSE_INSF_24placeholder_definition3914particlePusherINS11_6pusher5ProbeESP_EESU_Li0EEELi0EEELi0EEENS_17HandleGuardRegionINS_9particles8policies17ExchangeParticlesENS11_8boundary29CallPluginsAndDeleteParticlesEEESU_SU_EESB_N8mallocMC9AllocatorINS1Q_16CreationPolicies7ScatterINSF_16DeviceHeapConfigENS1S_11ScatterConf27DefaultScatterHashingParamsEEENS1Q_20DistributionPolicies4NoopENS1Q_11OOMPolicies10ReturnNullENS1Q_19ReservePoolPolicies16SimpleCudaMallocENS1Q_17AlignmentPolicies6ShrinkINS24_12ShrinkConfig19DefaultShrinkConfigEEEEELj2EE29OperatorCreatePairStaticArrayILj256EEENS4_IS7_SB_NSE_INS_24placeholder_definition249multiMaskENSE_INS_24placeholder_definition2312localCellIdxESX_Li0EEELi0EEES1H_S1O_SU_NSE_INS_12NextFramePtrINSS_3argILi1EEEEENSE_INS_16PreviousFramePtrIS2L_EESU_Li0EEELi0EEEEEEEEELj2EED1Ev+0x55)[0x13dc2b5]
[kepler021:05428] [ 4] /home/carste06/SCRATCH/sims/pullrequest/test20/input/bin/picongpu(_ZN5pmacc10GridBufferINS_9SuperCellINS_5FrameINS_15ParticlesBufferINS_19ParticleDescriptionINS_4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEENS_4math2CT3IntILi16ELi16ELi2147483647EEEN5boost3mpl6v_itemIN8picongpu24placeholder_definition326probeEENSE_INSF_24placeholder_definition336probeBENSE_INSF_24placeholder_definition258positionINSF_24placeholder_definition2712position_picENS_24placeholder_definition2213pmacc_isAliasEEENSD_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENSE_INSF_24placeholder_definition4513interpolationINSF_28FieldToParticleInterpolationINSF_9particles6shapes3TSCENSF_30AssignedTrilinearInterpolationEEESP_EENSE_INSF_24placeholder_definition385shapeIS13_SP_EENSE_INSF_24placeholder_definition3914particlePusherINS11_6pusher5ProbeESP_EESU_Li0EEELi0EEELi0EEENS_17HandleGuardRegionINS_9particles8policies17ExchangeParticlesENS11_8boundary29CallPluginsAndDeleteParticlesEEESU_SU_EESB_N8mallocMC9AllocatorINS1Q_16CreationPolicies7ScatterINSF_16DeviceHeapConfigENS1S_11ScatterConf27DefaultScatterHashingParamsEEENS1Q_20DistributionPolicies4NoopENS1Q_11OOMPolicies10ReturnNullENS1Q_19ReservePoolPolicies16SimpleCudaMallocENS1Q_17AlignmentPolicies6ShrinkINS24_12ShrinkConfig19DefaultShrinkConfigEEEEELj2EE29OperatorCreatePairStaticArrayILj256EEENS4_IS7_SB_NSE_INS_24placeholder_definition249multiMaskENSE_INS_24placeholder_definition2312localCellIdxESX_Li0EEELi0EEES1H_S1O_SU_NSE_INS_12NextFramePtrINSS_3argILi1EEEEENSE_INS_16PreviousFramePtrIS2L_EESU_Li0EEELi0EEEEEEEEELj2ES2T_ED1Ev+0xb7c)[0x13dd17c]
[kepler021:05428] [ 5] /home/carste06/SCRATCH/sims/pullrequest/test20/input/bin/picongpu(_ZThn32_N8picongpu9ParticlesIN5pmacc4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc/home/carste06/SCRATCH/sims/pullrequest/test20/input/bin/picongpu(_ZN16cupla_cuda_async13cuplaFreeHostEPv+0x80)[0x153b6f0]
[kepler021:05426] [ 3] /home/carste06/SCRATCH/sims/pullrequest/test20/input/bin/picongpu(_ZN5pmacc16HostBufferInternINS_9SuperCellINS_5FrameINS_15ParticlesBufferINS_19ParticleDescriptionINS_4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEENS_4math2CT3IntILi16ELi16ELi2147483647EEEN5boost3mpl6v_itemIN8picongpu24placeholder_definition326probeEENSE_INSF_24placeholder_definition336probeBENSE_INSF_24placeholder_definition258positionINSF_24placeholder_definition2712position_picENS_24placeholder_definition2213pmacc_isAliasEEENSD_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENSE_INSF_24placeholder_definition4513interpolationINSF_28FieldToParticleInterpolationINSF_9particles6shapes3TSCENSF_30AssignedTrilinearInterpolationEEESP_EENSE_INSF_24placeholder_definition385shapeIS13_SP_EENSE_INSF_24placeholder_definition3914particlePusherINS11_6pusher5ProbeESP_EESU_Li0EEELi0EEELi0EEENS_17HandleGuardRegionINS_9particles8policies17ExchangeParticlesENS11_8boundary29CallPluginsAndDeleteParticlesEEESU_SU_EESB_N8mallocMC9AllocatorINS1Q_16CreationPolicies7ScatterINSF_16DeviceHeapConfigENS1S_11ScatterConf27DefaultScatterHashingParamsEEENS1Q_20DistributionPolicies4NoopENS1Q_11OOMPolicies10ReturnNullENS1Q_19ReservePoolPolicies16SimpleCudaMallocENS1Q_17AlignmentPolicies6ShrinkINS24_12ShrinkConfig19DefaultShrinkConfigEEEEELj2EE29OperatorCreatePairStaticArrayILj256EEENS4_IS7_SB_NSE_INS_24placeholder_definition249multiMaskENSE_INS_24placeholder_definition2312localCellIdxESX_Li0EEELi0EEES1H_S1O_SU_NSE_INS_12NextFramePtrINSS_3argILi1EEEEENSE_INS_16PreviousFramePtrIS2L_EESU_Li0EEELi0EEEEEEEEELj2EED1Ev+0x55)[0x13dc2b5]
[kepler021:05426] [ 4] /home/carste06/SCRATCH/sims/pullrequest/test20/input/bin/picongpu(_ZN5pmacc10GridBufferINS_9SuperCellINS_5FrameINS_15ParticlesBufferINS_19ParticleDescriptionINS_4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEENS_4math2CT3IntILi16ELi16ELi2147483647EEEN5boost3mpl6v_itemIN8picongpu24placeholder_definition326probeEENSE_INSF_24placeholder_definition336probeBENSE_INSF_24placeholder_definition258positionINSF_24placeholder_definition2712position_picENS_24placeholder_definition2213pmacc_isAliasEEENSD_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENSE_INSF_24placeholder_definition4513interpolationINSF_28FieldToParticleInterpolationINSF_9particles6shapes3TSCENSF_30AssignedTrilinearInterpolationEEESP_EENSE_INSF_24placeholder_definition385shapeIS13_SP_EENSE_INSF_24placeholder_definition3914particlePusherINS11_6pusher5ProbeESP_EESU_Li0EEELi0EEELi0EEENS_17HandleGuardRegionINS_9particles8policies17ExchangeParticlesENS11_8boundary29CallPluginsAndDeleteParticlesEEESU_SU_EESB_N8mallocMC9AllocatorINS1Q_16CreationPolicies7ScatterINSF_16DeviceHeapConfigENS1S_11ScatterConf27DefaultScatterHashingParamsEEENS1Q_20DistributionPolicies4NoopENS1Q_11OOMPolicies10ReturnNullENS1Q_19ReservePoolPolicies16SimpleCudaMallocENS1Q_17AlignmentPolicies6ShrinkINS24_12ShrinkConfig19DefaultShrinkConfigEEEEELj2EE29OperatorCreatePairStaticArrayILj256EEENS4_IS7_SB_NSE_INS_24placeholder_definition249multiMaskENSE_INS_24placeholder_definition2312localCellIdxESX_Li0EEELi0EEES1H_S1O_SU_NSE_INS_12NextFramePtrINSS_3argILi1EEEEENSE_INS_16PreviousFramePtrIS2L_EESU_Li0EEELi0EEEEEEEEELj2ES2T_ED1Ev+0xb7c)[0x13dd17c]
[kepler021:05426] [ 5] /home/carste06/SCRATCH/sims/pullrequest/test20/input/bin/picongpu(_ZThn32_N8picongpu9ParticlesIN5pmacc4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc/home/carste06/SCRATCH/sims/pullrequest/test20/input/bin/picongpu(_ZN5pmacc16HostBufferInternINS_9SuperCellINS_5FrameINS_15ParticlesBufferINS_19ParticleDescriptionINS_4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEENS_4math2CT3IntILi16ELi16ELi2147483647EEEN5boost3mpl6v_itemIN8picongpu24placeholder_definition326probeEENSE_INSF_24placeholder_definition336probeBENSE_INSF_24placeholder_definition258positionINSF_24placeholder_definition2712position_picENS_24placeholder_definition2213pmacc_isAliasEEENSD_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENSE_INSF_24placeholder_definition4513interpolationINSF_28FieldToParticleInterpolationINSF_9particles6shapes3TSCENSF_30AssignedTrilinearInterpolationEEESP_EENSE_INSF_24placeholder_definition385shapeIS13_SP_EENSE_INSF_24placeholder_definition3914particlePusherINS11_6pusher5ProbeESP_EESU_Li0EEELi0EEELi0EEENS_17HandleGuardRegionINS_9particles8policies17ExchangeParticlesENS11_8boundary29CallPluginsAndDeleteParticlesEEESU_SU_EESB_N8mallocMC9AllocatorINS1Q_16CreationPolicies7ScatterINSF_16DeviceHeapConfigENS1S_11ScatterConf27DefaultScatterHashingParamsEEENS1Q_20DistributionPolicies4NoopENS1Q_11OOMPolicies10ReturnNullENS1Q_19ReservePoolPolicies16SimpleCudaMallocENS1Q_17AlignmentPolicies6ShrinkINS24_12ShrinkConfig19DefaultShrinkConfigEEEEELj2EE29OperatorCreatePairStaticArrayILj256EEENS4_IS7_SB_NSE_INS_24placeholder_definition249multiMaskENSE_INS_24placeholder_definition2312localCellIdxESX_Li0EEELi0EEES1H_S1O_SU_NSE_INS_12NextFramePtrINSS_3argILi1EEEEENSE_INS_16PreviousFramePtrIS2L_EESU_Li0EEELi0EEEEEEEEELj2EED1Ev+0x55)[0x13dc2b5]
[kepler021:05425] [ 3] /home/carste06/SCRATCH/sims/pullrequest/test20/input/bin/picongpu(_ZN5pmacc10GridBufferINS_9SuperCellINS_5FrameINS_15ParticlesBufferINS_19ParticleDescriptionINS_4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEENS_4math2CT3IntILi16ELi16ELi2147483647EEEN5boost3mpl6v_itemIN8picongpu24placeholder_definition326probeEENSE_INSF_24placeholder_definition336probeBENSE_INSF_24placeholder_definition258positionINSF_24placeholder_definition2712position_picENS_24placeholder_definition2213pmacc_isAliasEEENSD_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENSE_INSF_24placeholder_definition4513interpolationINSF_28FieldToParticleInterpolationINSF_9particles6shapes3TSCENSF_30AssignedTrilinearInterpolationEEESP_EENSE_INSF_24placeholder_definition385shapeIS13_SP_EENSE_INSF_24placeholder_definition3914particlePusherINS11_6pusher5ProbeESP_EESU_Li0EEELi0EEELi0EEENS_17HandleGuardRegionINS_9particles8policies17ExchangeParticlesENS11_8boundary29CallPluginsAndDeleteParticlesEEESU_SU_EESB_N8mallocMC9AllocatorINS1Q_16CreationPolicies7ScatterINSF_16DeviceHeapConfigENS1S_11ScatterConf27DefaultScatterHashingParamsEEENS1Q_20DistributionPolicies4NoopENS1Q_11OOMPolicies10ReturnNullENS1Q_19ReservePoolPolicies16SimpleCudaMallocENS1Q_17AlignmentPolicies6ShrinkINS24_12ShrinkConfig19DefaultShrinkConfigEEEEELj2EE29OperatorCreatePairStaticArrayILj256EEENS4_IS7_SB_NSE_INS_24placeholder_definition249multiMaskENSE_INS_24placeholder_definition2312localCellIdxESX_Li0EEELi0EEES1H_S1O_SU_NSE_INS_12NextFramePtrINSS_3argILi1EEEEENSE_INS_16PreviousFramePtrIS2L_EESU_Li0EEELi0EEEEEEEEELj2ES2T_ED1Ev+0xb7c)[0x13dd17c]
[kepler021:05425] [ 4] /home/carste06/SCRATCH/sims/pullrequest/test20/input/bin/picongpu(_ZThn32_N8picongpu9ParticlesIN5pmacc4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEEN5boost3mpl6v_itemINS_24place/home/carste06/SCRATCH/sims/pullrequest/test20/input/bin/picongpu(_ZN5pmacc16HostBufferInternINS_9SuperCellINS_5FrameINS_15ParticlesBufferINS_19ParticleDescriptionINS_4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEENS_4math2CT3IntILi16ELi16ELi2147483647EEEN5boost3mpl6v_itemIN8picongpu24placeholder_definition326probeEENSE_INSF_24placeholder_definition336probeBENSE_INSF_24placeholder_definition258positionINSF_24placeholder_definition2712position_picENS_24placeholder_definition2213pmacc_isAliasEEENSD_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENSE_INSF_24placeholder_definition4513interpolationINSF_28FieldToParticleInterpolationINSF_9particles6shapes3TSCENSF_30AssignedTrilinearInterpolationEEESP_EENSE_INSF_24placeholder_definition385shapeIS13_SP_EENSE_INSF_24placeholder_definition3914particlePusherINS11_6pusher5ProbeESP_EESU_Li0EEELi0EEELi0EEENS_17HandleGuardRegionINS_9particles8policies17ExchangeParticlesENS11_8boundary29CallPluginsAndDeleteParticlesEEESU_SU_EESB_N8mallocMC9AllocatorINS1Q_16CreationPolicies7ScatterINSF_16DeviceHeapConfigENS1S_11ScatterConf27DefaultScatterHashingParamsEEENS1Q_20DistributionPolicies4NoopENS1Q_11OOMPolicies10ReturnNullENS1Q_19ReservePoolPolicies16SimpleCudaMallocENS1Q_17AlignmentPolicies6ShrinkINS24_12ShrinkConfig19DefaultShrinkConfigEEEEELj2EE29OperatorCreatePairStaticArrayILj256EEENS4_IS7_SB_NSE_INS_24placeholder_definition249multiMaskENSE_INS_24placeholder_definition2312localCellIdxESX_Li0EEELi0EEES1H_S1O_SU_NSE_INS_12NextFramePtrINSS_3argILi1EEEEENSE_INS_16PreviousFramePtrIS2L_EESU_Li0EEELi0EEEEEEEEELj2EED1Ev+0x55)[0x13dc2b5]
[kepler021:05427] [ 3] /home/carste06/SCRATCH/sims/pullrequest/test20/input/bin/picongpu(_ZN5pmacc10GridBufferINS_9SuperCellINS_5FrameINS_15ParticlesBufferINS_19ParticleDescriptionINS_4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEENS_4math2CT3IntILi16ELi16ELi2147483647EEEN5boost3mpl6v_itemIN8picongpu24placeholder_definition326probeEENSE_INSF_24placeholder_definition336probeBENSE_INSF_24placeholder_definition258positionINSF_24placeholder_definition2712position_picENS_24placeholder_definition2213pmacc_isAliasEEENSD_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENSE_INSF_24placeholder_definition4513interpolationINSF_28FieldToParticleInterpolationINSF_9particles6shapes3TSCENSF_30AssignedTrilinearInterpolationEEESP_EENSE_INSF_24placeholder_definition385shapeIS13_SP_EENSE_INSF_24placeholder_definition3914particlePusherINS11_6pusher5ProbeESP_EESU_Li0EEELi0EEELi0EEENS_17HandleGuardRegionINS_9particles8policies17ExchangeParticlesENS11_8boundary29CallPluginsAndDeleteParticlesEEESU_SU_EESB_N8mallocMC9AllocatorINS1Q_16CreationPolicies7ScatterINSF_16DeviceHeapConfigENS1S_11ScatterConf27DefaultScatterHashingParamsEEENS1Q_20DistributionPolicies4NoopENS1Q_11OOMPolicies10ReturnNullENS1Q_19ReservePoolPolicies16SimpleCudaMallocENS1Q_17AlignmentPolicies6ShrinkINS24_12ShrinkConfig19DefaultShrinkConfigEEEEELj2EE29OperatorCreatePairStaticArrayILj256EEENS4_IS7_SB_NSE_INS_24placeholder_definition249multiMaskENSE_INS_24placeholder_definition2312localCellIdxESX_Li0EEELi0EEES1H_S1O_SU_NSE_INS_12NextFramePtrINSS_3argILi1EEEEENSE_INS_16PreviousFramePtrIS2L_EESU_Li0EEELi0EEEEEEEEELj2ES2T_ED1Ev+0xb7c)[0x13dd17c]
[kepler021:05427] [ 4] /home/carste06/SCRATCH/sims/pullrequest/test20/input/bin/picongpu(_ZThn32_N8picongpu9ParticlesIN5pmacc4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEEN5boost3mpl6v_itemINS_24placeholder_definition4513interpolationINS_28FieldToParticleInterpolationINS_9particles6shapes3TSCENS_30AssignedTrilinearInterpolationEEENS1_24placeholder_definition2213pmacc_isAliasEEENS7_INS_24placeholder_definition385shapeISD_SH_EENS7_INS_24placeholder_definition3914particlePusherINSB_6pusher5ProbeESH_EENS6_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENS7_INS_24placeholder_definition326probeEENS7_INS_24placeholder_definition336probeBENS7_INS_24placeholder_definition258positionINS_24placeholder_definition2712position_picESH_EESU_Li0EEELi0EEELi0EEEED0Ev+0x80)[0x13e6540]
[kepler021:05425] [ 5] /home/carste06/SCRATCH/sims/pullrequest/test20/input/bin/picongpu(_ZN5pmacc13DataConnectorD2Ev+0x152)[0x1360322]
[kepler021:05425] [ 6] /usr/lib64/libc.so.6(+0x39bd9)[0x2aaaae19bbd9]
[kepler021:05425] [ 7] /usr/lib64/libc.so.6(+0x39c27)[0x2aaaae19bc27]
[kepler021:05425] [ 8] /usr/lib64/libc.so.6(__libc_start_main+0xfc)[0x2aaaae18444c]
[kepler021:05425] [ 9] /home/carste06/SCRATCH/sims/pullrequest/test20/input/bin/picongpu[0x11fc3df]
[kepler021:05425] *** End of error message ***
0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEEN5boost3mpl6v_itemINS_24placeholder_definition4513interpolationINS_28FieldToParticleInterpolationINS_9particles6shapes3TSCENS_30AssignedTrilinearInterpolationEEENS1_24placeholder_definition2213pmacc_isAliasEEENS7_INS_24placeholder_definition385shapeISD_SH_EENS7_INS_24placeholder_definition3914particlePusherINSB_6pusher5ProbeESH_EENS6_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENS7_INS_24placeholder_definition326probeEENS7_INS_24placeholder_definition336probeBENS7_INS_24placeholder_definition258positionINS_24placeholder_definition2712position_picESH_EESU_Li0EEELi0EEELi0EEEED0Ev+0x80)[0x13e6540]
[kepler021:05428] [ 6] /home/carste06/SCRATCH/sims/pullrequest/test20/input/bin/picongpu(_ZN5pmacc13DataConnectorD2Ev+0x152)[0x1360322]
[kepler021:05428] [ 7] /usr/lib64/libc.so.6(+0x39bd9)[0x2aaaae19bbd9]
[kepler021:05428] [ 8] /usr/lib64/libc.so.6(+0x39c27)[0x2aaaae19bc27]
[kepler021:05428] [ 9] /usr/lib64/libc.so.6(__libc_start_main+0xfc)[0x2aaaae18444c]
[kepler021:05428] [10] /home/carste06/SCRATCH/sims/pullrequest/test20/input/bin/picongpu[0x11fc3df]
[kepler021:05428] *** End of error message ***
0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEEN5boost3mpl6v_itemINS_24placeholder_definition4513interpolationINS_28FieldToParticleInterpolationINS_9particles6shapes3TSCENS_30AssignedTrilinearInterpolationEEENS1_24placeholder_definition2213pmacc_isAliasEEENS7_INS_24placeholder_definition385shapeISD_SH_EENS7_INS_24placeholder_definition3914particlePusherINSB_6pusher5ProbeESH_EENS6_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENS7_INS_24placeholder_definition326probeEENS7_INS_24placeholder_definition336probeBENS7_INS_24placeholder_definition258positionINS_24placeholder_definition2712position_picESH_EESU_Li0EEELi0EEELi0EEEED0Ev+0x80)[0x13e6540]
[kepler021:05426] [ 6] /home/carste06/SCRATCH/sims/pullrequest/test20/input/bin/picongpu(_ZN5pmacc13DataConnectorD2Ev+0x152)[0x1360322]
[kepler021:05426] [ 7] /usr/lib64/libc.so.6(+0x39bd9)[0x2aaaae19bbd9]
[kepler021:05426] [ 8] /usr/lib64/libc.so.6(+0x39c27)[0x2aaaae19bc27]
[kepler021:05426] [ 9] /usr/lib64/libc.so.6(__libc_start_main+0xfc)[0x2aaaae18444c]
[kepler021:05426] [10] /home/carste06/SCRATCH/sims/pullrequest/test20/input/bin/picongpu[0x11fc3df]
[kepler021:05426] *** End of error message ***
holder_definition4513interpolationINS_28FieldToParticleInterpolationINS_9particles6shapes3TSCENS_30AssignedTrilinearInterpolationEEENS1_24placeholder_definition2213pmacc_isAliasEEENS7_INS_24placeholder_definition385shapeISD_SH_EENS7_INS_24placeholder_definition3914particlePusherINSB_6pusher5ProbeESH_EENS6_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENS7_INS_24placeholder_definition326probeEENS7_INS_24placeholder_definition336probeBENS7_INS_24placeholder_definition258positionINS_24placeholder_definition2712position_picESH_EESU_Li0EEELi0EEELi0EEEED0Ev+0x80)[0x13e6540]
[kepler021:05427] [ 5] /home/carste06/SCRATCH/sims/pullrequest/test20/input/bin/picongpu(_ZN5pmacc13DataConnectorD2Ev+0x152)[0x1360322]
[kepler021:05427] [ 6] /usr/lib64/libc.so.6(+0x39bd9)[0x2aaaae19bbd9]
[kepler021:05427] [ 7] /usr/lib64/libc.so.6(+0x39c27)[0x2aaaae19bc27]
[kepler021:05427] [ 8] /usr/lib64/libc.so.6(__libc_start_main+0xfc)[0x2aaaae18444c]
[kepler021:05427] [ 9] /home/carste06/SCRATCH/sims/pullrequest/test20/input/bin/picongpu[0x11fc3df]
[kepler021:05427] *** End of error message ***

and the following output:

stdout

Running program...
PIConGPU: 0.5.0-dev
  Build-Type: Release

Third party:
  OS:         Linux-3.10.0-693.11.6.el7.x86_64
  arch:       x86_64
  CXX:        GNU (7.3.0)
  CMake:      3.11.3
  CUDA:       10.0.130
  mallocMC:   2.3.1
  Boost:      1.68.0
  MPI:        
    standard: 3.1
    flavor:   OpenMPI (2.1.6)
  PNGwriter:  0.7.0
  libSplash:  1.7.0 (Format 4.0)
  ADIOS:      1.13.1
PIConGPUVerbose PHYSICS(1) | Sliding Window is OFF
PIConGPUVerbose PHYSICS(1) | used Random Number Generator: RNGProvider2XorMin seed: 42
PIConGPUVerbose PHYSICS(1) | Courant c*dt <= 1.00056 ? 1
PIConGPUVerbose PHYSICS(1) | species e: omega_p * dt <= 0.1 ? 0.160229
PIConGPUVerbose PHYSICS(1) | species H: omega_p * dt <= 0.1 ? 0.00373928
PIConGPUVerbose PHYSICS(1) | species C: omega_p * dt <= 0.1 ? 0.00647688
PIConGPUVerbose PHYSICS(1) | species N: omega_p * dt <= 0.1 ? 0.00699409
PIConGPUVerbose PHYSICS(1) | y-cells per wavelength: 384
PIConGPUVerbose PHYSICS(1) | macro particles per device: 2457600
PIConGPUVerbose PHYSICS(1) | typical macro particle weighting: 504.037
PIConGPUVerbose PHYSICS(1) | UNIT_SPEED 2.99792e+08
PIConGPUVerbose PHYSICS(1) | UNIT_TIME 4.91113e-18
PIConGPUVerbose PHYSICS(1) | UNIT_LENGTH 1.47232e-09
PIConGPUVerbose PHYSICS(1) | UNIT_MASS 4.59147e-28
PIConGPUVerbose PHYSICS(1) | UNIT_CHARGE 8.07557e-17
PIConGPUVerbose PHYSICS(1) | UNIT_EFIELD 3.47071e+14
PIConGPUVerbose PHYSICS(1) | UNIT_BFIELD 1.1577e+06
PIConGPUVerbose PHYSICS(1) | UNIT_ENERGY 4.12661e-11
initialization time:  5sec 882msec = 5 sec
  0 % =        0 | time elapsed:                  387msec | avg time per step:   0msec
full simulation time:  6sec 272msec = 6 sec
--------------------------------------------------------------------------
mpiexec noticed that process rank 3 with PID 5428 on node kepler021 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

------------------------------
- - - - - job epilog - - - - -
------------------------------
Job ID: 1826624
was running on nodes: kepler021
by user: carste06
in partition: k80
using account: k80
number of CPUs used: 8
number of nodes requested: 1
------------------------------
walltime reqd: 02:00:00
walltime used: 00:00:16
------------------------------
Tue Jul 30 09:24:33 CEST 2019
------------------------------

cuda bug

Source

finnolec

👍2

Most helpful comment

I checked that after #3015 is merged, the Foil example with 4.cfg successfully ran on k20 and k80. Closing the issue.

sbastrakov on 12 Aug 2019

👍3

All 9 comments

I just started the simulation on the k20 partition on hemera and the error message looks like:

stderr

Unloading module pngwriter/0.7.0 
Unloading module libpng/1.6.35 
Unloading module libsplash/1.7.0-cuda100 
Unloading module hdf5-parallel/1.8.20-cuda100 
Unloading module adios/1.13.1-cuda100 
Unloading module c-blosc/1.14.4 
Unloading module zlib/1.2.11 
Unloading module boost/1.68.0 
Unloading module openmpi/2.1.2-cuda100 
Unloading module cuda/10.0 
Unloading module gcc/7.3.0 
Loading module gcc/7.3.0 
Loading module cmake/3.11.3 
Loading module cuda/10.0 
Loading module openmpi/2.1.2-cuda100 
Loading module boost/1.68.0 
Loading module zlib/1.2.11 
Loading module c-blosc/1.14.4 
Loading module adios/1.13.1-cuda100 
Loading module hdf5-parallel/1.8.20-cuda100 
Loading module libsplash/1.7.0-cuda100 
Loading module libpng/1.6.35 
Loading module pngwriter/0.7.0 
[1,0]<stderr>:Unhandled exception of type 'St13runtime_error' with message '/home/carste06/src/picongpu/thirdParty/alpaka/include/alpaka/event/EventCudaRt.hpp(211) 'cudaEventRecord( event.m_spEventImpl->m_CudaEvent, queue.m_spQueueImpl->m_CudaQueue)' A previous CUDA call (not this one) set the error  : 'cudaErrorInvalidDeviceFunction': 'invalid device function'!', terminating
[1,1]<stderr>:Unhandled exception of type 'St13runtime_error' with message '/home/carste06/src/picongpu/thirdParty/alpaka/include/alpaka/event/EventCudaRt.hpp(211) 'cudaEventRecord( event.m_spEventImpl->m_CudaEvent, queue.m_spQueueImpl->m_CudaQueue)' A previous CUDA call (not this one) set the error  : 'cudaErrorInvalidDeviceFunction': 'invalid device function'!', terminating
[1,2]<stderr>:Unhandled exception of type 'St13runtime_error' with message '/home/carste06/src/picongpu/thirdParty/alpaka/include/alpaka/event/EventCudaRt.hpp(211) 'cudaEventRecord( event.m_spEventImpl->m_CudaEvent, queue.m_spQueueImpl->m_CudaQueue)' A previous CUDA call (not this one) set the error  : 'cudaErrorInvalidDeviceFunction': 'invalid device function'!', terminating
[1,3]<stderr>:Unhandled exception of type 'St13runtime_error' with message '/home/carste06/src/picongpu/thirdParty/alpaka/include/alpaka/event/EventCudaRt.hpp(211) 'cudaEventRecord( event.m_spEventImpl->m_CudaEvent, queue.m_spQueueImpl->m_CudaQueue)' A previous CUDA call (not this one) set the error  : 'cudaErrorInvalidDeviceFunction': 'invalid device function'!', terminating
[1,0]<stderr>:*** Error in `/home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu': free(): invalid pointer: 0x000000000dad1240 ***
[1,2]<stderr>:[kepler002:02446] *** Process received signal ***
[1,2]<stderr>:[kepler002:02446] Signal: Segmentation fault (11)
[1,2]<stderr>:[kepler002:02446] Signal code: Address not mapped (1)
[1,2]<stderr>:[kepler002:02446] Failing at address: 0x10
[1,3]<stderr>:[kepler002:02447] *** Process received signal ***
[1,3]<stderr>:[kepler002:02447] Signal: Segmentation fault (11)
[1,3]<stderr>:[kepler002:02447] Signal code:  (128)
[1,3]<stderr>:[kepler002:02447] Failing at address: (nil)
[1,2]<stderr>:[kepler002:02446] [ 0] /usr/lib64/libpthread.so.0(+0xf6d0)[0x2aaaaacde6d0]
[1,2]<stderr>:[kepler002:02446] [ 1] [1,0]<stderr>:======= Backtrace: =========
[1,0]<stderr>:/usr/lib64/libc.so.6(+0x81499)[0x2aaaae1e3499]
[1,3]<stderr>:[kepler002:02447] [ 0] /usr/lib64/libpthread.so.0(+0xf6d0)[0x2aaaaacde6d0]
[1,3]<stderr>:[kepler002:02447] [ 1] [1,2]<stderr>:/trinity/shared/pkg/compiler/gcc/7.3.0/lib64/libstdc++.so.6(_ZSt28_Rb_tree_rebalance_for_erasePSt18_Rb_tree_node_baseRS_+0x21b)[0x2aaaada4198b]
[1,2]<stderr>:[kepler002:02446] [ 2] [1,0]<stderr>:/home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZN16cupla_cuda_async13cuplaFreeHostEPv+0xab)[0x153b71b]
[1,3]<stderr>:/home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZN16cupla_cuda_async13cuplaFreeHostEPv+0x5c)[0x153b6cc]
[1,3]<stderr>:[kepler002:02447] [ 2] [1,2]<stderr>:/home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZN16cupla_cuda_async13cuplaFreeHostEPv+0x80)[0x153b6f0]
[1,2]<stderr>:[kepler002:02446] [ 3] [1,3]<stderr>:/home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZN5pmacc16HostBufferInternINS_9SuperCellINS_5FrameINS_15ParticlesBufferINS_19ParticleDescriptionINS_4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEENS_4math2CT3IntILi16ELi16ELi2147483647EEEN5boost3mpl6v_itemIN8picongpu24placeholder_definition326probeEENSE_INSF_24placeholder_definition336probeBENSE_INSF_24placeholder_definition258positionINSF_24placeholder_definition2712position_picENS_24placeholder_definition2213pmacc_isAliasEEENSD_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENSE_INSF_24placeholder_definition4513interpolationINSF_28FieldToParticleInterpolationINSF_9particles6shapes3TSCENSF_30AssignedTrilinearInterpolationEEESP_EENSE_INSF_24placeholder_definition385shapeIS13_SP_EENSE_INSF_24placeholder_definition3914particlePusherINS11_6pusher5ProbeESP_EESU_Li0EEELi0EEELi0EEENS_17HandleGuardRegionINS_9particles8policies17ExchangeParticlesENS11_8boundary29CallPluginsAndDeleteParticlesEEESU_SU_EESB_N8mallocMC9AllocatorINS1Q_16CreationPolicies7ScatterINSF_16DeviceHeapConfigENS1S_11ScatterConf27DefaultScatterHashingParamsEEENS1Q_20DistributionPolicies4NoopENS1Q_11OOMPolicies10ReturnNullENS1Q_19ReservePoolPolicies16SimpleCudaMallocENS1Q_17AlignmentPolicies6ShrinkINS24_12ShrinkConfig19DefaultShrinkConfigEEEEELj2EE29OperatorCreatePairStaticArrayILj256EEENS4_IS7_SB_NSE_INS_24placeholder_definition249multiMaskENSE_INS_24placeholder_definition2312localCellIdxESX_Li0EEELi0EEES1H_S1O_SU_NSE_INS_12NextFramePtrINSS_3argILi1EEEEENSE_INS_16PreviousFramePtrIS2L_EESU_Li0EEELi0EEEEEEEEELj2EED1Ev+0x55)[0x13dc2b5]
[1,3]<stderr>:[kepler002:02447] [ 3] [1,0]<stderr>:/home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZN5pmacc6BufferINS_9SuperCellINS_5FrameINS_15ParticlesBufferINS_19ParticleDescriptionINS_4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEENS_4math2CT3IntILi16ELi16ELi2147483647EEEN5boost3mpl6v_itemIN8picongpu24placeholder_definition326probeEENSE_INSF_24placeholder_definition336probeBENSE_INSF_24placeholder_definition258positionINSF_24placeholder_definition2712position_picENS_24placeholder_definition2213pmacc_isAliasEEENSD_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENSE_INSF_24placeholder_definition4513interpolationINSF_28FieldToParticleInterpolationINSF_9particles6shapes3TSCENSF_30AssignedTrilinearInterpolationEEESP_EENSE_INSF_24placeholder_definition385shapeIS13_SP_EENSE_INSF_24placeholder_definition3914particlePusherINS11_6pusher5ProbeESP_EESU_Li0EEELi0EEELi0EEENS_17HandleGuardRegionINS_9particles8policies17ExchangeParticlesENS11_8boundary29CallPluginsAndDeleteParticlesEEESU_SU_EESB_N8mallocMC9AllocatorINS1Q_16CreationPolicies7ScatterINSF_16DeviceHeapConfigENS1S_11ScatterConf27DefaultScatterHashingParamsEEENS1Q_20DistributionPolicies4NoopENS1Q_11OOMPolicies10ReturnNullENS1Q_19ReservePoolPolicies16SimpleCudaMallocENS1Q_17AlignmentPolicies6ShrinkINS24_12ShrinkConfig19DefaultShrinkConfigEEEEELj2EE29OperatorCreatePairStaticArrayILj256EEENS4_IS7_SB_NSE_INS_24placeholder_definition249multiMaskENSE_INS_24placeholder_definition2312localCellIdxESX_Li0EEELi0EEES1H_S1O_SU_NSE_INS_12NextFramePtrINSS_3argILi1EEEEENSE_INS_16PreviousFramePtrIS2L_EESU_Li0EEELi0EEEEEEEEELj2EED2Ev+0x11)[0x13dc041]
[1,2]<stderr>:/home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZN5pmacc16HostBufferInternINS_9SuperCellINS_5FrameINS_15ParticlesBufferINS_19ParticleDescriptionINS_4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEENS_4math2CT3IntILi16ELi16ELi2147483647EEEN5boost3mpl6v_itemIN8picongpu24placeholder_definition326probeEENSE_INSF_24placeholder_definition336probeBENSE_INSF_24placeholder_definition258positionINSF_24placeholder_definition2712position_picENS_24placeholder_definition2213pmacc_isAliasEEENSD_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENSE_INSF_24placeholder_definition4513interpolationINSF_28FieldToParticleInterpolationINSF_9particles6shapes3TSCENSF_30AssignedTrilinearInterpolationEEESP_EENSE_INSF_24placeholder_definition385shapeIS13_SP_EENSE_INSF_24placeholder_definition3914particlePusherINS11_6pusher5ProbeESP_EESU_Li0EEELi0EEELi0EEENS_17HandleGuardRegionINS_9particles8policies17ExchangeParticlesENS11_8boundary29CallPluginsAndDeleteParticlesEEESU_SU_EESB_N8mallocMC9AllocatorINS1Q_16CreationPolicies7ScatterINSF_16DeviceHeapConfigENS1S_11ScatterConf27DefaultScatterHashingParamsEEENS1Q_20DistributionPolicies4NoopENS1Q_11OOMPolicies10ReturnNullENS1Q_19ReservePoolPolicies16SimpleCudaMallocENS1Q_17AlignmentPolicies6ShrinkINS24_12ShrinkConfig19DefaultShrinkConfigEEEEELj2EE29OperatorCreatePairStaticArrayILj256EEENS4_IS7_SB_NSE_INS_24placeholder_definition249multiMaskENSE_INS_24placeholder_definition2312localCellIdxESX_Li0EEELi0EEES1H_S1O_SU_NSE_INS_12NextFramePtrINSS_3argILi1EEEEENSE_INS_16PreviousFramePtrIS2L_EESU_Li0EEELi0EEEEEEEEELj2EED1Ev+0x55)[0x13dc2b5]
[1,2]<stderr>:[kepler002:02446] [ 4] [1,0]<stderr>:/home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZN5pmacc10GridBufferINS_9SuperCellINS_5FrameINS_15ParticlesBufferINS_19ParticleDescriptionINS_4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEENS_4math2CT3IntILi16ELi16ELi2147483647EEEN5boost3mpl6v_itemIN8picongpu24placeholder_definition326probeEENSE_INSF_24placeholder_definition336probeBENSE_INSF_24placeholder_definition258positionINSF_24placeholder_definition2712position_picENS_24placeholder_definition2213pmacc_isAliasEEENSD_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENSE_INSF_24placeholder_definition4513interpolationINSF_28FieldToParticleInterpolationINSF_9particles6shapes3TSCENSF_30AssignedTrilinearInterpolationEEESP_EENSE_INSF_24placeholder_definition385shapeIS13_SP_EENSE_INSF_24placeholder_definition3914particlePusherINS11_6pusher5ProbeESP_EESU_Li0EEELi0EEELi0EEENS_17HandleGuardRegionINS_9particles8policies17ExchangeParticlesENS11_8boundary29CallPluginsAndDeleteParticlesEEESU_SU_EESB_N8mallocMC9AllocatorINS1Q_16CreationPolicies7ScatterINSF_16DeviceHeapConfigENS1S_11ScatterConf27DefaultScatterHashingParamsEEENS1Q_20DistributionPolicies4NoopENS1Q_11OOMPolicies10ReturnNullENS1Q_19ReservePoolPolicies16SimpleCudaMallocENS1Q_17AlignmentPolicies6ShrinkINS24_12ShrinkConfig19DefaultShrinkConfigEEEEELj2EE29OperatorCreatePairStaticArrayILj256EEENS4_IS7_SB_NSE_INS_24placeholder_definition249multiMaskENSE_INS_24placeholder_definition2312localCellIdxESX_Li0EEELi0EEES1H_S1O_SU_NSE_INS_12NextFramePtrINSS_3argILi1EEEEENSE_INS_16PreviousFramePtrIS2L_EESU_Li0EEELi0EEEEEEEEELj2ES2T_ED1Ev+0xb7c)[0x13dd17c]
[1,0]<stderr>:/home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZThn32_N8picongpu9ParticlesIN5pmacc4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEEN5boost3mpl6v_itemINS_24placeholder_definition4513interpolationINS_28FieldToParticleInterpolationINS_9particles6shapes3TSCENS_30AssignedTrilinearInterpolationEEENS1_24placeholder_definition2213pmacc_isAliasEEENS7_INS_24placeholder_definition385shapeISD_SH_EENS7_INS_24placeholder_definition3914particlePusherINSB_6pusher5ProbeESH_EENS6_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENS7_INS_24placeholder_definition326probeEENS7_INS_24placeholder_definition336probeBENS7_INS_24placeholder_definition258positionINS_24placeholder_definition2712position_picESH_EESU_Li0EEELi0EEELi0EEEED0Ev+0x80)[0x13e6540]
[1,0]<stderr>:/home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZN5pmacc13DataConnectorD2Ev+0x152)[0x1360322]
[1,0]<stderr>:/usr/lib64/libc.so.6(+0x39bd9)[0x2aaaae19bbd9]
[1,0]<stderr>:/usr/lib64/libc.so.6(+0x39c27)[0x2aaaae19bc27]
[1,0]<stderr>:/usr/lib64/libc.so.6(__libc_start_main+0xfc)[0x2aaaae18444c]
[1,0]<stderr>:/home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu[0x11fc3df]
[1,0]<stderr>:======= Memory map: ========
[1,0]<stderr>:00400000-01c0d000 r-xp 00000000 00:2e 2568948                            /bigdata/hplsim/scratch/carste06/sims/pullrequest/test21/input/bin/picongpu
[1,0]<stderr>:01e0d000-01e11000 r--p 0180d000 00:2e 2568948                            /bigdata/hplsim/scratch/carste06/sims/pullrequest/test21/input/bin/picongpu
[1,0]<stderr>:01e11000-01e14000 rw-p 01811000 00:2e 2568948                            /bigdata/hplsim/scratch/carste06/sims/pullrequest/test21/input/bin/picongpu
[1,0]<stderr>:01e14000-02003000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:02003000-02008000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:02008000-0207a000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:0207a000-0208a000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:0208a000-020cc000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:020cc000-020cd000 rw-p 00000000 00:00 0                                  [h[1,1]<stderr>:*** Error in `/home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu': free(): invalid pointer: 0x000000000dabc990 ***
[1,1]<stderr>:======= Backtrace: =========
[1,1]<stderr>:/usr/lib64/libc.so.6(+0x81499)[0x2aaaae1e3499]
[1,1]<stderr>:/home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZN16cupla_cuda_async13cuplaFreeHostEPv+0xab)[0x153b71b]
[1,1]<stderr>:/home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZN5pmacc6BufferINS_9SuperCellINS_5FrameINS_15ParticlesBufferINS_19ParticleDescriptionINS_4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEENS_4math2CT3IntILi16ELi16ELi2147483647EEEN5boost3mpl6v_itemIN8picongpu24placeholder_definition326probeEENSE_INSF_24placeholder_definition336probeBENSE_INSF_24placeholder_definition258positionINSF_24placeholder_definition2712position_picENS_24placeholder_definition2213pmacc_isAliasEEENSD_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENSE_INSF_24placeholder_definition4513interpolationINSF_28FieldToParticleInterpolationINSF_9particles6shapes3TSCENSF_30AssignedTrilinearInterpolationEEESP_EENSE_INSF_24placeholder_definition385shapeIS13_SP_EENSE_INSF_24placeholder_definition3914particlePusherINS11_6pusher5ProbeESP_EESU_Li0EEELi0EEELi0EEENS_17HandleGuardRegionINS_9particles8policies17ExchangeParticlesENS11_8boundary29CallPluginsAndDeleteParticlesEEESU_SU_EESB_N8mallocMC9AllocatorINS1Q_16CreationPolicies7ScatterINSF_16DeviceHeapConfigENS1S_11ScatterConf27DefaultScatterHashingParamsEEENS1Q_20DistributionPolicies4NoopENS1Q_11OOMPolicies10ReturnNullENS1Q_19ReservePoolPolicies16SimpleCudaMallocENS1Q_17AlignmentPolicies6ShrinkINS24_12ShrinkConfig19DefaultShrinkConfigEEEEELj2EE29OperatorCreatePairStaticArrayILj256EEENS4_IS7_SB_NSE_INS_24placeholder_definition249multiMaskENSE_INS_24placeholder_definition2312localCellIdxESX_Li0EEELi0EEES1H_S1O_SU_NSE_INS_12NextFramePtrINSS_3argILi1EEEEENSE_INS_16PreviousFramePtrIS2L_EESU_Li0EEELi0EEEEEEEEELj2EED2Ev+0x11)[0x13dc041]
[1,1]<stderr>:/home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZN5pmacc10GridBufferINS_9SuperCellINS_5FrameINS_15ParticlesBufferINS_19ParticleDescriptionINS_4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEENS_4math2CT3IntILi16ELi16ELi2147483647EEEN5boost3mpl6v_itemIN8picongpu24placeholder_definition326probeEENSE_INSF_24placeholder_definition336probeBENSE_INSF_24placeholder_definition258positionINSF_24placeholder_definition2712position_picENS_24placeholder_definition2213pmacc_isAliasEEENSD_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENSE_INSF_24placeholder_definition4513interpolationINSF_28FieldToParticleInterpolationINSF_9particles6shapes3TSCENSF_30AssignedTrilinearInterpolationEEESP_EENSE_INSF_24placeholder_definition385shapeIS13_SP_EENSE_INSF_24placeholder_definition3914particlePusherINS11_6pusher5ProbeESP_EESU_Li0EEELi0EEELi0EEENS_17HandleGuardRegionINS_9particles8policies17ExchangeParticlesENS11_8boundary29CallPluginsAndDeleteParticlesEEESU_SU_EESB_N8mallocMC9AllocatorINS1Q_16CreationPolicies7ScatterINSF_16DeviceHeapConfigENS1S_11ScatterConf27DefaultScatterHashingParamsEEENS1Q_20DistributionPolicies4NoopENS1Q_11OOMPolicies10ReturnNullENS1Q_19ReservePoolPolicies16SimpleCudaMallocENS1Q_17AlignmentPolicies6ShrinkINS24_12ShrinkConfig19DefaultShrinkConfigEEEEELj2EE29OperatorCreatePairStaticArrayILj256EEENS4_IS7_SB_NSE_INS_24placeholder_definition249multiMaskENSE_INS_24placeholder_definition2312localCellIdxESX_Li0EEELi0EEES1H_S1O_SU_NSE_INS_12NextFramePtrINSS_3argILi1EEEEENSE_INS_16PreviousFramePtrIS2L_EESU_Li0EEELi0EEEEEEEEELj2ES2T_ED1Ev+0xb7c)[0x13dd17c]
[1,1]<stderr>:/home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZThn32_N8picongpu9ParticlesIN5pmacc4meta6StringIJLc112ELc114ELc111ELc98ELc101E[1,3]<stderr>:/home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZN5pmacc10GridBufferINS_9SuperCellINS_5FrameINS_15ParticlesBufferINS_19ParticleDescriptionINS_4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEENS_4math2CT3IntILi16ELi16ELi2147483647EEEN5boost3mpl6v_itemIN8picongpu24placeholder_definition326probeEENSE_INSF_24placeholder_definition336probeBENSE_INSF_24placeholder_definition258positionINSF_24placeholder_definition2712position_picENS_24placeholder_definition2213pmacc_isAliasEEENSD_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENSE_INSF_24placeholder_definition4513interpolationINSF_28FieldToParticleInterpolationINSF_9particles6shapes3TSCENSF_30AssignedTrilinearInterpolationEEESP_EENSE_INSF_24placeholder_definition385shapeIS13_SP_EENSE_INSF_24placeholder_definition3914particlePusherINS11_6pusher5ProbeESP_EESU_Li0EEELi0EEELi0EEENS_17HandleGuardRegionINS_9particles8policies17ExchangeParticlesENS11_8boundary29CallPluginsAndDeleteParticlesEEESU_SU_EESB_N8mallocMC9AllocatorINS1Q_16CreationPolicies7ScatterINSF_16DeviceHeapConfigENS1S_11ScatterConf27DefaultScatterHashingParamsEEENS1Q_20DistributionPolicies4NoopENS1Q_11OOMPolicies10ReturnNullENS1Q_19ReservePoolPolicies16SimpleCudaMallocENS1Q_17AlignmentPolicies6ShrinkINS24_12ShrinkConfig19DefaultShrinkConfigEEEEELj2EE29OperatorCreatePairStaticArrayILj256EEENS4_IS7_SB_NSE_INS_24placeholder_definition249multiMaskENSE_INS_24placeholder_definition2312localCellIdxESX_Li0EEELi0EEES1H_S1O_SU_NSE_INS_12NextFramePtrINSS_3argILi1EEEEENSE_INS_16PreviousFramePtrIS2L_EESU_Li0EEELi0EEEEEEEEELj2ES2T_ED1Ev+0xb7c)[0x13dd17c]
[1,3]<stderr>:[kepler002:02447] [ 4] /home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZThn32_N8picongpu9ParticlesIN5pmacc4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEEN5boost3mpl6v_itemINS_24placeholder_definition4513interpolationINS_28FieldToParticleInterpolationINS_9particles6shapes3TSCENS_30AssignedTrilinearInterpolationEEENS1_24placeholder_definition2213pmacc_isAliasEEENS7_INS_24placeholder_definition385shapeISD_SH_EENS7_INS_24placeholder_definition3914particlePusherINSB_6pusher5ProbeESH_EENS6_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENS7_INS_24placeholder_definition326probeEENS7_INS_24placeholder_definition336probeBENS7_INS_24placeholder_definition258positionINS_24placeholder_definition2712position_picESH_EESU_Li0EEELi0EEELi0EEEED0Ev+0x80)[0x13e6540]
[1,3]<stderr>:[kepler002:02447] [ 5] /home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZN5pmacc13DataConnectorD2Ev+0x152)[0x1360322]
[1,3]<stderr>:[kepler002:02447] [ 6] /usr/lib64/libc.so.6(+0x39bd9)[0x2aaaae19bbd9]
[1,3]<stderr>:[kepler002:02447] [ 7] /usr/lib64/libc.so.6(+0x39c27)[0x2aaaae19bc27]
[1,3]<stderr>:[kepler002:02447] [ 8] /usr/lib64/libc.so.6(__libc_start_main+0xfc)[0x2aaaae18444c]
[1,3]<stderr>:[kepler002:02447] [ 9] /home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu[0x11fc3df]
[1,3]<stderr>:[kepler002:02447] *** End of error message ***
[1,2]<stderr>:/home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZN5pmacc10GridBufferINS_9SuperCellINS_5FrameINS_15ParticlesBufferINS_19ParticleDescriptionINS_4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEENS_4math2CT3IntILi16ELi16ELi2147483647EEEN5boost3mpl6v_itemIN8picongpu24placeholder_definition326probeEENSE_INSF_24placeholder_definition336probeBENSE_INSF_24placeholder_definition258positionINSF_24placeholder_definition2712position_picENS_24placeholder_definition2213pmacc_isAliasEEENSD_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENSE_INSF_24placeholder_definition4513interpolationINSF_28FieldToParticleInterpolationINSF_9particles6shapes3TSCENSF_30AssignedTrilinearInterpolationEEESP_EENSE_INSF_24placeholder_definition385shapeIS13_SP_EENSE_INSF_24placeholder_definition3914particlePusherINS11_6pusher5ProbeESP_EESU_Li0EEELi0EEELi0EEENS_17HandleGuardRegionINS_9particles8policies17ExchangeParticlesENS11_8boundary29CallPluginsAndDeleteParticlesEEESU_SU_EESB_N8mallocMC9AllocatorINS1Q_16CreationPolicies7ScatterINSF_16DeviceHeapConfigENS1S_11ScatterConf27DefaultScatterHashingParamsEEENS1Q_20DistributionPolicies4NoopENS1Q_11OOMPolicies10ReturnNullENS1Q_19ReservePoolPolicies16SimpleCudaMallocENS1Q_17AlignmentPolicies6ShrinkINS24_12ShrinkConfig19DefaultShrinkConfigEEEEELj2EE29OperatorCreatePairStaticArrayILj256EEENS4_IS7_SB_NSE_INS_24placeholder_definition249multiMaskENSE_INS_24placeholder_definition2312localCellIdxESX_Li0EEELi0EEES1H_S1O_SU_NSE_INS_12NextFramePtrINSS_3argILi1EEEEENSE_INS_16PreviousFramePtrIS2L_EESU_Li0EEELi0EEEEEEEEELj2ES2T_ED1Ev+0xb7c)[0x13dd17c]
[1,2]<stderr>:[kepler002:02446] [ 5] /home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZThn32_N8picongpu9ParticlesIN5pmacc4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEEN5boost3mpl6v_itemINS_24placeholder_definition4513interpolationINS_28FieldToParticleInterpolationINS_9particles6shapes3TSCENS_30AssignedTrilinearInterpolationEEENS1_24placeholder_definition2213pmacc_isAliasEEENS7_INS_24placeholder_definition385shapeISD_SH_EENS7_INS_24placeholder_definition3914particlePusherINSB_6pusher5ProbeESH_EENS6_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENS7_INS_24placeholder_definition326probeEENS7_INS_24placeholder_definition336probeBENS7_INS_24placeholder_definition258positionINS_24placeholder_definition2712position_picESH_EESU_Li0EEELi0EEELi0EEEED0Ev+0x80)[0x13e6540]
[1,2]<stderr>:[kepler002:02446] [ 6] /home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZN5pmacc13DataConnectorD2Ev+0x152)[0x1360322]
[1,2]<stderr>:[kepler002:02446] [ 7] /usr/lib64/libc.so.6(+0x39bd9)[0x2aaaae19bbd9]
[1,2]<stderr>:[kepler002:02446] [ 8] /usr/lib64/libc.so.6(+0x39c27)[0x2aaaae19bc27]
[1,2]<stderr>:[kepler002:02446] [ 9] /usr/lib64/libc.so.6(__libc_start_main+0xfc)[0x2aaaae18444c]
[1,2]<stderr>:[kepler002:02446] [10] /home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu[0x11fc3df]
[1,2]<stderr>:[kepler002:02446] *** End of error message ***
[1,0]<stderr>:eap]
[1,0]<stderr>:020cd000-020d2000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:020d2000-020d3000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:020d3000-020d5000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:020d5000-020d6000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:020d6000-020d8000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:020d8000-020dd000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:020dd000-020de000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:020de000-020e3000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:020e3000-020e5000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:020e5000-020fe000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:020fe000-020ff000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:020ff000-02118000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:02118000-0212b000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:0212b000-0212d000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:0212d000-02130000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:02130000-02132000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:02132000-02135000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:02135000-02146000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:02146000-02149000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:02149000-0215a000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:0215a000-0215d000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:0215d000-0216e000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:0216e000-02171000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:02171000-02182000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:02182000-02185000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:02185000-02196000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:02196000-02199000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:02199000-021aa000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:021aa000-021ad000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:021ad000-021be000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:021be000-021c1000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:021c1000-021d2000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:021d2000-118e5000 rw-p 00000000 00:00 0                                  [heap]
[1,0]<stderr>:200000000-200100000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:200100000-200104000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:200104000-200120000 ---p 00000000 00:00 0 
[1,0]<stderr>:200120000-200520000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:200520000-200524000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:200524000-200540000 ---p 00000000 00:00 0 
[1,0]<stderr>:200540000-200940000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:200940000-200944000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:200944000-200960000 ---p 00000000 00:00 0 
[1,0]<stderr>:200960000-200d60000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:200d60000-200d64000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:200d64000-200d80000 ---p 00000000 00:00 0 
[1,0]<stderr>:200d80000-201180000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:201180000-201184000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:201184000-2011a0000 ---p 00000000 00:00 0 
[1,0]<stderr>:2011a0000-2015a0000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:2015a0000-2015a4000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:2015a4000-2015c0000 ---p 00000000 00:00 0 
[1,0]<stderr>:2015c0000-2019c0000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:2019c0000-2019c4000 rw-s 00000000 00:05 18613    [1,1]<stderr>:Lc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEEN5boost3mpl6v_itemINS_24placeholder_definition4513interpolationINS_28FieldToParticleInterpolationINS_9particles6shapes3TSCENS_30AssignedTrilinearInterpolationEEENS1_24placeholder_definition2213pmacc_isAliasEEENS7_INS_24placeholder_definition385shapeISD_SH_EENS7_INS_24placeholder_definition3914particlePusherINSB_6pusher5ProbeESH_EENS6_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENS7_INS_24placeholder_definition326probeEENS7_INS_24placeholder_definition336probeBENS7_INS_24placeholder_definition258positionINS_24placeholder_definition2712position_picESH_EESU_Li0EEELi0EEELi0EEEED0Ev+0x80)[0x13e6540]
[1,1]<stderr>:/home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZN5pmacc13DataConnectorD2Ev+0x152)[0x1360322]
[1,1]<stderr>:/usr/lib64/libc.so.6(+0x39bd9)[0x2aaaae19bbd9]
[1,1]<stderr>:/usr/lib64/libc.so.6(+0x39c27)[0x2aaaae19bc27]
[1,1]<stderr>:/usr/lib64/libc.so.6(__libc_start_main+0xfc)[0x2aaaae18444c]
[1,1]<stderr>:/home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu[0x11fc3df]
[1,1]<stderr>:======= Memory map: ========
[1,1]<stderr>:00400000-01c0d000 r-xp 00000000 00:2e 2568948                            /bigdata/hplsim/scratch/carste06/sims/pullrequest/test21/input/bin/picongpu
[1,1]<stderr>:01e0d000-01e11000 r--p 0180d000 00:2e 2568948                            /bigdata/hplsim/scratch/carste06/sims/pullrequest/test21/input/bin/picongpu
[1,1]<stderr>:01e11000-01e14000 rw-p 01811000 00:2e 2568948                            /bigdata/hplsim/scratch/carste06/sims/pullrequest/test21/input/bin/picongpu
[1,1]<stderr>:01e14000-02004000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:02004000-02009000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:02009000-0207a000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:0207a000-0208a000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:0208a000-020cd000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:020cd000-020ce000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:020ce000-020d3000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:020d3000-020d4000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:020d4000-020d6000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:020d6000-020d7000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:020d7000-020d9000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:020d9000-020de000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:020de000-020df000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:020df000-020e4000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:020e4000-020e6000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:020e6000-020ff000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:020ff000-02100000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:02100000-02119000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:02119000-0212c000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:0212c000-0212e000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:0212e000-02131000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:02131000-02133000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:02133000-02136000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:02136000-02147000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:02147000-0214a000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:0214a000-0215b000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:0215b000-0215e000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:0215e000-0216f000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:0216f000-02172000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:02172000-02183000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:02183000-02186000 rw-p 00000000 00:0[1,0]<stderr>:                        /dev/nvidiactl
[1,0]<stderr>:2019c4000-2019e0000 ---p 00000000 00:00 0 
[1,0]<stderr>:2019e0000-201de0000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:201de0000-201de4000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:201de4000-201e00000 ---p 00000000 00:00 0 
[1,0]<stderr>:201e00000-202200000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:202200000-202204000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:202204000-202220000 ---p 00000000 00:00 0 
[1,0]<stderr>:202220000-202620000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:202620000-202624000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:202624000-202640000 ---p 00000000 00:00 0 
[1,0]<stderr>:202640000-202a40000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:202a40000-202a44000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:202a44000-202a60000 ---p 00000000 00:00 0 
[1,0]<stderr>:202a60000-202e60000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:202e60000-202e64000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:202e64000-202e80000 ---p 00000000 00:00 0 
[1,0]<stderr>:202e80000-203280000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:203280000-203284000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:203284000-2032a0000 ---p 00000000 00:00 0 
[1,0]<stderr>:2032a0000-2036a0000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:2036a0000-2036a4000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:2036a4000-2036c0000 ---p 00000000 00:00 0 
[1,0]<stderr>:2036c0000-203ac0000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:203ac0000-203ac4000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:203ac4000-203ae0000 ---p 00000000 00:00 0 
[1,0]<stderr>:203ae0000-203ee0000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:203ee0000-203ee4000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:203ee4000-203f00000 ---p 00000000 00:00 0 
[1,0]<stderr>:203f00000-204300000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:204300000-204400000 rw-s 00000000 00:04 33625057                         /dev/zero (deleted)
[1,0]<stderr>:204400000-204500000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:204500000-204600000 rw-s 00000000 00:04 33625058                         /dev/zero (deleted)
[1,0]<stderr>:204600000-204700000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:204700000-204800000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:204800000-204882000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,0]<stderr>:204882000-2048a0000 ---p 00000000 00:00 0 
[1,0]<stderr>:2048a0000-2049a0000 rw-s 00000000 00:04 33625060                         /dev/zero (deleted)
[1,0]<stderr>:2049a0000-208ec0000 ---p 00000000 00:00 0 
[1,0]<stderr>:208ec0000-208fc1000 rw-s 00000000 00:04 33625061                         /dev/zero (deleted)
[1,0]<stderr>:208fc1000-1900000000 ---p 00000000 00:00 0 
[1,0]<stderr>:2aaaaaaab000-2aaaaaacd000 r-xp 00000000 08:02 1059967                    /usr/lib64/ld-2.17.so
[1,0]<stderr>:2aaaaaacd000-2aaaaaacf000 r-xp 00000000 00:00 0                          [vdso]
[1,0]<stderr>:2aaaaaacf000-2aaaaaad0000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaaaaad0000-2aaaaaad1000 r--s 00000000 00:05 18894                      /dev/nvidia0
[1,0]<stderr>:2aaaaaad1000-2aaaaaad2000 r--s 00000000 00:05 18898                      /dev/nvidia1
[1,0]<stderr>:2aaaaaad2000-2aaaaaad3000 r--s 00000000 00:05 23565                      /dev/nvidia2
[1,0]<stderr>:2aaaaaad3000-2aaaaaad4000 r--s 00000000 00:05 21067                      /dev/nvidia3
[1,0]<stderr>:2aaaaaad4000-2aaaaaad5000 r--s 00000000 08:02 1966097                    /tmp/openmpi-sessions-9413@kepler002_0/6259/pmix_dstor_2439/initial-pmix_shared-segment-0
[1,0]<stderr>:2aaaaaad5000-2aaaaaad6000 rw-s 00000000 08:02 1966098                    /tmp/openmpi-sessions-9413@kepler002_0/6259/pmix_dstor_2439/dstore_sm.lock
[1,0]<stderr>:2aaaaaad6000-2aaaaaad7000 -w-s 00000000 00:05 23595                      /dev/infiniband/uverbs0
[1,0]<stderr>:2aaaaaad7000-2aaaaaad8000 -w-s 00001000 00:05 23595                      /dev/infiniband/uverbs0
[1,0]<stderr>:2aaaaaad8000-2aaaaaad9000 r-[1,1]<stderr>:0 0                                  [heap]
[1,1]<stderr>:02186000-02197000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:02197000-0219a000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:0219a000-021ab000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:021ab000-021ae000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:021ae000-021bf000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:021bf000-021c2000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:021c2000-021d3000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:021d3000-11846000 rw-p 00000000 00:00 0                                  [heap]
[1,1]<stderr>:200000000-200100000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:200100000-200104000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:200104000-200120000 ---p 00000000 00:00 0 
[1,1]<stderr>:200120000-200520000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:200520000-200524000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:200524000-200540000 ---p 00000000 00:00 0 
[1,1]<stderr>:200540000-200940000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:200940000-200944000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:200944000-200960000 ---p 00000000 00:00 0 
[1,1]<stderr>:200960000-200d60000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:200d60000-200d64000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:200d64000-200d80000 ---p 00000000 00:00 0 
[1,1]<stderr>:200d80000-201180000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:201180000-201184000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:201184000-2011a0000 ---p 00000000 00:00 0 
[1,1]<stderr>:2011a0000-2015a0000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:2015a0000-2015a4000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:2015a4000-2015c0000 ---p 00000000 00:00 0 
[1,1]<stderr>:2015c0000-2019c0000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:2019c0000-2019c4000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:2019c4000-2019e0000 ---p 00000000 00:00 0 
[1,1]<stderr>:2019e0000-201de0000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:201de0000-201de4000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:201de4000-201e00000 ---p 00000000 00:00 0 
[1,1]<stderr>:201e00000-202200000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:202200000-202204000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:202204000-202220000 ---p 00000000 00:00 0 
[1,1]<stderr>:202220000-202620000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:202620000-202624000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:202624000-202640000 ---p 00000000 00:00 0 
[1,1]<stderr>:202640000-202a40000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:202a40000-202a44000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:202a44000-202a60000 ---p 00000000 00:00 0 
[1,1]<stderr>:202a60000-202e60000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:202e60000-202e64000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:202e64000-202e80000 ---p 00000000 00:00 0 
[1,1]<stderr>:202e80000-203280000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:203280000-203284000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:203284000-2032a0000 ---p 00000000 00:00 0 
[1,1]<stderr>:2032a0000-2036a0000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:2036a0000-2036a4000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:2036a4000-2036c0000 ---p 00000000 00:00 0 
[1,1]<stderr>:2036c0000-203ac0000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:203ac0000-203ac4000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:203ac4000-203ae0000 ---p 00000000 00:00 0 
[1,1]<stderr>:203ae0000-203ee0000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:203ee0000-203ee4000 rw-s 00000000 00:05[1,0]<stderr>:-s 00003000 00:05 23595                      /dev/infiniband/uverbs0
[1,0]<stderr>:2aaaaaad9000-2aaaaaadb000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaaaaadb000-2aaaaaadc000 rw-s 00000000 08:02 1966108                    /tmp/openmpi-sessions-9413@kepler002_0/6259/1/shared_mem_btl_module.kepler002 (deleted)
[1,0]<stderr>:2aaaaaadc000-2aaaaaadd000 -w-s 00000000 00:05 23595                      /dev/infiniband/uverbs0
[1,0]<stderr>:2aaaaaadd000-2aaaaaade000 -w-s 00001000 00:05 23595                      /dev/infiniband/uverbs0
[1,0]<stderr>:2aaaaaade000-2aaaaaadf000 r--s 00003000 00:05 23595                      /dev/infiniband/uverbs0
[1,0]<stderr>:2aaaaaadf000-2aaaaaae0000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaaaaae0000-2aaaaaae3000 rw-s 00e02000 00:05 23595                      /dev/infiniband/uverbs0
[1,0]<stderr>:2aaaaaae3000-2aaaaaae4000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaaaaae4000-2aaaaab95000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaaaab95000-2aaaaab99000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaaaab99000-2aaaaab9a000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,0]<stderr>:2aaaaab9a000-2aaaaab9c000 rw-s 00d02000 00:05 23595                      /dev/infiniband/uverbs0
[1,0]<stderr>:2aaaaab9c000-2aaaaab9e000 rw-s 00d02000 00:05 23595                      /dev/infiniband/uverbs0
[1,0]<stderr>:2aaaaab9e000-2aaaaab9f000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,0]<stderr>:2aaaaab9f000-2aaaaaba0000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,0]<stderr>:2aaaaaba0000-2aaaaabb0000 rw-s 01002000 00:05 23595                      /dev/infiniband/uverbs0
[1,0]<stderr>:2aaaaabb0000-2aaaaabc0000 rw-s 01002000 00:05 23595                      /dev/infiniband/uverbs0
[1,0]<stderr>:2aaaaabc0000-2aaaaabe2000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaaaabe2000-2aaaaac12000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaaaac12000-2aaaaac13000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,0]<stderr>:2aaaaac13000-2aaaaac14000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,0]<stderr>:2aaaaac14000-2aaaaac15000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,0]<stderr>:2aaaaac15000-2aaaaac16000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,0]<stderr>:2aaaaac16000-2aaaaac17000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,0]<stderr>:2aaaaac17000-2aaaaac18000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,0]<stderr>:2aaaaac18000-2aaaaac20000 rw-s 00f02000 00:05 23595                      /dev/infiniband/uverbs0
[1,0]<stderr>:2aaaaac20000-2aaaaac30000 rw-s 01002000 00:05 23595                      /dev/infiniband/uverbs0
[1,0]<stderr>:2aaaaac30000-2aaaaac31000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaaaac31000-2aaaaac92000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaaaac92000-2aaaaac93000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaaaac93000-2aaaaac94000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,0]<stderr>:2aaaaac94000-2aaaaac95000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,0]<stderr>:2aaaaac95000-2aaaaac96000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,0]<stderr>:2aaaaac96000-2aaaaac97000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,0]<stderr>:2aaaaac97000-2aaaaac98000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,0]<stderr>:2aaaaac98000-2aaaaac99000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,0]<stderr>:2aaaaac99000-2aaaaac9a000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,0]<stderr>:2aaaaac9a000-2aaaaac9c000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaaaaccc000-2aaaaaccd000 r--p 00021000 08:02 1059967                    /usr/lib64/ld-2.17.so
[1,0]<stderr>:2aaaaaccd000-2aaaaacce000 rw-p 00022000 08:02 1059967                    /usr/lib64/ld-2.17.so
[1,0]<stderr>:2aaaaacce000-2aaaaaccf000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaaaaccf000-2aaaaace6000 r-xp 00000000 08:02 1060375                    /usr/lib64/libpthread-2.17.so
[1,0]<stderr>:2aaaaace6000-2aaaaaee5000 ---p 00017000 08:02 1060375                    /usr/lib64/libpthread-2.17.so
[1,0]<stderr>:2aaaaaee5000-2aaaaaee6000 r--p 00016000 08:02 1060375                    /usr/lib64/libpthread-2.17.so
[1,0]<stderr>:2aaaaaee6000-2aaaaaee7000 rw-p 00017000 08:02 1060375                    /usr/lib64/libpthread-2.17.so
[1,0]<stderr>:2aaaaaee7000-2aaaaaeeb000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaaaaeeb000-2aaaaaeed000 r-xp 00000000 08:02 1060083                    /usr/lib64/libdl-2.17.so
[1,0]<stderr>:2aaaaaeed000-2aaaab0ed000 ---p 00002000 08:02 1060083                    /usr/lib64/libdl-2.17.so
[1,0]<stderr>:2aaaab0ed000-2aaaab0ee000 r--p 00002000 08:02 1060083[1,1]<stderr>: 18613                            /dev/nvidiactl
[1,1]<stderr>:203ee4000-203f00000 ---p 00000000 00:00 0 
[1,1]<stderr>:203f00000-204300000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:204300000-204400000 rw-s 00000000 00:04 33641541                         /dev/zero (deleted)
[1,1]<stderr>:204400000-204500000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:204500000-204600000 rw-s 00000000 00:04 33641542                         /dev/zero (deleted)
[1,1]<stderr>:204600000-204700000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:204700000-204800000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:204800000-204882000 rw-s 00000000 00:05 18613                            /dev/nvidiactl
[1,1]<stderr>:204882000-2048a0000 ---p 00000000 00:00 0 
[1,1]<stderr>:2048a0000-2049a0000 rw-s 00000000 00:04 33641544                         /dev/zero (deleted)
[1,1]<stderr>:2049a0000-208ec0000 ---p 00000000 00:00 0 
[1,1]<stderr>:208ec0000-208fc1000 rw-s 00000000 00:04 33639266                         /dev/zero (deleted)
[1,1]<stderr>:208fc1000-1900000000 ---p 00000000 00:00 0 
[1,1]<stderr>:2aaaaaaab000-2aaaaaacd000 r-xp 00000000 08:02 1059967                    /usr/lib64/ld-2.17.so
[1,1]<stderr>:2aaaaaacd000-2aaaaaacf000 r-xp 00000000 00:00 0                          [vdso]
[1,1]<stderr>:2aaaaaacf000-2aaaaaad0000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaaaaad0000-2aaaaaad1000 r--s 00000000 00:05 18894                      /dev/nvidia0
[1,1]<stderr>:2aaaaaad1000-2aaaaaad2000 r--s 00000000 00:05 18898                      /dev/nvidia1
[1,1]<stderr>:2aaaaaad2000-2aaaaaad3000 r--s 00000000 00:05 23565                      /dev/nvidia2
[1,1]<stderr>:2aaaaaad3000-2aaaaaad4000 r--s 00000000 00:05 21067                      /dev/nvidia3
[1,1]<stderr>:2aaaaaad4000-2aaaaaad5000 r--s 00000000 08:02 1966097                    /tmp/openmpi-sessions-9413@kepler002_0/6259/pmix_dstor_2439/initial-pmix_shared-segment-0
[1,1]<stderr>:2aaaaaad5000-2aaaaaad6000 rw-s 00000000 08:02 1966098                    /tmp/openmpi-sessions-9413@kepler002_0/6259/pmix_dstor_2439/dstore_sm.lock
[1,1]<stderr>:2aaaaaad6000-2aaaaaad7000 -w-s 00000000 00:05 23595                      /dev/infiniband/uverbs0
[1,1]<stderr>:2aaaaaad7000-2aaaaaad8000 -w-s 00001000 00:05 23595                      /dev/infiniband/uverbs0
[1,1]<stderr>:2aaaaaad8000-2aaaaaad9000 r--s 00003000 00:05 23595                      /dev/infiniband/uverbs0
[1,1]<stderr>:2aaaaaad9000-2aaaaaadb000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaaaaadb000-2aaaaaadc000 -w-s 00000000 00:05 23595                      /dev/infiniband/uverbs0
[1,1]<stderr>:2aaaaaadc000-2aaaaaadd000 -w-s 00001000 00:05 23595                      /dev/infiniband/uverbs0
[1,1]<stderr>:2aaaaaadd000-2aaaaaade000 r--s 00003000 00:05 23595                      /dev/infiniband/uverbs0
[1,1]<stderr>:2aaaaaade000-2aaaaaae0000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaaaaae0000-2aaaaaae3000 rw-s 00e02000 00:05 23595                      /dev/infiniband/uverbs0
[1,1]<stderr>:2aaaaaae3000-2aaaaaae4000 rw-s 00000000 08:02 1966108                    /tmp/openmpi-sessions-9413@kepler002_0/6259/1/shared_mem_btl_module.kepler002 (deleted)
[1,1]<stderr>:2aaaaaae4000-2aaaaab95000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaaaab95000-2aaaaab99000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaaaab99000-2aaaaab9a000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,1]<stderr>:2aaaaab9a000-2aaaaab9c000 rw-s 00d02000 00:05 23595                      /dev/infiniband/uverbs0
[1,1]<stderr>:2aaaaab9c000-2aaaaab9e000 rw-s 00d02000 00:05 23595                      /dev/infiniband/uverbs0
[1,1]<stderr>:2aaaaab9e000-2aaaaab9f000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,1]<stderr>:2aaaaab9f000-2aaaaaba0000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,1]<stderr>:2aaaaaba0000-2aaaaabb0000 rw-s 01002000 00:05 23595                      /dev/infiniband/uverbs0
[1,1]<stderr>:2aaaaabb0000-2aaaaabc0000 rw-s 01002000 00:05 23595                      /dev/infiniband/uverbs0
[1,1]<stderr>:2aaaaabc0000-2aaaaabe2000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaaaabe2000-2aaaaac12000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaaaac12000-2aaaaac13000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,1]<stderr>:2aaaaac13000-2aaaaac14000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,1]<stderr>:2aaaaac14000-2aaaaac15000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,1]<stderr>:2aaaaac15000-2aaaaac16000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,1]<stderr>:2aaaaac16000-2aaaaac17000 rw-s 00000000 00:05[1,0]<stderr>:                    /usr/lib64/libdl-2.17.so
[1,0]<stderr>:2aaaab0ee000-2aaaab0ef000 rw-p 00003000 08:02 1060083                    /usr/lib64/libdl-2.17.so
[1,0]<stderr>:2aaaab0ef000-2aaaab0f6000 r-xp 00000000 08:02 1060399                    /usr/lib64/librt-2.17.so
[1,0]<stderr>:2aaaab0f6000-2aaaab2f5000 ---p 00007000 08:02 1060399                    /usr/lib64/librt-2.17.so
[1,0]<stderr>:2aaaab2f5000-2aaaab2f6000 r--p 00006000 08:02 1060399                    /usr/lib64/librt-2.17.so
[1,0]<stderr>:2aaaab2f6000-2aaaab2f7000 rw-p 00007000 08:02 1060399                    /usr/lib64/librt-2.17.so
[1,0]<stderr>:2aaaab2f7000-2aaaab36c000 r-xp 00000000 00:2a 20411532315                /trinity/shared/pkg/devel/cuda/10.0/lib64/libcudart.so.10.0.130
[1,0]<stderr>:2aaaab36c000-2aaaab56c000 ---p 00075000 00:2a 20411532315                /trinity/shared/pkg/devel/cuda/10.0/lib64/libcudart.so.10.0.130
[1,0]<stderr>:2aaaab56c000-2aaaab570000 rw-p 00075000 00:2a 20411532315                /trinity/shared/pkg/devel/cuda/10.0/lib64/libcudart.so.10.0.130
[1,0]<stderr>:2aaaab570000-2aaaab571000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaaab571000-2aaaab656000 r-xp 00000000 00:2a 20427944072                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmpi.so.20.10.3
[1,0]<stderr>:2aaaab656000-2aaaab856000 ---p 000e5000 00:2a 20427944072                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmpi.so.20.10.3
[1,0]<stderr>:2aaaab856000-2aaaab857000 r--p 000e5000 00:2a 20427944072                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmpi.so.20.10.3
[1,0]<stderr>:2aaaab857000-2aaaab867000 rw-p 000e6000 00:2a 20427944072                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmpi.so.20.10.3
[1,0]<stderr>:2aaaab867000-2aaaab879000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaaab879000-2aaaab891000 r-xp 00000000 00:2a 20427944076                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmpi_cxx.so.20.10.0
[1,0]<stderr>:2aaaab891000-2aaaaba90000 ---p 00018000 00:2a 20427944076                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmpi_cxx.so.20.10.0
[1,0]<stderr>:2aaaaba90000-2aaaaba92000 r--p 00017000 00:2a 20427944076                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmpi_cxx.so.20.10.0
[1,0]<stderr>:2aaaaba92000-2aaaaba93000 rw-p 00019000 00:2a 20427944076                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmpi_cxx.so.20.10.0
[1,0]<stderr>:2aaaaba93000-2aaaabaac000 r-xp 00000000 00:2a 9677772441                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_filesystem.so.1.68.0
[1,0]<stderr>:2aaaabaac000-2aaaabcab000 ---p 00019000 00:2a 9677772441                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_filesystem.so.1.68.0
[1,0]<stderr>:2aaaabcab000-2aaaabcac000 r--p 00018000 00:2a 9677772441                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_filesystem.so.1.68.0
[1,0]<stderr>:2aaaabcac000-2aaaabcad000 rw-p 00019000 00:2a 9677772441                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_filesystem.so.1.68.0
[1,0]<stderr>:2aaaabcad000-2aaaabcb1000 r-xp 00000000 00:2a 9677772420                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_system.so.1.68.0
[1,0]<stderr>:2aaaabcb1000-2aaaabeb0000 ---p 00004000 00:2a 9677772420                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_system.so.1.68.0
[1,0]<stderr>:2aaaabeb0000-2aaaabeb1000 r--p 00003000 00:2a 9677772420                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_system.so.1.68.0
[1,0]<stderr>:2aaaabeb1000-2aaaabeb2000 rw-p 00004000 00:2a 9677772420                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_system.so.1.68.0
[1,0]<stderr>:2aaaabeb2000-2aaaabf06000 r-xp 00000000 00:2a 9677772466                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_math_tr1.so.1.68.0
[1,0]<stderr>:2aaaabf06000-2aaaac105000 ---p 00054000 00:2a 9677772466                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_math_tr1.so.1.68.0
[1,0]<stderr>:2aaaac105000-2aaaac106000 r--p 00053000 00:2a 9677772466                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_math_tr1.so.1.68.0
[1,0]<stderr>:2aaaac106000-2aaaac107000 rw-p 00054000 00:2a 9677772466                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost[1,1]<stderr>: 18613                      /dev/nvidiactl
[1,1]<stderr>:2aaaaac17000-2aaaaac18000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,1]<stderr>:2aaaaac18000-2aaaaac20000 rw-s 00f02000 00:05 23595                      /dev/infiniband/uverbs0
[1,1]<stderr>:2aaaaac20000-2aaaaac30000 rw-s 01002000 00:05 23595                      /dev/infiniband/uverbs0
[1,1]<stderr>:2aaaaac30000-2aaaaac31000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaaaac31000-2aaaaac92000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaaaac92000-2aaaaac93000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaaaac93000-2aaaaac94000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,1]<stderr>:2aaaaac94000-2aaaaac95000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,1]<stderr>:2aaaaac95000-2aaaaac96000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,1]<stderr>:2aaaaac96000-2aaaaac97000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,1]<stderr>:2aaaaac97000-2aaaaac98000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,1]<stderr>:2aaaaac98000-2aaaaac99000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,1]<stderr>:2aaaaac99000-2aaaaac9a000 rw-s 00000000 00:05 18613                      /dev/nvidiactl
[1,1]<stderr>:2aaaaac9a000-2aaaaac9b000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaaaaccc000-2aaaaaccd000 r--p 00021000 08:02 1059967                    /usr/lib64/ld-2.17.so
[1,1]<stderr>:2aaaaaccd000-2aaaaacce000 rw-p 00022000 08:02 1059967                    /usr/lib64/ld-2.17.so
[1,1]<stderr>:2aaaaacce000-2aaaaaccf000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaaaaccf000-2aaaaace6000 r-xp 00000000 08:02 1060375                    /usr/lib64/libpthread-2.17.so
[1,1]<stderr>:2aaaaace6000-2aaaaaee5000 ---p 00017000 08:02 1060375                    /usr/lib64/libpthread-2.17.so
[1,1]<stderr>:2aaaaaee5000-2aaaaaee6000 r--p 00016000 08:02 1060375                    /usr/lib64/libpthread-2.17.so
[1,1]<stderr>:2aaaaaee6000-2aaaaaee7000 rw-p 00017000 08:02 1060375                    /usr/lib64/libpthread-2.17.so
[1,1]<stderr>:2aaaaaee7000-2aaaaaeeb000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaaaaeeb000-2aaaaaeed000 r-xp 00000000 08:02 1060083                    /usr/lib64/libdl-2.17.so
[1,1]<stderr>:2aaaaaeed000-2aaaab0ed000 ---p 00002000 08:02 1060083                    /usr/lib64/libdl-2.17.so
[1,1]<stderr>:2aaaab0ed000-2aaaab0ee000 r--p 00002000 08:02 1060083                    /usr/lib64/libdl-2.17.so
[1,1]<stderr>:2aaaab0ee000-2aaaab0ef000 rw-p 00003000 08:02 1060083                    /usr/lib64/libdl-2.17.so
[1,1]<stderr>:2aaaab0ef000-2aaaab0f6000 r-xp 00000000 08:02 1060399                    /usr/lib64/librt-2.17.so
[1,1]<stderr>:2aaaab0f6000-2aaaab2f5000 ---p 00007000 08:02 1060399                    /usr/lib64/librt-2.17.so
[1,1]<stderr>:2aaaab2f5000-2aaaab2f6000 r--p 00006000 08:02 1060399                    /usr/lib64/librt-2.17.so
[1,1]<stderr>:2aaaab2f6000-2aaaab2f7000 rw-p 00007000 08:02 1060399                    /usr/lib64/librt-2.17.so
[1,1]<stderr>:2aaaab2f7000-2aaaab36c000 r-xp 00000000 00:2a 20411532315                /trinity/shared/pkg/devel/cuda/10.0/lib64/libcudart.so.10.0.130
[1,1]<stderr>:2aaaab36c000-2aaaab56c000 ---p 00075000 00:2a 20411532315                /trinity/shared/pkg/devel/cuda/10.0/lib64/libcudart.so.10.0.130
[1,1]<stderr>:2aaaab56c000-2aaaab570000 rw-p 00075000 00:2a 20411532315                /trinity/shared/pkg/devel/cuda/10.0/lib64/libcudart.so.10.0.130
[1,1]<stderr>:2aaaab570000-2aaaab571000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaaab571000-2aaaab656000 r-xp 00000000 00:2a 20427944072                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmpi.so.20.10.3
[1,1]<stderr>:2aaaab656000-2aaaab856000 ---p 000e5000 00:2a 20427944072                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmpi.so.20.10.3
[1,1]<stderr>:2aaaab856000-2aaaab857000 r--p 000e5000 00:2a 20427944072                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmpi.so.20.10.3
[1,1]<stderr>:2aaaab857000-2aaaab867000 rw-p 000e6000 00:2a 20427944072                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmpi.so.20.10.3
[1,1]<stderr>:2aaaab867000-2aaaab879000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaaab879000-2aaaab891000 r-xp 00000000 00:2a 20427944076                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmpi_cxx.so.20.10.0
[1,1]<stderr>:2aaaab891000-2aaaaba90000 ---p 00018000 00:2a 20427944076                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmpi_cxx.so.20.10.0
[1,1]<stderr>:2aaaaba90000-2aaaaba92000 r--p 00017000 00:2a 204279[1,0]<stderr>:_math_tr1.so.1.68.0
[1,0]<stderr>:2aaaac107000-2aaaac10c000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaaac10c000-2aaaac125000 r-xp 00000000 00:2a 10746597907                /trinity/shared/pkg/devel/zlib/1.2.11/lib/libz.so.1.2.11
[1,0]<stderr>:2aaaac125000-2aaaac325000 ---p 00019000 00:2a 10746597907                /trinity/shared/pkg/devel/zlib/1.2.11/lib/libz.so.1.2.11
[1,0]<stderr>:2aaaac325000-2aaaac326000 r--p 00019000 00:2a 10746597907                /trinity/shared/pkg/devel/zlib/1.2.11/lib/libz.so.1.2.11
[1,0]<stderr>:2aaaac326000-2aaaac327000 rw-p 0001a000 00:2a 10746597907                /trinity/shared/pkg/devel/zlib/1.2.11/lib/libz.so.1.2.11
[1,0]<stderr>:2aaaac327000-2aaaac3a7000 r-xp 00000000 00:2a 9677772478                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_program_options.so.1.68.0
[1,0]<stderr>:2aaaac3a7000-2aaaac5a7000 ---p 00080000 00:2a 9677772478                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_program_options.so.1.68.0
[1,0]<stderr>:2aaaac5a7000-2aaaac5aa000 r--p 00080000 00:2a 9677772478                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_program_options.so.1.68.0
[1,0]<stderr>:2aaaac5aa000-2aaaac5ab000 rw-p 00083000 00:2a 9677772478                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_program_options.so.1.68.0
[1,0]<stderr>:2aaaac5ab000-2aaaac5e7000 r-xp 00000000 00:2a 9677772449                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_serialization.so.1.68.0
[1,0]<stderr>:2aaaac5e7000-2aaaac7e6000 ---p 0003c000 00:2a 9677772449                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_serialization.so.1.68.0
[1,0]<stderr>:2aaaac7e6000-2aaaac7e9000 r--p 0003b000 00:2a 9677772449                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_serialization.so.1.68.0
[1,0]<stderr>:2aaaac7e9000-2aaaac7ea000 rw-p 0003e000 00:2a 9677772449                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_serialization.so.1.68.0
[1,0]<stderr>:2aaaac7ea000-2aaaac802000 r-xp 00000000 08:02 1059049                    /usr/lib64/libibverbs.so.1.0.0
[1,0]<stderr>:2aaaac802000-2aaaaca01000 ---p 00018000 08:02 1059049                    /usr/lib64/libibverbs.so.1.0.0
[1,0]<stderr>:2aaaaca01000-2aaaaca02000 r--p 00017000 08:02 1059049                    /usr/lib64/libibverbs.so.1.0.0
[1,0]<stderr>:2aaaaca02000-2aaaaca03000 rw-p 00018000 08:02 1059049                    /usr/lib64/libibverbs.so.1.0.0
[1,0]<stderr>:2aaaaca03000-2aaaacab4000 r-xp 00000000 00:2a 15049318663                /trinity/shared/pkg/devel/c-blosc/1.14.4/gcc/7.3.0/lib/libblosc.so.1.14.4
[1,0]<stderr>:2aaaacab4000-2aaaaccb4000 ---p 000b1000 00:2a 15049318663                /trinity/shared/pkg/devel/c-blosc/1.14.4/gcc/7.3.0/lib/libblosc.so.1.14.4
[1,0]<stderr>:2aaaaccb4000-2aaaaccb5000 r--p 000b1000 00:2a 15049318663                /trinity/shared/pkg/devel/c-blosc/1.14.4/gcc/7.3.0/lib/libblosc.so.1.14.4
[1,0]<stderr>:2aaaaccb5000-2aaaaccb6000 rw-p 000b2000 00:2a 15049318663                /trinity/shared/pkg/devel/c-blosc/1.14.4/gcc/7.3.0/lib/libblosc.so.1.14.4
[1,0]<stderr>:2aaaaccb6000-2aaaacdb7000 r-xp 00000000 08:02 1060116                    /usr/lib64/libm-2.17.so
[1,0]<stderr>:2aaaacdb7000-2aaaacfb6000 ---p 00101000 08:02 1060116                    /usr/lib64/libm-2.17.so
[1,0]<stderr>:2aaaacfb6000-2aaaacfb7000 r--p 00100000 08:02 1060116                    /usr/lib64/libm-2.17.so
[1,0]<stderr>:2aaaacfb7000-2aaaacfb8000 rw-p 00101000 08:02 1060116                    /usr/lib64/libm-2.17.so
[1,0]<stderr>:2aaaacfb8000-2aaaad2b5000 r-xp 00000000 00:2a 20427944122                /trinity/shared/pkg/filelib/hdf5-parallel/1.8.20-cuda100/gcc/7.3.0/openmpi/2.1.2-cuda100/lib/libhdf5.so.10.3.1
[1,0]<stderr>:2aaaad2b5000-2aaaad4b5000 ---p 002fd000 00:2a 20427944122                /trinity/shared/pkg/filelib/hdf5-parallel/1.8.20-cuda100/gcc/7.3.0/openmpi/2.1.2-cuda100/lib/libhdf5.so.10.3.1
[1,0]<stderr>:2aaaad4b5000-2aaaad4ba000 r--p 002fd000 00:2a 20427944122                /trinity/shared/pkg/filelib/hdf5-parallel/1.8.20-cuda100/gcc/7.3.0/openmpi/2.1.2-cuda100/lib/libhdf5.so.10.3.1
[1,0]<stderr>:2aaaad4ba000-2aaaad4c1000 rw-p 00302000 00:2a 20427944122                /trinity/shared/pkg/filelib/hdf5-parallel/1.8.20-cuda100/gcc/7.3.0/openmpi/2.1.2-cuda100/lib/libhdf5.so.10.3.1
[1,0]<stderr>:2aaaad4c1000-2aaaad4c2000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaaad4c2000-2aaaad4f4000 r-xp 00000000 00:2a 1091[1,1]<stderr>:44076                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmpi_cxx.so.20.10.0
[1,1]<stderr>:2aaaaba92000-2aaaaba93000 rw-p 00019000 00:2a 20427944076                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmpi_cxx.so.20.10.0
[1,1]<stderr>:2aaaaba93000-2aaaabaac000 r-xp 00000000 00:2a 9677772441                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_filesystem.so.1.68.0
[1,1]<stderr>:2aaaabaac000-2aaaabcab000 ---p 00019000 00:2a 9677772441                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_filesystem.so.1.68.0
[1,1]<stderr>:2aaaabcab000-2aaaabcac000 r--p 00018000 00:2a 9677772441                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_filesystem.so.1.68.0
[1,1]<stderr>:2aaaabcac000-2aaaabcad000 rw-p 00019000 00:2a 9677772441                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_filesystem.so.1.68.0
[1,1]<stderr>:2aaaabcad000-2aaaabcb1000 r-xp 00000000 00:2a 9677772420                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_system.so.1.68.0
[1,1]<stderr>:2aaaabcb1000-2aaaabeb0000 ---p 00004000 00:2a 9677772420                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_system.so.1.68.0
[1,1]<stderr>:2aaaabeb0000-2aaaabeb1000 r--p 00003000 00:2a 9677772420                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_system.so.1.68.0
[1,1]<stderr>:2aaaabeb1000-2aaaabeb2000 rw-p 00004000 00:2a 9677772420                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_system.so.1.68.0
[1,1]<stderr>:2aaaabeb2000-2aaaabf06000 r-xp 00000000 00:2a 9677772466                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_math_tr1.so.1.68.0
[1,1]<stderr>:2aaaabf06000-2aaaac105000 ---p 00054000 00:2a 9677772466                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_math_tr1.so.1.68.0
[1,1]<stderr>:2aaaac105000-2aaaac106000 r--p 00053000 00:2a 9677772466                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_math_tr1.so.1.68.0
[1,1]<stderr>:2aaaac106000-2aaaac107000 rw-p 00054000 00:2a 9677772466                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_math_tr1.so.1.68.0
[1,1]<stderr>:2aaaac107000-2aaaac10c000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaaac10c000-2aaaac125000 r-xp 00000000 00:2a 10746597907                /trinity/shared/pkg/devel/zlib/1.2.11/lib/libz.so.1.2.11
[1,1]<stderr>:2aaaac125000-2aaaac325000 ---p 00019000 00:2a 10746597907                /trinity/shared/pkg/devel/zlib/1.2.11/lib/libz.so.1.2.11
[1,1]<stderr>:2aaaac325000-2aaaac326000 r--p 00019000 00:2a 10746597907                /trinity/shared/pkg/devel/zlib/1.2.11/lib/libz.so.1.2.11
[1,1]<stderr>:2aaaac326000-2aaaac327000 rw-p 0001a000 00:2a 10746597907                /trinity/shared/pkg/devel/zlib/1.2.11/lib/libz.so.1.2.11
[1,1]<stderr>:2aaaac327000-2aaaac3a7000 r-xp 00000000 00:2a 9677772478                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_program_options.so.1.68.0
[1,1]<stderr>:2aaaac3a7000-2aaaac5a7000 ---p 00080000 00:2a 9677772478                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_program_options.so.1.68.0
[1,1]<stderr>:2aaaac5a7000-2aaaac5aa000 r--p 00080000 00:2a 9677772478                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_program_options.so.1.68.0
[1,1]<stderr>:2aaaac5aa000-2aaaac5ab000 rw-p 00083000 00:2a 9677772478                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_program_options.so.1.68.0
[1,1]<stderr>:2aaaac5ab000-2aaaac5e7000 r-xp 00000000 00:2a 9677772449                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_serialization.so.1.68.0
[1,1]<stderr>:2aaaac5e7000-2aaaac7e6000 ---p 0003c000 00:2a 9677772449                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_serialization.so.1.68.0
[1,1]<stderr>:2aaaac7e6000-2aaaac7e9000 r--p 0003b000 00:2a 9677772449                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_serialization.so.1.68.0
[1,1]<stderr>:2aaaac7e9000-2aaaac7ea000 rw-p 0003e000 00:2a 9677772449                 /trinity/shared/pkg/devel/boost/1.68.0/gcc/7.3.0/lib/libboost_serialization.so.1.68.0
[1,1]<stderr>:2aaaac7ea000-2aaaac802000 r-xp 00000000 08:02 1059049                    /usr/lib64/libibverbs.so.1.0.0
[1,1]<stderr>:2aaaac802000-2aaaaca01000 ---p 000180[1,0]<stderr>:428159                 /trinity/shared/pkg/filelib/libpng/1.6.35/gcc/7.3.0/lib/libpng16.so.16.35.0
[1,0]<stderr>:2aaaad4f4000-2aaaad6f3000 ---p 00032000 00:2a 1091428159                 /trinity/shared/pkg/filelib/libpng/1.6.35/gcc/7.3.0/lib/libpng16.so.16.35.0
[1,0]<stderr>:2aaaad6f3000-2aaaad6f4000 r--p 00031000 00:2a 1091428159                 /trinity/shared/pkg/filelib/libpng/1.6.35/gcc/7.3.0/lib/libpng16.so.16.35.0
[1,0]<stderr>:2aaaad6f4000-2aaaad6f5000 rw-p 00032000 00:2a 1091428159                 /trinity/shared/pkg/filelib/libpng/1.6.35/gcc/7.3.0/lib/libpng16.so.16.35.0
[1,0]<stderr>:2aaaad6f5000-2aaaad795000 r-xp 00000000 08:02 1060158                    /usr/lib64/libfreetype.so.6.10.0
[1,0]<stderr>:2aaaad795000-2aaaad994000 ---p 000a0000 08:02 1060158                    /usr/lib64/libfreetype.so.6.10.0
[1,0]<stderr>:2aaaad994000-2aaaad99a000 r--p 0009f000 08:02 1060158                    /usr/lib64/libfreetype.so.6.10.0
[1,0]<stderr>:2aaaad99a000-2aaaad99b000 rw-p 000a5000 08:02 1060158                    /usr/lib64/libfreetype.so.6.10.0
[1,0]<stderr>:2aaaad99b000-2aaaadb0d000 r-xp 00000000 00:2a 30064775831                /trinity/shared/pkg/compiler/gcc/7.3.0/lib64/libstdc++.so.6.0.24
[1,0]<stderr>:2aaaadb0d000-2aaaadd0d000 ---p 00172000 00:2a 30064775831                /trinity/shared/pkg/compiler/gcc/7.3.0/lib64/libstdc++.so.6.0.24
[1,0]<stderr>:2aaaadd0d000-2aaaadd17000 r--p 00172000 00:2a 30064775831                /trinity/shared/pkg/compiler/gcc/7.3.0/lib64/libstdc++.so.6.0.24
[1,0]<stderr>:2aaaadd17000-2aaaadd19000 rw-p 0017c000 00:2a 30064775831                /trinity/shared/pkg/compiler/gcc/7.3.0/lib64/libstdc++.so.6.0.24
[1,0]<stderr>:2aaaadd19000-2aaaadd1d000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaaadd1d000-2aaaadd4a000 r-xp 00000000 00:2a 30074904971                /trinity/shared/pkg/compiler/gcc/7.3.0/lib64/libgomp.so.1.0.0
[1,0]<stderr>:2aaaadd4a000-2aaaadf49000 ---p 0002d000 00:2a 30074904971                /trinity/shared/pkg/compiler/gcc/7.3.0/lib64/libgomp.so.1.0.0
[1,0]<stderr>:2aaaadf49000-2aaaadf4a000 r--p 0002c000 00:2a 30074904971                /trinity/shared/pkg/compiler/gcc/7.3.0/lib64/libgomp.so.1.0.0
[1,0]<stderr>:2aaaadf4a000-2aaaadf4b000 rw-p 0002d000 00:2a 30074904971                /trinity/shared/pkg/compiler/gcc/7.3.0/lib64/libgomp.so.1.0.0
[1,0]<stderr>:2aaaadf4b000-2aaaadf61000 r-xp 00000000 00:2a 30064775824                /trinity/shared/pkg/compiler/gcc/7.3.0/lib64/libgcc_s.so.1
[1,0]<stderr>:2aaaadf61000-2aaaae160000 ---p 00016000 00:2a 30064775824                /trinity/shared/pkg/compiler/gcc/7.3.0/lib64/libgcc_s.so.1
[1,0]<stderr>:2aaaae160000-2aaaae161000 r--p 00015000 00:2a 30064775824                /trinity/shared/pkg/compiler/gcc/7.3.0/lib64/libgcc_s.so.1
[1,0]<stderr>:2aaaae161000-2aaaae162000 rw-p 00016000 00:2a 30064775824                /trinity/shared/pkg/compiler/gcc/7.3.0/lib64/libgcc_s.so.1
[1,0]<stderr>:2aaaae162000-2aaaae257000 r-xp 00000000 08:02 1059968                    /usr/lib64/libc-2.17.so
[1,0]<stderr>:2aaaae257000-2aaaae258000 r-xp 000f5000 08:02 1059968                    /usr/lib64/libc-2.17.so
[1,0]<stderr>:2aaaae258000-2aaaae25b000 r-xp 000f6000 08:02 1059968                    /usr/lib64/libc-2.17.so
[1,0]<stderr>:2aaaae25b000-2aaaae25c000 r-xp 000f9000 08:02 1059968                    /usr/lib64/libc-2.17.so
[1,0]<stderr>:2aaaae25c000-2aaaae261000 r-xp 000fa000 08:02 1059968                    /usr/lib64/libc-2.17.so
[1,0]<stderr>:2aaaae261000-2aaaae263000 r-xp 000ff000 08:02 1059968                    /usr/lib64/libc-2.17.so
[1,0]<stderr>:2aaaae263000-2aaaae325000 r-xp 00101000 08:02 1059968                    /usr/lib64/libc-2.17.so
[1,0]<stderr>:2aaaae325000-2aaaae524000 ---p 001c3000 08:02 1059968                    /usr/lib64/libc-2.17.so
[1,0]<stderr>:2aaaae524000-2aaaae528000 r--p 001c2000 08:02 1059968                    /usr/lib64/libc-2.17.so
[1,0]<stderr>:2aaaae528000-2aaaae52a000 rw-p 001c6000 08:02 1059968                    /usr/lib64/libc-2.17.so
[1,0]<stderr>:2aaaae52a000-2aaaae52f000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaaae52f000-2aaaae5b4000 r-xp 00000000 00:2a 20427944068                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libopen-rte.so.20.10.4
[1,0]<stderr>:2aaaae5b4000-2aaaae7b4000 ---p 00085000 00:2a 20427944068                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libopen-rte.so.20.10.4
[1,0]<stderr>:2aaaae7b4000-2aaaae7b5000 r--p 00085000 00:2a 20427944068                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3[1,1]<stderr>:00 08:02 1059049                    /usr/lib64/libibverbs.so.1.0.0
[1,1]<stderr>:2aaaaca01000-2aaaaca02000 r--p 00017000 08:02 1059049                    /usr/lib64/libibverbs.so.1.0.0
[1,1]<stderr>:2aaaaca02000-2aaaaca03000 rw-p 00018000 08:02 1059049                    /usr/lib64/libibverbs.so.1.0.0
[1,1]<stderr>:2aaaaca03000-2aaaacab4000 r-xp 00000000 00:2a 15049318663                /trinity/shared/pkg/devel/c-blosc/1.14.4/gcc/7.3.0/lib/libblosc.so.1.14.4
[1,1]<stderr>:2aaaacab4000-2aaaaccb4000 ---p 000b1000 00:2a 15049318663                /trinity/shared/pkg/devel/c-blosc/1.14.4/gcc/7.3.0/lib/libblosc.so.1.14.4
[1,1]<stderr>:2aaaaccb4000-2aaaaccb5000 r--p 000b1000 00:2a 15049318663                /trinity/shared/pkg/devel/c-blosc/1.14.4/gcc/7.3.0/lib/libblosc.so.1.14.4
[1,1]<stderr>:2aaaaccb5000-2aaaaccb6000 rw-p 000b2000 00:2a 15049318663                /trinity/shared/pkg/devel/c-blosc/1.14.4/gcc/7.3.0/lib/libblosc.so.1.14.4
[1,1]<stderr>:2aaaaccb6000-2aaaacdb7000 r-xp 00000000 08:02 1060116                    /usr/lib64/libm-2.17.so
[1,1]<stderr>:2aaaacdb7000-2aaaacfb6000 ---p 00101000 08:02 1060116                    /usr/lib64/libm-2.17.so
[1,1]<stderr>:2aaaacfb6000-2aaaacfb7000 r--p 00100000 08:02 1060116                    /usr/lib64/libm-2.17.so
[1,1]<stderr>:2aaaacfb7000-2aaaacfb8000 rw-p 00101000 08:02 1060116                    /usr/lib64/libm-2.17.so
[1,1]<stderr>:2aaaacfb8000-2aaaad2b5000 r-xp 00000000 00:2a 20427944122                /trinity/shared/pkg/filelib/hdf5-parallel/1.8.20-cuda100/gcc/7.3.0/openmpi/2.1.2-cuda100/lib/libhdf5.so.10.3.1
[1,1]<stderr>:2aaaad2b5000-2aaaad4b5000 ---p 002fd000 00:2a 20427944122                /trinity/shared/pkg/filelib/hdf5-parallel/1.8.20-cuda100/gcc/7.3.0/openmpi/2.1.2-cuda100/lib/libhdf5.so.10.3.1
[1,1]<stderr>:2aaaad4b5000-2aaaad4ba000 r--p 002fd000 00:2a 20427944122                /trinity/shared/pkg/filelib/hdf5-parallel/1.8.20-cuda100/gcc/7.3.0/openmpi/2.1.2-cuda100/lib/libhdf5.so.10.3.1
[1,1]<stderr>:2aaaad4ba000-2aaaad4c1000 rw-p 00302000 00:2a 20427944122                /trinity/shared/pkg/filelib/hdf5-parallel/1.8.20-cuda100/gcc/7.3.0/openmpi/2.1.2-cuda100/lib/libhdf5.so.10.3.1
[1,1]<stderr>:2aaaad4c1000-2aaaad4c2000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaaad4c2000-2aaaad4f4000 r-xp 00000000 00:2a 1091428159                 /trinity/shared/pkg/filelib/libpng/1.6.35/gcc/7.3.0/lib/libpng16.so.16.35.0
[1,1]<stderr>:2aaaad4f4000-2aaaad6f3000 ---p 00032000 00:2a 1091428159                 /trinity/shared/pkg/filelib/libpng/1.6.35/gcc/7.3.0/lib/libpng16.so.16.35.0
[1,1]<stderr>:2aaaad6f3000-2aaaad6f4000 r--p 00031000 00:2a 1091428159                 /trinity/shared/pkg/filelib/libpng/1.6.35/gcc/7.3.0/lib/libpng16.so.16.35.0
[1,1]<stderr>:2aaaad6f4000-2aaaad6f5000 rw-p 00032000 00:2a 1091428159                 /trinity/shared/pkg/filelib/libpng/1.6.35/gcc/7.3.0/lib/libpng16.so.16.35.0
[1,1]<stderr>:2aaaad6f5000-2aaaad795000 r-xp 00000000 08:02 1060158                    /usr/lib64/libfreetype.so.6.10.0
[1,1]<stderr>:2aaaad795000-2aaaad994000 ---p 000a0000 08:02 1060158                    /usr/lib64/libfreetype.so.6.10.0
[1,1]<stderr>:2aaaad994000-2aaaad99a000 r--p 0009f000 08:02 1060158                    /usr/lib64/libfreetype.so.6.10.0
[1,1]<stderr>:2aaaad99a000-2aaaad99b000 rw-p 000a5000 08:02 1060158                    /usr/lib64/libfreetype.so.6.10.0
[1,1]<stderr>:2aaaad99b000-2aaaadb0d000 r-xp 00000000 00:2a 30064775831                /trinity/shared/pkg/compiler/gcc/7.3.0/lib64/libstdc++.so.6.0.24
[1,1]<stderr>:2aaaadb0d000-2aaaadd0d000 ---p 00172000 00:2a 30064775831                /trinity/shared/pkg/compiler/gcc/7.3.0/lib64/libstdc++.so.6.0.24
[1,1]<stderr>:2aaaadd0d000-2aaaadd17000 r--p 00172000 00:2a 30064775831                /trinity/shared/pkg/compiler/gcc/7.3.0/lib64/libstdc++.so.6.0.24
[1,1]<stderr>:2aaaadd17000-2aaaadd19000 rw-p 0017c000 00:2a 30064775831                /trinity/shared/pkg/compiler/gcc/7.3.0/lib64/libstdc++.so.6.0.24
[1,1]<stderr>:2aaaadd19000-2aaaadd1d000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaaadd1d000-2aaaadd4a000 r-xp 00000000 00:2a 30074904971                /trinity/shared/pkg/compiler/gcc/7.3.0/lib64/libgomp.so.1.0.0
[1,1]<stderr>:2aaaadd4a000-2aaaadf49000 ---p 0002d000 00:2a 30074904971                /trinity/shared/pkg/compiler/gcc/7.3.0/lib64/libgomp.so.1.0.0
[1,1]<stderr>:2aaaadf49000-2aaaadf4a000 r--p 0002c000 00:2a 30074904971                /trinity/shared/pkg/compiler/gcc/7.3.0/lib64/libgomp.so.1.0.0
[1,1]<stderr>:2aaaadf4a000-2aaaadf4b000 rw-p 000[1,0]<stderr>:.0/lib/libopen-rte.so.20.10.4
[1,0]<stderr>:2aaaae7b5000-2aaaae7b9000 rw-p 00086000 00:2a 20427944068                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libopen-rte.so.20.10.4
[1,0]<stderr>:2aaaae7b9000-2aaaae7ba000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaaae7ba000-2aaaae8ab000 r-xp 00000000 00:2a 20427940279                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libopen-pal.so.20.10.4
[1,0]<stderr>:2aaaae8ab000-2aaaaeaab000 ---p 000f1000 00:2a 20427940279                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libopen-pal.so.20.10.4
[1,0]<stderr>:2aaaaeaab000-2aaaaeaaf000 r--p 000f1000 00:2a 20427940279                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libopen-pal.so.20.10.4
[1,0]<stderr>:2aaaaeaaf000-2aaaaeab4000 rw-p 000f5000 00:2a 20427940279                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libopen-pal.so.20.10.4
[1,0]<stderr>:2aaaaeab4000-2aaaaeabb000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaaaeabb000-2aaaaeabd000 r-xp 00000000 08:02 1060477                    /usr/lib64/libutil-2.17.so
[1,0]<stderr>:2aaaaeabd000-2aaaaecbc000 ---p 00002000 08:02 1060477                    /usr/lib64/libutil-2.17.so
[1,0]<stderr>:2aaaaecbc000-2aaaaecbd000 r--p 00001000 08:02 1060477                    /usr/lib64/libutil-2.17.so
[1,0]<stderr>:2aaaaecbd000-2aaaaecbe000 rw-p 00002000 08:02 1060477                    /usr/lib64/libutil-2.17.so
[1,0]<stderr>:2aaaaecbe000-2aaaaed22000 r-xp 00000000 08:02 1060339                    /usr/lib64/libnl-route-3.so.200.23.0
[1,0]<stderr>:2aaaaed22000-2aaaaef21000 ---p 00064000 08:02 1060339                    /usr/lib64/libnl-route-3.so.200.23.0
[1,0]<stderr>:2aaaaef21000-2aaaaef24000 r--p 00063000 08:02 1060339                    /usr/lib64/libnl-route-3.so.200.23.0
[1,0]<stderr>:2aaaaef24000-2aaaaef29000 rw-p 00066000 08:02 1060339                    /usr/lib64/libnl-route-3.so.200.23.0
[1,0]<stderr>:2aaaaef29000-2aaaaef2b000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaaaef2b000-2aaaaef49000 r-xp 00000000 08:02 1060329                    /usr/lib64/libnl-3.so.200.23.0
[1,0]<stderr>:2aaaaef49000-2aaaaf149000 ---p 0001e000 08:02 1060329                    /usr/lib64/libnl-3.so.200.23.0
[1,0]<stderr>:2aaaaf149000-2aaaaf14b000 r--p 0001e000 08:02 1060329                    /usr/lib64/libnl-3.so.200.23.0
[1,0]<stderr>:2aaaaf14b000-2aaaaf14c000 rw-p 00020000 08:02 1060329                    /usr/lib64/libnl-3.so.200.23.0
[1,0]<stderr>:2aaaaf14c000-2aaaafece000 r-xp 00000000 00:2a 32251582098                /trinity/shared/pkg/devel/cuda/10.0/lib64/libcuda.so.410.79
[1,0]<stderr>:2aaaafece000-2aaab00ce000 ---p 00d82000 00:2a 32251582098                /trinity/shared/pkg/devel/cuda/10.0/lib64/libcuda.so.410.79
[1,0]<stderr>:2aaab00ce000-2aaab023c000 rw-p 00d82000 00:2a 32251582098                /trinity/shared/pkg/devel/cuda/10.0/lib64/libcuda.so.410.79
[1,0]<stderr>:2aaab023c000-2aaab024c000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaab024c000-2aaab0289000 r-xp 00000000 08:02 1060514                    /usr/lib64/libnvidia-fatbinaryloader.so.410.79
[1,0]<stderr>:2aaab0289000-2aaab0489000 ---p 0003d000 08:02 1060514                    /usr/lib64/libnvidia-fatbinaryloader.so.410.79
[1,0]<stderr>:2aaab0489000-2aaab0494000 rw-p 0003d000 08:02 1060514                    /usr/lib64/libnvidia-fatbinaryloader.so.410.79
[1,0]<stderr>:2aaab0494000-2aaab0499000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaab0499000-2aaab049a000 ---p 00000000 00:00 0 
[1,0]<stderr>:2aaab049a000-2aaab069a000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaab069a000-2aaab069b000 r-xp 00000000 00:2a 21510337711                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_patcher_overwrite.so
[1,0]<stderr>:2aaab069b000-2aaab089b000 ---p 00001000 00:2a 21510337711                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_patcher_overwrite.so
[1,0]<stderr>:2aaab089b000-2aaab089c000 r--p 00001000 00:2a 21510337711                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_patcher_overwrite.so
[1,0]<stderr>:2aaab089c000-2aaab089d000 rw-p 00002000 00:2a 21510337711                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_patcher_overwrite.so
[1,0]<stderr>:2aaab089d000-2aaab08a0000 r-xp 00000000 00:2a 21510337725                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_shmem_mmap.so
[1,0]<stderr>:2aaab08a0000-2aaab0a9f000 ---p 00003000 00:2a 21510337725                /trinity/shared/pkg/mpi/openmpi/[1,1]<stderr>:2d000 00:2a 30074904971                /trinity/shared/pkg/compiler/gcc/7.3.0/lib64/libgomp.so.1.0.0
[1,1]<stderr>:2aaaadf4b000-2aaaadf61000 r-xp 00000000 00:2a 30064775824                /trinity/shared/pkg/compiler/gcc/7.3.0/lib64/libgcc_s.so.1
[1,1]<stderr>:2aaaadf61000-2aaaae160000 ---p 00016000 00:2a 30064775824                /trinity/shared/pkg/compiler/gcc/7.3.0/lib64/libgcc_s.so.1
[1,1]<stderr>:2aaaae160000-2aaaae161000 r--p 00015000 00:2a 30064775824                /trinity/shared/pkg/compiler/gcc/7.3.0/lib64/libgcc_s.so.1
[1,1]<stderr>:2aaaae161000-2aaaae162000 rw-p 00016000 00:2a 30064775824                /trinity/shared/pkg/compiler/gcc/7.3.0/lib64/libgcc_s.so.1
[1,1]<stderr>:2aaaae162000-2aaaae257000 r-xp 00000000 08:02 1059968                    /usr/lib64/libc-2.17.so
[1,1]<stderr>:2aaaae257000-2aaaae258000 r-xp 000f5000 08:02 1059968                    /usr/lib64/libc-2.17.so
[1,1]<stderr>:2aaaae258000-2aaaae25b000 r-xp 000f6000 08:02 1059968                    /usr/lib64/libc-2.17.so
[1,1]<stderr>:2aaaae25b000-2aaaae25c000 r-xp 000f9000 08:02 1059968                    /usr/lib64/libc-2.17.so
[1,1]<stderr>:2aaaae25c000-2aaaae261000 r-xp 000fa000 08:02 1059968                    /usr/lib64/libc-2.17.so
[1,1]<stderr>:2aaaae261000-2aaaae263000 r-xp 000ff000 08:02 1059968                    /usr/lib64/libc-2.17.so
[1,1]<stderr>:2aaaae263000-2aaaae325000 r-xp 00101000 08:02 1059968                    /usr/lib64/libc-2.17.so
[1,1]<stderr>:2aaaae325000-2aaaae524000 ---p 001c3000 08:02 1059968                    /usr/lib64/libc-2.17.so
[1,1]<stderr>:2aaaae524000-2aaaae528000 r--p 001c2000 08:02 1059968                    /usr/lib64/libc-2.17.so
[1,1]<stderr>:2aaaae528000-2aaaae52a000 rw-p 001c6000 08:02 1059968                    /usr/lib64/libc-2.17.so
[1,1]<stderr>:2aaaae52a000-2aaaae52f000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaaae52f000-2aaaae5b4000 r-xp 00000000 00:2a 20427944068                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libopen-rte.so.20.10.4
[1,1]<stderr>:2aaaae5b4000-2aaaae7b4000 ---p 00085000 00:2a 20427944068                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libopen-rte.so.20.10.4
[1,1]<stderr>:2aaaae7b4000-2aaaae7b5000 r--p 00085000 00:2a 20427944068                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libopen-rte.so.20.10.4
[1,1]<stderr>:2aaaae7b5000-2aaaae7b9000 rw-p 00086000 00:2a 20427944068                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libopen-rte.so.20.10.4
[1,1]<stderr>:2aaaae7b9000-2aaaae7ba000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaaae7ba000-2aaaae8ab000 r-xp 00000000 00:2a 20427940279                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libopen-pal.so.20.10.4
[1,1]<stderr>:2aaaae8ab000-2aaaaeaab000 ---p 000f1000 00:2a 20427940279                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libopen-pal.so.20.10.4
[1,1]<stderr>:2aaaaeaab000-2aaaaeaaf000 r--p 000f1000 00:2a 20427940279                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libopen-pal.so.20.10.4
[1,1]<stderr>:2aaaaeaaf000-2aaaaeab4000 rw-p 000f5000 00:2a 20427940279                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libopen-pal.so.20.10.4
[1,1]<stderr>:2aaaaeab4000-2aaaaeabb000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaaaeabb000-2aaaaeabd000 r-xp 00000000 08:02 1060477                    /usr/lib64/libutil-2.17.so
[1,1]<stderr>:2aaaaeabd000-2aaaaecbc000 ---p 00002000 08:02 1060477                    /usr/lib64/libutil-2.17.so
[1,1]<stderr>:2aaaaecbc000-2aaaaecbd000 r--p 00001000 08:02 1060477                    /usr/lib64/libutil-2.17.so
[1,1]<stderr>:2aaaaecbd000-2aaaaecbe000 rw-p 00002000 08:02 1060477                    /usr/lib64/libutil-2.17.so
[1,1]<stderr>:2aaaaecbe000-2aaaaed22000 r-xp 00000000 08:02 1060339                    /usr/lib64/libnl-route-3.so.200.23.0
[1,1]<stderr>:2aaaaed22000-2aaaaef21000 ---p 00064000 08:02 1060339                    /usr/lib64/libnl-route-3.so.200.23.0
[1,1]<stderr>:2aaaaef21000-2aaaaef24000 r--p 00063000 08:02 1060339                    /usr/lib64/libnl-route-3.so.200.23.0
[1,1]<stderr>:2aaaaef24000-2aaaaef29000 rw-p 00066000 08:02 1060339                    /usr/lib64/libnl-route-3.so.200.23.0
[1,1]<stderr>:2aaaaef29000-2aaaaef2b000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaaaef2b000-2aaaaef49000 r-xp 00000000 08:02 1060329                    /usr/lib64/libnl-3.so.200.23.0
[1,1]<stderr>:2aaaaef49000-2aaaaf149000 ---p 0001e000 08:02 1060329                    /usr/lib64/libnl-3.so.200.23.0
[1,1]<stderr>:2aaaa[1,0]<stderr>:2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_shmem_mmap.so
[1,0]<stderr>:2aaab0a9f000-2aaab0aa0000 r--p 00002000 00:2a 21510337725                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_shmem_mmap.so
[1,0]<stderr>:2aaab0aa0000-2aaab0aa1000 rw-p 00003000 00:2a 21510337725                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_shmem_mmap.so
[1,0]<stderr>:2aaab0aa1000-2aaab0aa2000 r-xp 00000000 00:2a 21510337723                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_sec_basic.so
[1,0]<stderr>:2aaab0aa2000-2aaab0ca1000 ---p 00001000 00:2a 21510337723                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_sec_basic.so
[1,0]<stderr>:2aaab0ca1000-2aaab0ca2000 r--p 00000000 00:2a 21510337723                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_sec_basic.so
[1,0]<stderr>:2aaab0ca2000-2aaab0ca3000 rw-p 00001000 00:2a 21510337723                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_sec_basic.so
[1,0]<stderr>:2aaab0ca3000-2aaab0ca7000 r-xp 00000000 00:2a 21510341659                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_ess_pmi.so
[1,0]<stderr>:2aaab0ca7000-2aaab0ea6000 ---p 00004000 00:2a 21510341659                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_ess_pmi.so
[1,0]<stderr>:2aaab0ea6000-2aaab0ea7000 r--p 00003000 00:2a 21510341659                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_ess_pmi.so
[1,0]<stderr>:2aaab0ea7000-2aaab0ea8000 rw-p 00004000 00:2a 21510341659                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_ess_pmi.so
[1,0]<stderr>:2aaab0ea8000-2aaab0f0e000 r-xp 00000000 00:2a 21510337713                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_pmix_pmix112.so
[1,0]<stderr>:2aaab0f0e000-2aaab110e000 ---p 00066000 00:2a 21510337713                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_pmix_pmix112.so
[1,0]<stderr>:2aaab110e000-2aaab110f000 r--p 00066000 00:2a 21510337713                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_pmix_pmix112.so
[1,0]<stderr>:2aaab110f000-2aaab1111000 rw-p 00067000 00:2a 21510337713                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_pmix_pmix112.so
[1,0]<stderr>:2aaab1111000-2aaab1113000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaab1113000-2aaab1114000 ---p 00000000 00:00 0 
[1,0]<stderr>:2aaab1114000-2aaab1314000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaab1314000-2aaab1315000 ---p 00000000 00:00 0 
[1,0]<stderr>:2aaab1315000-2aaab1515000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaab1515000-2aaab1915000 r--s 00000000 08:02 1966099                    /tmp/openmpi-sessions-9413@kepler002_0/6259/pmix_dstor_2439/smseg-410189825-0
[1,0]<stderr>:2aaab1915000-2aaab1d15000 r--s 00000000 08:02 1966100                    /tmp/openmpi-sessions-9413@kepler002_0/6259/pmix_dstor_2439/smdataseg-410189825-0
[1,0]<stderr>:2aaab1d15000-2aaab1d17000 r-xp 00000000 00:2a 21510337695                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_allocator_bucket.so
[1,0]<stderr>:2aaab1d17000-2aaab1f16000 ---p 00002000 00:2a 21510337695                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_allocator_bucket.so
[1,0]<stderr>:2aaab1f16000-2aaab1f17000 r--p 00001000 00:2a 21510337695                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_allocator_bucket.so
[1,0]<stderr>:2aaab1f17000-2aaab1f18000 rw-p 00002000 00:2a 21510337695                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_allocator_bucket.so
[1,0]<stderr>:2aaab1f18000-2aaab1f1a000 r-xp 00000000 00:2a 21510341645                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_errmgr_default_app.so
[1,0]<stderr>:2aaab1f1a000-2aaab2119000 ---p 00002000 00:2a 21510341645                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_errmgr_default_app.so
[1,0]<stderr>:2aaab2119000-2aaab211a000 r--p 00001000 00:2a 21510341645                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_errmgr_default_app.so
[1,0]<stderr>:2aaab211a000-2aaab211b000 rw-p 00002000 00:2a 21510341645                /trinity/shared/pkg/mpi/openmpi/2.1.2-c[1,1]<stderr>:f149000-2aaaaf14b000 r--p 0001e000 08:02 1060329                    /usr/lib64/libnl-3.so.200.23.0
[1,1]<stderr>:2aaaaf14b000-2aaaaf14c000 rw-p 00020000 08:02 1060329                    /usr/lib64/libnl-3.so.200.23.0
[1,1]<stderr>:2aaaaf14c000-2aaaafece000 r-xp 00000000 00:2a 32251582098                /trinity/shared/pkg/devel/cuda/10.0/lib64/libcuda.so.410.79
[1,1]<stderr>:2aaaafece000-2aaab00ce000 ---p 00d82000 00:2a 32251582098                /trinity/shared/pkg/devel/cuda/10.0/lib64/libcuda.so.410.79
[1,1]<stderr>:2aaab00ce000-2aaab023c000 rw-p 00d82000 00:2a 32251582098                /trinity/shared/pkg/devel/cuda/10.0/lib64/libcuda.so.410.79
[1,1]<stderr>:2aaab023c000-2aaab024c000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaab024c000-2aaab0289000 r-xp 00000000 08:02 1060514                    /usr/lib64/libnvidia-fatbinaryloader.so.410.79
[1,1]<stderr>:2aaab0289000-2aaab0489000 ---p 0003d000 08:02 1060514                    /usr/lib64/libnvidia-fatbinaryloader.so.410.79
[1,1]<stderr>:2aaab0489000-2aaab0494000 rw-p 0003d000 08:02 1060514                    /usr/lib64/libnvidia-fatbinaryloader.so.410.79
[1,1]<stderr>:2aaab0494000-2aaab0499000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaab0499000-2aaab049a000 ---p 00000000 00:00 0 
[1,1]<stderr>:2aaab049a000-2aaab069a000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaab069a000-2aaab069b000 r-xp 00000000 00:2a 21510337711                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_patcher_overwrite.so
[1,1]<stderr>:2aaab069b000-2aaab089b000 ---p 00001000 00:2a 21510337711                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_patcher_overwrite.so
[1,1]<stderr>:2aaab089b000-2aaab089c000 r--p 00001000 00:2a 21510337711                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_patcher_overwrite.so
[1,1]<stderr>:2aaab089c000-2aaab089d000 rw-p 00002000 00:2a 21510337711                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_patcher_overwrite.so
[1,1]<stderr>:2aaab089d000-2aaab08a0000 r-xp 00000000 00:2a 21510337725                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_shmem_mmap.so
[1,1]<stderr>:2aaab08a0000-2aaab0a9f000 ---p 00003000 00:2a 21510337725                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_shmem_mmap.so
[1,1]<stderr>:2aaab0a9f000-2aaab0aa0000 r--p 00002000 00:2a 21510337725                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_shmem_mmap.so
[1,1]<stderr>:2aaab0aa0000-2aaab0aa1000 rw-p 00003000 00:2a 21510337725                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_shmem_mmap.so
[1,1]<stderr>:2aaab0aa1000-2aaab0aa2000 r-xp 00000000 00:2a 21510337723                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_sec_basic.so
[1,1]<stderr>:2aaab0aa2000-2aaab0ca1000 ---p 00001000 00:2a 21510337723                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_sec_basic.so
[1,1]<stderr>:2aaab0ca1000-2aaab0ca2000 r--p 00000000 00:2a 21510337723                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_sec_basic.so
[1,1]<stderr>:2aaab0ca2000-2aaab0ca3000 rw-p 00001000 00:2a 21510337723                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_sec_basic.so
[1,1]<stderr>:2aaab0ca3000-2aaab0ca7000 r-xp 00000000 00:2a 21510341659                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_ess_pmi.so
[1,1]<stderr>:2aaab0ca7000-2aaab0ea6000 ---p 00004000 00:2a 21510341659                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_ess_pmi.so
[1,1]<stderr>:2aaab0ea6000-2aaab0ea7000 r--p 00003000 00:2a 21510341659                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_ess_pmi.so
[1,1]<stderr>:2aaab0ea7000-2aaab0ea8000 rw-p 00004000 00:2a 21510341659                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_ess_pmi.so
[1,1]<stderr>:2aaab0ea8000-2aaab0f0e000 r-xp 00000000 00:2a 21510337713                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_pmix_pmix112.so
[1,1]<stderr>:2aaab0f0e000-2aaab110e000 ---p 00066000 00:2a 21510337713                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_pmix_pmix112.so
[1,1]<stderr>:2aaab110e000-2aaab110f000 r--p 00066000 00:2a 21510337713   [1,0]<stderr>:uda100/gcc/7.3.0/lib/openmpi/mca_errmgr_default_app.so
[1,0]<stderr>:2aaab211b000-2aaab211e000 r-xp 00000000 00:2a 21510341665                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_grpcomm_direct.so
[1,0]<stderr>:2aaab211e000-2aaab231d000 ---p 00003000 00:2a 21510341665                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_grpcomm_direct.so
[1,0]<stderr>:2aaab231d000-2aaab231e000 r--p 00002000 00:2a 21510341665                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_grpcomm_direct.so
[1,0]<stderr>:2aaab231e000-2aaab231f000 rw-p 00003000 00:2a 21510341665                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_grpcomm_direct.so
[1,0]<stderr>:2aaab2320000-2aaab2324000 r-xp 00000000 00:2a 21510341909                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_routed_radix.so
[1,0]<stderr>:2aaab2324000-2aaab2523000 ---p 00004000 00:2a 21510341909                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_routed_radix.so
[1,0]<stderr>:2aaab2523000-2aaab2524000 r--p 00003000 00:2a 21510341909                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_routed_radix.so
[1,0]<stderr>:2aaab2524000-2aaab2525000 rw-p 00004000 00:2a 21510341909                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_routed_radix.so
[1,0]<stderr>:2aaab2525000-2aaab2526000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaab2526000-2aaab2587000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaab2587000-2aaab2589000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaab2589000-2aaab25ea000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaab25ea000-2aaab25eb000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaab2600000-2aaab2720000 rw-s 01502000 00:05 23595                      /dev/infiniband/uverbs0
[1,0]<stderr>:2aaab2729000-2aaab272b000 r-xp 00000000 00:2a 21510341931                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_state_app.so
[1,0]<stderr>:2aaab272b000-2aaab292a000 ---p 00002000 00:2a 21510341931                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_state_app.so
[1,0]<stderr>:2aaab292a000-2aaab292b000 r--p 00001000 00:2a 21510341931                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_state_app.so
[1,0]<stderr>:2aaab292b000-2aaab292c000 rw-p 00002000 00:2a 21510341931                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_state_app.so
[1,0]<stderr>:2aaab292c000-2aaab293f000 r-xp 00000000 00:2a 21510341681                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_oob_tcp.so
[1,0]<stderr>:2aaab293f000-2aaab2b3f000 ---p 00013000 00:2a 21510341681                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_oob_tcp.so
[1,0]<stderr>:2aaab2b3f000-2aaab2b40000 r--p 00013000 00:2a 21510341681                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_oob_tcp.so
[1,0]<stderr>:2aaab2b40000-2aaab2b41000 rw-p 00014000 00:2a 21510341681                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_oob_tcp.so
[1,0]<stderr>:2aaab2b41000-2aaab2b52000 r-xp 00000000 00:2a 21510341683                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_oob_ud.so
[1,0]<stderr>:2aaab2b52000-2aaab2d51000 ---p 00011000 00:2a 21510341683                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_oob_ud.so
[1,0]<stderr>:2aaab2d51000-2aaab2d52000 r--p 00010000 00:2a 21510341683                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_oob_ud.so
[1,0]<stderr>:2aaab2d52000-2aaab2d53000 rw-p 00011000 00:2a 21510341683                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_oob_ud.so
[1,0]<stderr>:2aaab2d53000-2aaab2d57000 r-xp 00000000 00:2a 20427940287                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmca_common_verbs.so.20.10.1
[1,0]<stderr>:2aaab2d57000-2aaab2f56000 ---p 00004000 00:2a 20427940287                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmca_common_verbs.so.20.10.1
[1,0]<stderr>:2aaab2f56000-2aaab2f57000 r--p 00003000 00:2a 20427940287                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmca_common_verbs.so.20.10.1
[1,0]<stderr>:2aaab2f57000-2aaab2f58000 rw-p 0000[1,1]<stderr>:             /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_pmix_pmix112.so
[1,1]<stderr>:2aaab110f000-2aaab1111000 rw-p 00067000 00:2a 21510337713                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_pmix_pmix112.so
[1,1]<stderr>:2aaab1111000-2aaab1113000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaab1113000-2aaab1114000 ---p 00000000 00:00 0 
[1,1]<stderr>:2aaab1114000-2aaab1314000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaab1314000-2aaab1315000 ---p 00000000 00:00 0 
[1,1]<stderr>:2aaab1315000-2aaab1515000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaab1515000-2aaab1915000 r--s 00000000 08:02 1966099                    /tmp/openmpi-sessions-9413@kepler002_0/6259/pmix_dstor_2439/smseg-410189825-0
[1,1]<stderr>:2aaab1915000-2aaab1d15000 r--s 00000000 08:02 1966100                    /tmp/openmpi-sessions-9413@kepler002_0/6259/pmix_dstor_2439/smdataseg-410189825-0
[1,1]<stderr>:2aaab1d15000-2aaab1d17000 r-xp 00000000 00:2a 21510337695                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_allocator_bucket.so
[1,1]<stderr>:2aaab1d17000-2aaab1f16000 ---p 00002000 00:2a 21510337695                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_allocator_bucket.so
[1,1]<stderr>:2aaab1f16000-2aaab1f17000 r--p 00001000 00:2a 21510337695                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_allocator_bucket.so
[1,1]<stderr>:2aaab1f17000-2aaab1f18000 rw-p 00002000 00:2a 21510337695                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_allocator_bucket.so
[1,1]<stderr>:2aaab1f18000-2aaab1f1a000 r-xp 00000000 00:2a 21510341645                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_errmgr_default_app.so
[1,1]<stderr>:2aaab1f1a000-2aaab2119000 ---p 00002000 00:2a 21510341645                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_errmgr_default_app.so
[1,1]<stderr>:2aaab2119000-2aaab211a000 r--p 00001000 00:2a 21510341645                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_errmgr_default_app.so
[1,1]<stderr>:2aaab211a000-2aaab211b000 rw-p 00002000 00:2a 21510341645                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_errmgr_default_app.so
[1,1]<stderr>:2aaab211b000-2aaab211e000 r-xp 00000000 00:2a 21510341665                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_grpcomm_direct.so
[1,1]<stderr>:2aaab211e000-2aaab231d000 ---p 00003000 00:2a 21510341665                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_grpcomm_direct.so
[1,1]<stderr>:2aaab231d000-2aaab231e000 r--p 00002000 00:2a 21510341665                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_grpcomm_direct.so
[1,1]<stderr>:2aaab231e000-2aaab231f000 rw-p 00003000 00:2a 21510341665                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_grpcomm_direct.so
[1,1]<stderr>:2aaab2320000-2aaab2324000 r-xp 00000000 00:2a 21510341909                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_routed_radix.so
[1,1]<stderr>:2aaab2324000-2aaab2523000 ---p 00004000 00:2a 21510341909                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_routed_radix.so
[1,1]<stderr>:2aaab2523000-2aaab2524000 r--p 00003000 00:2a 21510341909                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_routed_radix.so
[1,1]<stderr>:2aaab2524000-2aaab2525000 rw-p 00004000 00:2a 21510341909                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_routed_radix.so
[1,1]<stderr>:2aaab2525000-2aaab2526000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaab2526000-2aaab2587000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaab2587000-2aaab2589000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaab2589000-2aaab25ea000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaab25ea000-2aaab25eb000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaab2600000-2aaab2720000 rw-s 01502000 00:05 23595                      /dev/infiniband/uverbs0
[1,1]<stderr>:2aaab2729000-2aaab272b000 r-xp 00000000 00:2a 21510341931                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_state_app.so
[1,1]<stderr>:2aaab272b000-2aaab292a000 ---p 00002000 00:2a 21510341931                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_state_app.so
[1,1]<stderr>:2[1,0]<stderr>:4000 00:2a 20427940287                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmca_common_verbs.so.20.10.1
[1,0]<stderr>:2aaab2f58000-2aaab2f64000 r-xp 00000000 00:2a 21510341685                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_oob_usock.so
[1,0]<stderr>:2aaab2f64000-2aaab3164000 ---p 0000c000 00:2a 21510341685                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_oob_usock.so
[1,0]<stderr>:2aaab3164000-2aaab3165000 r--p 0000c000 00:2a 21510341685                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_oob_usock.so
[1,0]<stderr>:2aaab3165000-2aaab3166000 rw-p 0000d000 00:2a 21510341685                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_oob_usock.so
[1,0]<stderr>:2aaab3166000-2aaab316a000 r-xp 00000000 08:02 1059129                    /usr/lib64/librxe-rdmav2.so
[1,0]<stderr>:2aaab316a000-2aaab3369000 ---p 00004000 08:02 1059129                    /usr/lib64/librxe-rdmav2.so
[1,0]<stderr>:2aaab3369000-2aaab336a000 r--p 00003000 08:02 1059129                    /usr/lib64/librxe-rdmav2.so
[1,0]<stderr>:2aaab336a000-2aaab336b000 rw-p 00004000 08:02 1059129                    /usr/lib64/librxe-rdmav2.so
[1,0]<stderr>:2aaab336b000-2aaab338d000 r-xp 00000000 08:02 1059051                    /usr/lib64/libmlx4-rdmav2.so
[1,0]<stderr>:2aaab338d000-2aaab358d000 ---p 00022000 08:02 1059051                    /usr/lib64/libmlx4-rdmav2.so
[1,0]<stderr>:2aaab358d000-2aaab358e000 r--p 00022000 08:02 1059051                    /usr/lib64/libmlx4-rdmav2.so
[1,0]<stderr>:2aaab358e000-2aaab3590000 rw-p 00023000 08:02 1059051                    /usr/lib64/libmlx4-rdmav2.so
[1,0]<stderr>:2aaab3590000-2aaab35e9000 r-xp 00000000 08:02 1059055                    /usr/lib64/libmlx5.so.1.0.0
[1,0]<stderr>:2aaab35e9000-2aaab37e8000 ---p 00059000 08:02 1059055                    /usr/lib64/libmlx5.so.1.0.0
[1,0]<stderr>:2aaab37e8000-2aaab37e9000 r--p 00058000 08:02 1059055                    /usr/lib64/libmlx5.so.1.0.0
[1,0]<stderr>:2aaab37e9000-2aaab37eb000 rw-p 00059000 08:02 1059055                    /usr/lib64/libmlx5.so.1.0.0
[1,0]<stderr>:2aaab37eb000-2aaab37f5000 r-xp 00000000 08:02 1060266                    /usr/lib64/libnuma.so.1
[1,0]<stderr>:2aaab37f5000-2aaab39f5000 ---p 0000a000 08:02 1060266                    /usr/lib64/libnuma.so.1
[1,0]<stderr>:2aaab39f5000-2aaab39f6000 r--p 0000a000 08:02 1060266                    /usr/lib64/libnuma.so.1
[1,0]<stderr>:2aaab39f6000-2aaab39f7000 rw-p 0000b000 08:02 1060266                    /usr/lib64/libnuma.so.1
[1,0]<stderr>:2aaab3a00000-2aaab3c00000 rw-s 01502000 00:05 23595                      /dev/infiniband/uverbs0
[1,0]<stderr>:2aaab3c00000-2aaab3e00000 rw-s 01502000 00:05 23595                      /dev/infiniband/uverbs0
[1,0]<stderr>:2aaab3e00000-2aaab3e01000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaab3e01000-2aaab3f02000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaab3f02000-2aaab3f04000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaab3f04000-2aaab3f65000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaab3f65000-2aaab3f67000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaab3f67000-2aaab3fc8000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaab3fc8000-2aaab3fc9000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaab4000000-2aaab4021000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaab4021000-2aaab8000000 ---p 00000000 00:00 0 
[1,0]<stderr>:2aaab8000000-2aaab8021000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaab8021000-2aaabc000000 ---p 00000000 00:00 0 
[1,0]<stderr>:2aaabc000000-2aaabc001000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaabc001000-2aaabc401000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaabc401000-2aaabc402000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaabc402000-2aaabc405000 r-xp 00000000 00:2a 21510341905                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rml_oob.so
[1,0]<stderr>:2aaabc405000-2aaabc605000 ---p 00003000 00:2a 21510341905                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rml_oob.so
[1,0]<stderr>:2aaabc605000-2aaabc606000 r--p 00003000 00:2a 21510341905                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rml_oob.so
[1,0]<stderr>:2aaabc606000-2aaabc607000 rw-p 00004000 00:2a 21510341905                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rml_oob.so
[1,0]<stderr>:2aaabc607000-2aaabc609000 r-xp 00000000 00:2a 21510337693                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_allocator_basic.so
[1,0]<stderr>:2aaabc609000-2aaabc809000 ---p 00002000 00:2a 21510337693      [1,1]<stderr>:aaab292a000-2aaab292b000 r--p 00001000 00:2a 21510341931                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_state_app.so
[1,1]<stderr>:2aaab292b000-2aaab292c000 rw-p 00002000 00:2a 21510341931                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_state_app.so
[1,1]<stderr>:2aaab292c000-2aaab293f000 r-xp 00000000 00:2a 21510341681                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_oob_tcp.so
[1,1]<stderr>:2aaab293f000-2aaab2b3f000 ---p 00013000 00:2a 21510341681                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_oob_tcp.so
[1,1]<stderr>:2aaab2b3f000-2aaab2b40000 r--p 00013000 00:2a 21510341681                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_oob_tcp.so
[1,1]<stderr>:2aaab2b40000-2aaab2b41000 rw-p 00014000 00:2a 21510341681                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_oob_tcp.so
[1,1]<stderr>:2aaab2b41000-2aaab2b52000 r-xp 00000000 00:2a 21510341683                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_oob_ud.so
[1,1]<stderr>:2aaab2b52000-2aaab2d51000 ---p 00011000 00:2a 21510341683                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_oob_ud.so
[1,1]<stderr>:2aaab2d51000-2aaab2d52000 r--p 00010000 00:2a 21510341683                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_oob_ud.so
[1,1]<stderr>:2aaab2d52000-2aaab2d53000 rw-p 00011000 00:2a 21510341683                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_oob_ud.so
[1,1]<stderr>:2aaab2d53000-2aaab2d57000 r-xp 00000000 00:2a 20427940287                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmca_common_verbs.so.20.10.1
[1,1]<stderr>:2aaab2d57000-2aaab2f56000 ---p 00004000 00:2a 20427940287                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmca_common_verbs.so.20.10.1
[1,1]<stderr>:2aaab2f56000-2aaab2f57000 r--p 00003000 00:2a 20427940287                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmca_common_verbs.so.20.10.1
[1,1]<stderr>:2aaab2f57000-2aaab2f58000 rw-p 00004000 00:2a 20427940287                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmca_common_verbs.so.20.10.1
[1,1]<stderr>:2aaab2f58000-2aaab2f64000 r-xp 00000000 00:2a 21510341685                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_oob_usock.so
[1,1]<stderr>:2aaab2f64000-2aaab3164000 ---p 0000c000 00:2a 21510341685                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_oob_usock.so
[1,1]<stderr>:2aaab3164000-2aaab3165000 r--p 0000c000 00:2a 21510341685                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_oob_usock.so
[1,1]<stderr>:2aaab3165000-2aaab3166000 rw-p 0000d000 00:2a 21510341685                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_oob_usock.so
[1,1]<stderr>:2aaab3166000-2aaab316a000 r-xp 00000000 08:02 1059129                    /usr/lib64/librxe-rdmav2.so
[1,1]<stderr>:2aaab316a000-2aaab3369000 ---p 00004000 08:02 1059129                    /usr/lib64/librxe-rdmav2.so
[1,1]<stderr>:2aaab3369000-2aaab336a000 r--p 00003000 08:02 1059129                    /usr/lib64/librxe-rdmav2.so
[1,1]<stderr>:2aaab336a000-2aaab336b000 rw-p 00004000 08:02 1059129                    /usr/lib64/librxe-rdmav2.so
[1,1]<stderr>:2aaab336b000-2aaab338d000 r-xp 00000000 08:02 1059051                    /usr/lib64/libmlx4-rdmav2.so
[1,1]<stderr>:2aaab338d000-2aaab358d000 ---p 00022000 08:02 1059051                    /usr/lib64/libmlx4-rdmav2.so
[1,1]<stderr>:2aaab358d000-2aaab358e000 r--p 00022000 08:02 1059051                    /usr/lib64/libmlx4-rdmav2.so
[1,1]<stderr>:2aaab358e000-2aaab3590000 rw-p 00023000 08:02 1059051                    /usr/lib64/libmlx4-rdmav2.so
[1,1]<stderr>:2aaab3590000-2aaab35e9000 r-xp 00000000 08:02 1059055                    /usr/lib64/libmlx5.so.1.0.0
[1,1]<stderr>:2aaab35e9000-2aaab37e8000 ---p 00059000 08:02 1059055                    /usr/lib64/libmlx5.so.1.0.0
[1,1]<stderr>:2aaab37e8000-2aaab37e9000 r--p 00058000 08:02 1059055                    /usr/lib64/libmlx5.so.1.0.0
[1,1]<stderr>:2aaab37e9000-2aaab37eb000 rw-p 00059000 08:02 1059055                    /usr/lib64/libmlx5.so.1.0.0
[1,1]<stderr>:2aaab37eb000-2aaab37f5000 r-xp 00[1,0]<stderr>:          /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_allocator_basic.so
[1,0]<stderr>:2aaabc809000-2aaabc80a000 r--p 00002000 00:2a 21510337693                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_allocator_basic.so
[1,0]<stderr>:2aaabc80a000-2aaabc80b000 rw-p 00003000 00:2a 21510337693                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_allocator_basic.so
[1,0]<stderr>:2aaabc80b000-2aaabc80e000 r-xp 00000000 00:2a 21510337717                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rcache_grdma.so
[1,0]<stderr>:2aaabc80e000-2aaabca0e000 ---p 00003000 00:2a 21510337717                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rcache_grdma.so
[1,0]<stderr>:2aaabca0e000-2aaabca0f000 r--p 00003000 00:2a 21510337717                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rcache_grdma.so
[1,0]<stderr>:2aaabca0f000-2aaabca10000 rw-p 00004000 00:2a 21510337717                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rcache_grdma.so
[1,0]<stderr>:2aaabca10000-2aaabca17000 r-xp 00000000 00:2a 20427944064                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmca_common_cuda.so.20.10.0
[1,0]<stderr>:2aaabca17000-2aaabcc17000 ---p 00007000 00:2a 20427944064                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmca_common_cuda.so.20.10.0
[1,0]<stderr>:2aaabcc17000-2aaabcc18000 r--p 00007000 00:2a 20427944064                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmca_common_cuda.so.20.10.0
[1,0]<stderr>:2aaabcc18000-2aaabcc19000 rw-p 00008000 00:2a 20427944064                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmca_common_cuda.so.20.10.0
[1,0]<stderr>:2aaabcc19000-2aaabcc1d000 r-xp 00000000 00:2a 21510337719                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rcache_rgpusm.so
[1,0]<stderr>:2aaabcc1d000-2aaabce1c000 ---p 00004000 00:2a 21510337719                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rcache_rgpusm.so
[1,0]<stderr>:2aaabce1c000-2aaabce1d000 r--p 00003000 00:2a 21510337719                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rcache_rgpusm.so
[1,0]<stderr>:2aaabce1d000-2aaabce1e000 rw-p 00004000 00:2a 21510337719                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rcache_rgpusm.so
[1,0]<stderr>:2aaabce1e000-2aaabce20000 r-xp 00000000 00:2a 21510337721                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rcache_gpusm.so
[1,0]<stderr>:2aaabce20000-2aaabd01f000 ---p 00002000 00:2a 21510337721                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rcache_gpusm.so
[1,0]<stderr>:2aaabd01f000-2aaabd020000 r--p 00001000 00:2a 21510337721                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rcache_gpusm.so
[1,0]<stderr>:2aaabd020000-2aaabd021000 rw-p 00002000 00:2a 21510337721                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rcache_gpusm.so
[1,0]<stderr>:2aaabd021000-2aaabd024000 r-xp 00000000 00:2a 21510337709                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_mpool_hugepage.so
[1,0]<stderr>:2aaabd024000-2aaabd223000 ---p 00003000 00:2a 21510337709                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_mpool_hugepage.so
[1,0]<stderr>:2aaabd223000-2aaabd224000 r--p 00002000 00:2a 21510337709                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_mpool_hugepage.so
[1,0]<stderr>:2aaabd224000-2aaabd225000 rw-p 00003000 00:2a 21510337709                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_mpool_hugepage.so
[1,0]<stderr>:2aaabd225000-2aaabd229000 r-xp 00000000 00:2a 21510341939                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_bml_r2.so
[1,0]<stderr>:2aaabd229000-2aaabd428000 ---p 00004000 00:2a 21510341939                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_bml_r2.so
[1,0]<stderr>:2aaabd428000-2aaabd429000 r--p 00003000 00:2a 21510341939                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda10[1,1]<stderr>:000000 08:02 1060266                    /usr/lib64/libnuma.so.1
[1,1]<stderr>:2aaab37f5000-2aaab39f5000 ---p 0000a000 08:02 1060266                    /usr/lib64/libnuma.so.1
[1,1]<stderr>:2aaab39f5000-2aaab39f6000 r--p 0000a000 08:02 1060266                    /usr/lib64/libnuma.so.1
[1,1]<stderr>:2aaab39f6000-2aaab39f7000 rw-p 0000b000 08:02 1060266                    /usr/lib64/libnuma.so.1
[1,1]<stderr>:2aaab3a00000-2aaab3c00000 rw-s 01502000 00:05 23595                      /dev/infiniband/uverbs0
[1,1]<stderr>:2aaab3c00000-2aaab3e00000 rw-s 01502000 00:05 23595                      /dev/infiniband/uverbs0
[1,1]<stderr>:2aaab3e00000-2aaab3e01000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaab3e01000-2aaab3f02000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaab3f02000-2aaab3f04000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaab3f04000-2aaab3f65000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaab3f65000-2aaab3f67000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaab3f67000-2aaab3fc8000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaab3fc8000-2aaab3fc9000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaab4000000-2aaab4021000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaab4021000-2aaab8000000 ---p 00000000 00:00 0 
[1,1]<stderr>:2aaab8000000-2aaab8021000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaab8021000-2aaabc000000 ---p 00000000 00:00 0 
[1,1]<stderr>:2aaabc000000-2aaabc001000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaabc001000-2aaabc401000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaabc401000-2aaabc402000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaabc402000-2aaabc405000 r-xp 00000000 00:2a 21510341905                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rml_oob.so
[1,1]<stderr>:2aaabc405000-2aaabc605000 ---p 00003000 00:2a 21510341905                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rml_oob.so
[1,1]<stderr>:2aaabc605000-2aaabc606000 r--p 00003000 00:2a 21510341905                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rml_oob.so
[1,1]<stderr>:2aaabc606000-2aaabc607000 rw-p 00004000 00:2a 21510341905                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rml_oob.so
[1,1]<stderr>:2aaabc607000-2aaabc609000 r-xp 00000000 00:2a 21510337693                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_allocator_basic.so
[1,1]<stderr>:2aaabc609000-2aaabc809000 ---p 00002000 00:2a 21510337693                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_allocator_basic.so
[1,1]<stderr>:2aaabc809000-2aaabc80a000 r--p 00002000 00:2a 21510337693                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_allocator_basic.so
[1,1]<stderr>:2aaabc80a000-2aaabc80b000 rw-p 00003000 00:2a 21510337693                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_allocator_basic.so
[1,1]<stderr>:2aaabc80b000-2aaabc80e000 r-xp 00000000 00:2a 21510337717                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rcache_grdma.so
[1,1]<stderr>:2aaabc80e000-2aaabca0e000 ---p 00003000 00:2a 21510337717                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rcache_grdma.so
[1,1]<stderr>:2aaabca0e000-2aaabca0f000 r--p 00003000 00:2a 21510337717                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rcache_grdma.so
[1,1]<stderr>:2aaabca0f000-2aaabca10000 rw-p 00004000 00:2a 21510337717                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rcache_grdma.so
[1,1]<stderr>:2aaabca10000-2aaabca17000 r-xp 00000000 00:2a 20427944064                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmca_common_cuda.so.20.10.0
[1,1]<stderr>:2aaabca17000-2aaabcc17000 ---p 00007000 00:2a 20427944064                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmca_common_cuda.so.20.10.0
[1,1]<stderr>:2aaabcc17000-2aaabcc18000 r--p 00007000 00:2a 20427944064                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmca_common_cuda.so.20.10.0
[1,1]<stderr>:2aaabcc18000-2aaabcc19000 rw-p 00008000 00:2a 20427944064                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmca_common_cuda.so.20.10.0
[1,1]<stderr>:2aaabcc19000-2aaabcc1d000 r-xp 00000000 00:2a 21510337719                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rcache_rgpusm.so
[1,1]<stderr>:2aaabcc1d000-2aaabce1c000 ---p 00004000 00:2a 21510337719                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/[1,0]<stderr>:0/gcc/7.3.0/lib/openmpi/mca_bml_r2.so
[1,0]<stderr>:2aaabd429000-2aaabd42a000 rw-p 00004000 00:2a 21510341939                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_bml_r2.so
[1,0]<stderr>:2aaabd42a000-2aaabd42d000 r-xp 00000000 00:2a 21510337697                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_btl_self.so
[1,0]<stderr>:2aaabd42d000-2aaabd62c000 ---p 00003000 00:2a 21510337697                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_btl_self.so
[1,0]<stderr>:2aaabd62c000-2aaabd62d000 r--p 00002000 00:2a 21510337697                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_btl_self.so
[1,0]<stderr>:2aaabd62d000-2aaabd62e000 rw-p 00003000 00:2a 21510337697                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_btl_self.so
[1,0]<stderr>:2aaabd62e000-2aaabd636000 r-xp 00000000 00:2a 21510337699                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_btl_sm.so
[1,0]<stderr>:2aaabd636000-2aaabd835000 ---p 00008000 00:2a 21510337699                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_btl_sm.so
[1,0]<stderr>:2aaabd835000-2aaabd836000 r--p 00007000 00:2a 21510337699                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_btl_sm.so
[1,0]<stderr>:2aaabd836000-2aaabd837000 rw-p 00008000 00:2a 21510337699                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_btl_sm.so
[1,0]<stderr>:2aaabd837000-2aaabd839000 r-xp 00000000 00:2a 20427940283                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmca_common_sm.so.20.10.1
[1,0]<stderr>:2aaabd839000-2aaabda38000 ---p 00002000 00:2a 20427940283                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmca_common_sm.so.20.10.1
[1,0]<stderr>:2aaabda38000-2aaabda39000 r--p 00001000 00:2a 20427940283                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmca_common_sm.so.20.10.1
[1,0]<stderr>:2aaabda39000-2aaabda3a000 rw-p 00002000 00:2a 20427940283                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmca_common_sm.so.20.10.1
[1,0]<stderr>:2aaabda3a000-2aaabda71000 r-xp 00000000 00:2a 21510337703                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_btl_openib.so
[1,0]<stderr>:2aaabda71000-2aaabdc70000 ---p 00037000 00:2a 21510337703                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_btl_openib.so
[1,0]<stderr>:2aaabdc70000-2aaabdc71000 r--p 00036000 00:2a 21510337703                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_btl_openib.so
[1,0]<stderr>:2aaabdc71000-2aaabdc73000 rw-p 00037000 00:2a 21510337703                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_btl_openib.so
[1,0]<stderr>:2aaabdc73000-2aaabdc74000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaabdc74000-2aaabdc8e000 r-xp 00000000 08:02 1060079                    /usr/lib64/librdmacm.so.1.0.0
[1,0]<stderr>:2aaabdc8e000-2aaabde8d000 ---p 0001a000 08:02 1060079                    /usr/lib64/librdmacm.so.1.0.0
[1,0]<stderr>:2aaabde8d000-2aaabde8e000 r--p 00019000 08:02 1060079                    /usr/lib64/librdmacm.so.1.0.0
[1,0]<stderr>:2aaabde8e000-2aaabde8f000 rw-p 0001a000 08:02 1060079                    /usr/lib64/librdmacm.so.1.0.0
[1,0]<stderr>:2aaabde8f000-2aaabde90000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaabde90000-2aaabde93000 r-xp 00000000 00:2a 21510341941                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_sync.so
[1,0]<stderr>:2aaabde93000-2aaabe092000 ---p 00003000 00:2a 21510341941                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_sync.so
[1,0]<stderr>:2aaabe092000-2aaabe093000 r--p 00002000 00:2a 21510341941                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_sync.so
[1,0]<stderr>:2aaabe093000-2aaabe094000 rw-p 00003000 00:2a 21510341941                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_sync.so
[1,0]<stderr>:2aaabe09a000-2aaabe0c1000 r-xp 00000000 00:2a 21510345247                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_pml_ob1.so
[1,0]<stderr>:2aaabe0c1000-2aaabe2c0000 ---p 00027000 00:2a 21510345247                /tri[1,1]<stderr>:7.3.0/lib/openmpi/mca_rcache_rgpusm.so
[1,1]<stderr>:2aaabce1c000-2aaabce1d000 r--p 00003000 00:2a 21510337719                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rcache_rgpusm.so
[1,1]<stderr>:2aaabce1d000-2aaabce1e000 rw-p 00004000 00:2a 21510337719                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rcache_rgpusm.so
[1,1]<stderr>:2aaabce1e000-2aaabce20000 r-xp 00000000 00:2a 21510337721                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rcache_gpusm.so
[1,1]<stderr>:2aaabce20000-2aaabd01f000 ---p 00002000 00:2a 21510337721                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rcache_gpusm.so
[1,1]<stderr>:2aaabd01f000-2aaabd020000 r--p 00001000 00:2a 21510337721                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rcache_gpusm.so
[1,1]<stderr>:2aaabd020000-2aaabd021000 rw-p 00002000 00:2a 21510337721                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_rcache_gpusm.so
[1,1]<stderr>:2aaabd021000-2aaabd024000 r-xp 00000000 00:2a 21510337709                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_mpool_hugepage.so
[1,1]<stderr>:2aaabd024000-2aaabd223000 ---p 00003000 00:2a 21510337709                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_mpool_hugepage.so
[1,1]<stderr>:2aaabd223000-2aaabd224000 r--p 00002000 00:2a 21510337709                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_mpool_hugepage.so
[1,1]<stderr>:2aaabd224000-2aaabd225000 rw-p 00003000 00:2a 21510337709                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_mpool_hugepage.so
[1,1]<stderr>:2aaabd225000-2aaabd229000 r-xp 00000000 00:2a 21510341939                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_bml_r2.so
[1,1]<stderr>:2aaabd229000-2aaabd428000 ---p 00004000 00:2a 21510341939                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_bml_r2.so
[1,1]<stderr>:2aaabd428000-2aaabd429000 r--p 00003000 00:2a 21510341939                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_bml_r2.so
[1,1]<stderr>:2aaabd429000-2aaabd42a000 rw-p 00004000 00:2a 21510341939                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_bml_r2.so
[1,1]<stderr>:2aaabd42a000-2aaabd42d000 r-xp 00000000 00:2a 21510337697                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_btl_self.so
[1,1]<stderr>:2aaabd42d000-2aaabd62c000 ---p 00003000 00:2a 21510337697                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_btl_self.so
[1,1]<stderr>:2aaabd62c000-2aaabd62d000 r--p 00002000 00:2a 21510337697                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_btl_self.so
[1,1]<stderr>:2aaabd62d000-2aaabd62e000 rw-p 00003000 00:2a 21510337697                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_btl_self.so
[1,1]<stderr>:2aaabd62e000-2aaabd636000 r-xp 00000000 00:2a 21510337699                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_btl_sm.so
[1,1]<stderr>:2aaabd636000-2aaabd835000 ---p 00008000 00:2a 21510337699                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_btl_sm.so
[1,1]<stderr>:2aaabd835000-2aaabd836000 r--p 00007000 00:2a 21510337699                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_btl_sm.so
[1,1]<stderr>:2aaabd836000-2aaabd837000 rw-p 00008000 00:2a 21510337699                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_btl_sm.so
[1,1]<stderr>:2aaabd837000-2aaabd839000 r-xp 00000000 00:2a 20427940283                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmca_common_sm.so.20.10.1
[1,1]<stderr>:2aaabd839000-2aaabda38000 ---p 00002000 00:2a 20427940283                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmca_common_sm.so.20.10.1
[1,1]<stderr>:2aaabda38000-2aaabda39000 r--p 00001000 00:2a 20427940283                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmca_common_sm.so.20.10.1
[1,1]<stderr>:2aaabda39000-2aaabda3a000 rw-p 00002000 00:2a 20427940283                /trinity/shar[1,0]<stderr>:nity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_pml_ob1.so
[1,0]<stderr>:2aaabe2c0000-2aaabe2c1000 r--p 00026000 00:2a 21510345247                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_pml_ob1.so
[1,0]<stderr>:2aaabe2c1000-2aaabe2c2000 rw-p 00027000 00:2a 21510345247                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_pml_ob1.so
[1,0]<stderr>:2aaabe2c2000-2aaabe2ca000 r-xp 00000000 00:2a 21510345257                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_vprotocol_pessimist.so
[1,0]<stderr>:2aaabe2ca000-2aaabe4c9000 ---p 00008000 00:2a 21510345257                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_vprotocol_pessimist.so
[1,0]<stderr>:2aaabe4c9000-2aaabe4ca000 r--p 00007000 00:2a 21510345257                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_vprotocol_pessimist.so
[1,0]<stderr>:2aaabe4ca000-2aaabe4cb000 rw-p 00008000 00:2a 21510345257                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_vprotocol_pessimist.so
[1,0]<stderr>:2aaabe4cb000-2aaabe4d1000 r-xp 00000000 00:2a 21510341943                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_sm.so
[1,0]<stderr>:2aaabe4d1000-2aaabe6d0000 ---p 00006000 00:2a 21510341943                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_sm.so
[1,0]<stderr>:2aaabe6d0000-2aaabe6d1000 r--p 00005000 00:2a 21510341943                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_sm.so
[1,0]<stderr>:2aaabe6d1000-2aaabe6d2000 rw-p 00006000 00:2a 21510341943                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_sm.so
[1,0]<stderr>:2aaabe6d2000-2aaabe6f0000 r-xp 00000000 00:2a 21510341945                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_libnbc.so
[1,0]<stderr>:2aaabe6f0000-2aaabe8ef000 ---p 0001e000 00:2a 21510341945                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_libnbc.so
[1,0]<stderr>:2aaabe8ef000-2aaabe8f0000 r--p 0001d000 00:2a 21510341945                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_libnbc.so
[1,0]<stderr>:2aaabe8f0000-2aaabe8f1000 rw-p 0001e000 00:2a 21510341945                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_libnbc.so
[1,0]<stderr>:2aaabe8f1000-2aaabe8f5000 r-xp 00000000 00:2a 21510341947                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_inter.so
[1,0]<stderr>:2aaabe8f5000-2aaabeaf4000 ---p 00004000 00:2a 21510341947                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_inter.so
[1,0]<stderr>:2aaabeaf4000-2aaabeaf5000 r--p 00003000 00:2a 21510341947                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_inter.so
[1,0]<stderr>:2aaabeaf5000-2aaabeaf6000 rw-p 00004000 00:2a 21510341947                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_inter.so
[1,0]<stderr>:2aaabeaf6000-2aaabeb03000 r-xp 00000000 00:2a 21510341949                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_basic.so
[1,0]<stderr>:2aaabeb03000-2aaabed02000 ---p 0000d000 00:2a 21510341949                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_basic.so
[1,0]<stderr>:2aaabed02000-2aaabed03000 r--p 0000c000 00:2a 21510341949                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_basic.so
[1,0]<stderr>:2aaabed03000-2aaabed04000 rw-p 0000d000 00:2a 21510341949                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_basic.so
[1,0]<stderr>:2aaabed04000-2aaabed14000 r-xp 00000000 00:2a 21510341951                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_tuned.so
[1,0]<stderr>:2aaabed14000-2aaabef13000 ---p 00010000 00:2a 21510341951                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_tuned.so
[1,0]<stderr>:2aaabef13000-2aaabef14000 r--p 0000f000 00:2a 21510341951                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_tuned.so
[1,0]<stderr>:2aaabef14000-2aaabef[1,1]<stderr>:ed/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/libmca_common_sm.so.20.10.1
[1,1]<stderr>:2aaabda3a000-2aaabda71000 r-xp 00000000 00:2a 21510337703                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_btl_openib.so
[1,1]<stderr>:2aaabda71000-2aaabdc70000 ---p 00037000 00:2a 21510337703                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_btl_openib.so
[1,1]<stderr>:2aaabdc70000-2aaabdc71000 r--p 00036000 00:2a 21510337703                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_btl_openib.so
[1,1]<stderr>:2aaabdc71000-2aaabdc73000 rw-p 00037000 00:2a 21510337703                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_btl_openib.so
[1,1]<stderr>:2aaabdc73000-2aaabdc74000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaabdc74000-2aaabdc8e000 r-xp 00000000 08:02 1060079                    /usr/lib64/librdmacm.so.1.0.0
[1,1]<stderr>:2aaabdc8e000-2aaabde8d000 ---p 0001a000 08:02 1060079                    /usr/lib64/librdmacm.so.1.0.0
[1,1]<stderr>:2aaabde8d000-2aaabde8e000 r--p 00019000 08:02 1060079                    /usr/lib64/librdmacm.so.1.0.0
[1,1]<stderr>:2aaabde8e000-2aaabde8f000 rw-p 0001a000 08:02 1060079                    /usr/lib64/librdmacm.so.1.0.0
[1,1]<stderr>:2aaabde8f000-2aaabde90000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaabde90000-2aaabde93000 r-xp 00000000 00:2a 21510341941                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_sync.so
[1,1]<stderr>:2aaabde93000-2aaabe092000 ---p 00003000 00:2a 21510341941                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_sync.so
[1,1]<stderr>:2aaabe092000-2aaabe093000 r--p 00002000 00:2a 21510341941                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_sync.so
[1,1]<stderr>:2aaabe093000-2aaabe094000 rw-p 00003000 00:2a 21510341941                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_sync.so
[1,1]<stderr>:2aaabe09a000-2aaabe0c1000 r-xp 00000000 00:2a 21510345247                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_pml_ob1.so
[1,1]<stderr>:2aaabe0c1000-2aaabe2c0000 ---p 00027000 00:2a 21510345247                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_pml_ob1.so
[1,1]<stderr>:2aaabe2c0000-2aaabe2c1000 r--p 00026000 00:2a 21510345247                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_pml_ob1.so
[1,1]<stderr>:2aaabe2c1000-2aaabe2c2000 rw-p 00027000 00:2a 21510345247                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_pml_ob1.so
[1,1]<stderr>:2aaabe2c2000-2aaabe2ca000 r-xp 00000000 00:2a 21510345257                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_vprotocol_pessimist.so
[1,1]<stderr>:2aaabe2ca000-2aaabe4c9000 ---p 00008000 00:2a 21510345257                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_vprotocol_pessimist.so
[1,1]<stderr>:2aaabe4c9000-2aaabe4ca000 r--p 00007000 00:2a 21510345257                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_vprotocol_pessimist.so
[1,1]<stderr>:2aaabe4ca000-2aaabe4cb000 rw-p 00008000 00:2a 21510345257                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_vprotocol_pessimist.so
[1,1]<stderr>:2aaabe4cb000-2aaabe4d1000 r-xp 00000000 00:2a 21510341943                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_sm.so
[1,1]<stderr>:2aaabe4d1000-2aaabe6d0000 ---p 00006000 00:2a 21510341943                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_sm.so
[1,1]<stderr>:2aaabe6d0000-2aaabe6d1000 r--p 00005000 00:2a 21510341943                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_sm.so
[1,1]<stderr>:2aaabe6d1000-2aaabe6d2000 rw-p 00006000 00:2a 21510341943                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_sm.so
[1,1]<stderr>:2aaabe6d2000-2aaabe6f0000 r-xp 00000000 00:2a 21510341945                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_libnbc.so
[1,1]<stderr>:2aaabe6f0000-2aaabe8ef000 ---p 0001e000 00:2a 21510341945                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_libnbc.so
[1,1]<stderr>:2aa[1,0]<stderr>:15000 rw-p 00010000 00:2a 21510341951                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_tuned.so
[1,0]<stderr>:2aaabef15000-2aaabef17000 r-xp 00000000 00:2a 21510345217                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_self.so
[1,0]<stderr>:2aaabef17000-2aaabf116000 ---p 00002000 00:2a 21510345217                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_self.so
[1,0]<stderr>:2aaabf116000-2aaabf117000 r--p 00001000 00:2a 21510345217                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_self.so
[1,0]<stderr>:2aaabf117000-2aaabf118000 rw-p 00002000 00:2a 21510345217                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_self.so
[1,0]<stderr>:2aaabf118000-2aaabf11b000 r-xp 00000000 00:2a 21510345219                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_cuda.so
[1,0]<stderr>:2aaabf11b000-2aaabf31a000 ---p 00003000 00:2a 21510345219                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_cuda.so
[1,0]<stderr>:2aaabf31a000-2aaabf31b000 r--p 00002000 00:2a 21510345219                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_cuda.so
[1,0]<stderr>:2aaabf31b000-2aaabf31c000 rw-p 00003000 00:2a 21510345219                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_cuda.so
[1,0]<stderr>:2aaabf31c000-2aaabf322000 r-xp 00000000 00:2a 21510345239                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_osc_sm.so
[1,0]<stderr>:2aaabf322000-2aaabf521000 ---p 00006000 00:2a 21510345239                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_osc_sm.so
[1,0]<stderr>:2aaabf521000-2aaabf522000 r--p 00005000 00:2a 21510345239                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_osc_sm.so
[1,0]<stderr>:2aaabf522000-2aaabf524000 rw-p 00006000 00:2a 21510345239                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_osc_sm.so
[1,0]<stderr>:2aaabf524000-2aaabf544000 r-xp 00000000 00:2a 21510345241                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_osc_pt2pt.so
[1,0]<stderr>:2aaabf544000-2aaabf743000 ---p 00020000 00:2a 21510345241                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_osc_pt2pt.so
[1,0]<stderr>:2aaabf743000-2aaabf744000 r--p 0001f000 00:2a 21510345241                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_osc_pt2pt.so
[1,0]<stderr>:2aaabf744000-2aaabf745000 rw-p 00020000 00:2a 21510345241                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_osc_pt2pt.so
[1,0]<stderr>:2aaabf745000-2aaabf772000 r-xp 00000000 00:2a 21510345243                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_osc_rdma.so
[1,0]<stderr>:2aaabf772000-2aaabf972000 ---p 0002d000 00:2a 21510345243                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_osc_rdma.so
[1,0]<stderr>:2aaabf972000-2aaabf973000 r--p 0002d000 00:2a 21510345243                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_osc_rdma.so
[1,0]<stderr>:2aaabf973000-2aaabf974000 rw-p 0002e000 00:2a 21510345243                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_osc_rdma.so
[1,0]<stderr>:2aaabf974000-2aaabf975000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaabf975000-2aaabfa76000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaabfa76000-2aaabfa78000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaabfa78000-2aaabfad9000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaabfad9000-2aaabfada000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaabfb00000-2aaabfc00000 rw-s 01402000 00:05 23595                      /dev/infiniband/uverbs0
[1,0]<stderr>:2aaabfc00000-2aaabfd00000 rw-s 01402000 00:05 23595                      /dev/infiniband/uverbs0
[1,0]<stderr>:2aaabfd00000-2aaabfd01000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaabfd01000-2aaabfd62000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaabfd62000-2aaabfd64000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaabfd64000-2aaabfdc5000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaabfdc5000-2aaabfdc6000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaabfdc6000-2aaabfdc7000 r-xp 00000000 00:2a 21510345255                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/[1,1]<stderr>:abe8ef000-2aaabe8f0000 r--p 0001d000 00:2a 21510341945                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_libnbc.so
[1,1]<stderr>:2aaabe8f0000-2aaabe8f1000 rw-p 0001e000 00:2a 21510341945                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_libnbc.so
[1,1]<stderr>:2aaabe8f1000-2aaabe8f5000 r-xp 00000000 00:2a 21510341947                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_inter.so
[1,1]<stderr>:2aaabe8f5000-2aaabeaf4000 ---p 00004000 00:2a 21510341947                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_inter.so
[1,1]<stderr>:2aaabeaf4000-2aaabeaf5000 r--p 00003000 00:2a 21510341947                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_inter.so
[1,1]<stderr>:2aaabeaf5000-2aaabeaf6000 rw-p 00004000 00:2a 21510341947                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_inter.so
[1,1]<stderr>:2aaabeaf6000-2aaabeb03000 r-xp 00000000 00:2a 21510341949                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_basic.so
[1,1]<stderr>:2aaabeb03000-2aaabed02000 ---p 0000d000 00:2a 21510341949                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_basic.so
[1,1]<stderr>:2aaabed02000-2aaabed03000 r--p 0000c000 00:2a 21510341949                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_basic.so
[1,1]<stderr>:2aaabed03000-2aaabed04000 rw-p 0000d000 00:2a 21510341949                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_basic.so
[1,1]<stderr>:2aaabed04000-2aaabed14000 r-xp 00000000 00:2a 21510341951                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_tuned.so
[1,1]<stderr>:2aaabed14000-2aaabef13000 ---p 00010000 00:2a 21510341951                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_tuned.so
[1,1]<stderr>:2aaabef13000-2aaabef14000 r--p 0000f000 00:2a 21510341951                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_tuned.so
[1,1]<stderr>:2aaabef14000-2aaabef15000 rw-p 00010000 00:2a 21510341951                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_tuned.so
[1,1]<stderr>:2aaabef15000-2aaabef17000 r-xp 00000000 00:2a 21510345217                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_self.so
[1,1]<stderr>:2aaabef17000-2aaabf116000 ---p 00002000 00:2a 21510345217                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_self.so
[1,1]<stderr>:2aaabf116000-2aaabf117000 r--p 00001000 00:2a 21510345217                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_self.so
[1,1]<stderr>:2aaabf117000-2aaabf118000 rw-p 00002000 00:2a 21510345217                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_self.so
[1,1]<stderr>:2aaabf118000-2aaabf11b000 r-xp 00000000 00:2a 21510345219                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_cuda.so
[1,1]<stderr>:2aaabf11b000-2aaabf31a000 ---p 00003000 00:2a 21510345219                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_cuda.so
[1,1]<stderr>:2aaabf31a000-2aaabf31b000 r--p 00002000 00:2a 21510345219                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_cuda.so
[1,1]<stderr>:2aaabf31b000-2aaabf31c000 rw-p 00003000 00:2a 21510345219                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_coll_cuda.so
[1,1]<stderr>:2aaabf31c000-2aaabf322000 r-xp 00000000 00:2a 21510345239                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_osc_sm.so
[1,1]<stderr>:2aaabf322000-2aaabf521000 ---p 00006000 00:2a 21510345239                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_osc_sm.so
[1,1]<stderr>:2aaabf521000-2aaabf522000 r--p 00005000 00:2a 21510345239                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_osc_sm.so
[1,1]<stderr>:2aaabf522000-2aaabf524000 rw-p 00006000 00:2a 21510345239                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/m[1,0]<stderr>:lib/openmpi/mca_topo_basic.so
[1,0]<stderr>:2aaabfdc7000-2aaabffc6000 ---p 00001000 00:2a 21510345255                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_topo_basic.so
[1,0]<stderr>:2aaabffc6000-2aaabffc7000 r--p 00000000 00:2a 21510345255                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_topo_basic.so
[1,0]<stderr>:2aaabffc7000-2aaabffc8000 rw-p 00001000 00:2a 21510345255                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_topo_basic.so
[1,0]<stderr>:2aaac0000000-2aaac0001000 rw-s 00000000 00:05 18894                      /dev/nvidia0
[1,0]<stderr>:2aaac0001000-2aaac0002000 rw-s 00000000 00:05 18894                      /dev/nvidia0
[1,0]<stderr>:2aaac0002000-2aaac0003000 rw-s 00000000 00:05 18894                      /dev/nvidia0
[1,0]<stderr>:2aaac0003000-2aaac0004000 rw-s 00000000 00:05 18894                      /dev/nvidia0
[1,0]<stderr>:2aaac0004000-2aaac0005000 rw-s 00000000 00:05 18894                      /dev/nvidia0
[1,0]<stderr>:2aaac0005000-2aaac0006000 rw-s 00000000 00:05 18894                      /dev/nvidia0
[1,0]<stderr>:2aaac0006000-2aaac0007000 rw-s 00000000 00:05 18894                      /dev/nvidia0
[1,0]<stderr>:2aaac0007000-2aaac0008000 rw-s 00000000 00:05 18894                      /dev/nvidia0
[1,0]<stderr>:2aaac0008000-2aaac0009000 rw-s 00000000 00:05 18894                      /dev/nvidia0
[1,0]<stderr>:2aaac0009000-2aaac000a000 rw-s 00000000 00:05 18894                      /dev/nvidia0
[1,0]<stderr>:2aaac000a000-2aaac000b000 rw-s 00000000 00:05 18894                      /dev/nvidia0
[1,0]<stderr>:2aaac000b000-2aaac000c000 rw-s 00000000 00:05 18894                      /dev/nvidia0
[1,0]<stderr>:2aaac000c000-2aaac000d000 rw-s 00000000 00:05 18894                      /dev/nvidia0
[1,0]<stderr>:2aaac000d000-2aaac000e000 rw-s 00000000 00:05 18894                      /dev/nvidia0
[1,0]<stderr>:2aaac000e000-2aaac000f000 rw-s 00000000 00:05 18894                      /dev/nvidia0
[1,0]<stderr>:2aaac000f000-2aaac0010000 rw-s 00000000 00:05 18894                      /dev/nvidia0
[1,0]<stderr>:2aaac0010000-2aaad0000000 ---p 00000000 00:00 0 
[1,0]<stderr>:2aaad0000000-2aaad8001000 rw-s 00000000 08:02 1966106                    /tmp/openmpi-sessions-9413@kepler002_0/6259/1/shared_mem_pool.kepler002 (deleted)
[1,0]<stderr>:2aaad8001000-2aaad8002000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaad8002000-2aaad8403000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaad8403000-2aaad8405000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaad8405000-2aaad8806000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaad8806000-2aaad8808000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaad8808000-2aaad8c09000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaad8c09000-2aaad8c0b000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaad8c0b000-2aaad900c000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaad900c000-2aaad900e000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaad900e000-2aaad940f000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaad940f000-2aaad9411000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaad9411000-2aaad9812000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaad9812000-2aaad9814000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaad9814000-2aaad9c15000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaad9c15000-2aaad9c17000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaad9c17000-2aaada018000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaada018000-2aaada019000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaada019000-2aaada01a000 ---p 00000000 00:00 0 
[1,0]<stderr>:2aaada01a000-2aaada21a000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaada21a000-2aaada21b000 ---p 00000000 00:00 0 
[1,0]<stderr>:2aaada21b000-2aaada41b000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaada41b000-2aaada41c000 ---p 00000000 00:00 0 
[1,0]<stderr>:2aaada41c000-2aaada61c000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaada61c000-2aaada61d000 ---p 00000000 00:00 0 
[1,0]<stderr>:2aaada61d000-2aaada81d000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaada81d000-2aaada85f000 r-xp 00000000 00:2a 21510345235                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_io_romio314.so
[1,0]<stderr>:2aaada85f000-2aaadaa5e000 ---p 00042000 00:2a 21510345235                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_io_romio314.so
[1,0]<stderr>:2aaadaa5e000-2aaadaa5f000 r--p 00041000 00:2a 21510345235                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_io_romio314.so
[1,0]<stderr>:2aaadaa5f000-2aaadaa61000 rw-p 00042000 00:2a 21510345235                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_io_romio314.so
[1,0]<stderr>:2aaadaa61000-2aaadaa75000 r-xp 00000000 08:02 1185165                    /usr/lpp/mmfs/lib/libgpfs.so
[1,0]<stderr>:2aaadaa75000-2aaadac74000 ---p 00014000 08[1,1]<stderr>:ca_osc_sm.so
[1,1]<stderr>:2aaabf524000-2aaabf544000 r-xp 00000000 00:2a 21510345241                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_osc_pt2pt.so
[1,1]<stderr>:2aaabf544000-2aaabf743000 ---p 00020000 00:2a 21510345241                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_osc_pt2pt.so
[1,1]<stderr>:2aaabf743000-2aaabf744000 r--p 0001f000 00:2a 21510345241                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_osc_pt2pt.so
[1,1]<stderr>:2aaabf744000-2aaabf745000 rw-p 00020000 00:2a 21510345241                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_osc_pt2pt.so
[1,1]<stderr>:2aaabf745000-2aaabf772000 r-xp 00000000 00:2a 21510345243                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_osc_rdma.so
[1,1]<stderr>:2aaabf772000-2aaabf972000 ---p 0002d000 00:2a 21510345243                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_osc_rdma.so
[1,1]<stderr>:2aaabf972000-2aaabf973000 r--p 0002d000 00:2a 21510345243                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_osc_rdma.so
[1,1]<stderr>:2aaabf973000-2aaabf974000 rw-p 0002e000 00:2a 21510345243                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_osc_rdma.so
[1,1]<stderr>:2aaabf974000-2aaabf975000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaabf975000-2aaabfa76000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaabfa76000-2aaabfa78000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaabfa78000-2aaabfad9000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaabfad9000-2aaabfada000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaabfb00000-2aaabfc00000 rw-s 01402000 00:05 23595                      /dev/infiniband/uverbs0
[1,1]<stderr>:2aaabfc00000-2aaabfd00000 rw-s 01402000 00:05 23595                      /dev/infiniband/uverbs0
[1,1]<stderr>:2aaabfd00000-2aaabfd01000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaabfd01000-2aaabfd62000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaabfd62000-2aaabfd64000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaabfd64000-2aaabfdc5000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaabfdc5000-2aaabfdc6000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaabfdc6000-2aaabfdc7000 r-xp 00000000 00:2a 21510345255                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_topo_basic.so
[1,1]<stderr>:2aaabfdc7000-2aaabffc6000 ---p 00001000 00:2a 21510345255                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_topo_basic.so
[1,1]<stderr>:2aaabffc6000-2aaabffc7000 r--p 00000000 00:2a 21510345255                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_topo_basic.so
[1,1]<stderr>:2aaabffc7000-2aaabffc8000 rw-p 00001000 00:2a 21510345255                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_topo_basic.so
[1,1]<stderr>:2aaac0000000-2aaac0001000 rw-s 00000000 00:05 18898                      /dev/nvidia1
[1,1]<stderr>:2aaac0001000-2aaac0002000 rw-s 00000000 00:05 18898                      /dev/nvidia1
[1,1]<stderr>:2aaac0002000-2aaac0003000 rw-s 00000000 00:05 18898                      /dev/nvidia1
[1,1]<stderr>:2aaac0003000-2aaac0004000 rw-s 00000000 00:05 18898                      /dev/nvidia1
[1,1]<stderr>:2aaac0004000-2aaac0005000 rw-s 00000000 00:05 18898                      /dev/nvidia1
[1,1]<stderr>:2aaac0005000-2aaac0006000 rw-s 00000000 00:05 18898                      /dev/nvidia1
[1,1]<stderr>:2aaac0006000-2aaac0007000 rw-s 00000000 00:05 18898                      /dev/nvidia1
[1,1]<stderr>:2aaac0007000-2aaac0008000 rw-s 00000000 00:05 18898                      /dev/nvidia1
[1,1]<stderr>:2aaac0008000-2aaac0009000 rw-s 00000000 00:05 18898                      /dev/nvidia1
[1,1]<stderr>:2aaac0009000-2aaac000a000 rw-s 00000000 00:05 18898                      /dev/nvidia1
[1,1]<stderr>:2aaac000a000-2aaac000b000 rw-s 00000000 00:05 18898                      /dev/nvidia1
[1,1]<stderr>:2aaac000b000-2aaac000c000 rw-s 00000000 00:05 18898                      /dev/nvidia1
[1,1]<stderr>:2aaac000c000-2aaac000d000 rw-s 00000000 00:05 18898                      /dev/nvidia1
[1,1]<stderr>:2aaac000d000-2aaac000e000 rw-s 00000000 00:05 18898                      /dev/nvidia1
[1,1]<stderr>:2aaac000e000-2aaac000f000 rw-s 00000000 00:05 18898                      /dev/nvidia1
[1,1]<stderr>:2aaac000f000-2aaac0010000 rw-s 00000000 00:05 18898                      /dev/nvidia1
[1,1]<stderr>:2aaac0010000-2aaad0000000 ---p 00000000 00:00 0 
[1,1]<stderr>:2aaad0000000-2aaad8001000 rw-s 00000000 08:02 1966106                    /tmp/[1,0]<stderr>::02 1185165                    /usr/lpp/mmfs/lib/libgpfs.so
[1,0]<stderr>:2aaadac74000-2aaadac75000 r--p 00013000 08:02 1185165                    /usr/lpp/mmfs/lib/libgpfs.so
[1,0]<stderr>:2aaadac75000-2aaadac76000 rw-p 00014000 08:02 1185165                    /usr/lpp/mmfs/lib/libgpfs.so
[1,0]<stderr>:2aaadc000000-2aaadc021000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaadc021000-2aaae0000000 ---p 00000000 00:00 0 
[1,0]<stderr>:2aaae0000000-2aaae0021000 rw-p 00000000 00:00 0 
[1,0]<stderr>:2aaae0021000-2aaae4000000 ---p 00000000 00:00 0 
[1,0]<stderr>:7ffffffd6000-7ffffffff000 rw-p 00000000 00:00 0                          [stack]
[1,0]<stderr>:ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
[1,0]<stderr>:[kepler002:02444] *** Process received signal ***
[1,0]<stderr>:[kepler002:02444] Signal: Aborted (6)
[1,0]<stderr>:[kepler002:02444] Signal code:  (-6)
[1,0]<stderr>:[kepler002:02444] [ 0] /usr/lib64/libpthread.so.0(+0xf6d0)[0x2aaaaacde6d0]
[1,0]<stderr>:[kepler002:02444] [ 1] /usr/lib64/libc.so.6(gsignal+0x37)[0x2aaaae198277]
[1,0]<stderr>:[kepler002:02444] [ 2] /usr/lib64/libc.so.6(abort+0x148)[0x2aaaae199968]
[1,0]<stderr>:[kepler002:02444] [ 3] /usr/lib64/libc.so.6(+0x78d37)[0x2aaaae1dad37]
[1,0]<stderr>:[kepler002:02444] [ 4] /usr/lib64/libc.so.6(+0x81499)[0x2aaaae1e3499]
[1,0]<stderr>:[kepler002:02444] [ 5] /home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZN16cupla_cuda_async13cuplaFreeHostEPv+0xab)[0x153b71b]
[1,0]<stderr>:[kepler002:02444] [ 6] /home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZN5pmacc6BufferINS_9SuperCellINS_5FrameINS_15ParticlesBufferINS_19ParticleDescriptionINS_4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEENS_4math2CT3IntILi16ELi16ELi2147483647EEEN5boost3mpl6v_itemIN8picongpu24placeholder_definition326probeEENSE_INSF_24placeholder_definition336probeBENSE_INSF_24placeholder_definition258positionINSF_24placeholder_definition2712position_picENS_24placeholder_definition2213pmacc_isAliasEEENSD_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENSE_INSF_24placeholder_definition4513interpolationINSF_28FieldToParticleInterpolationINSF_9particles6shapes3TSCENSF_30AssignedTrilinearInterpolationEEESP_EENSE_INSF_24placeholder_definition385shapeIS13_SP_EENSE_INSF_24placeholder_definition3914particlePusherINS11_6pusher5ProbeESP_EESU_Li0EEELi0EEELi0EEENS_17HandleGuardRegionINS_9particles8policies17ExchangeParticlesENS11_8boundary29CallPluginsAndDeleteParticlesEEESU_SU_EESB_N8mallocMC9AllocatorINS1Q_16CreationPolicies7ScatterINSF_16DeviceHeapConfigENS1S_11ScatterConf27DefaultScatterHashingParamsEEENS1Q_20DistributionPolicies4NoopENS1Q_11OOMPolicies10ReturnNullENS1Q_19ReservePoolPolicies16SimpleCudaMallocENS1Q_17AlignmentPolicies6ShrinkINS24_12ShrinkConfig19DefaultShrinkConfigEEEEELj2EE29OperatorCreatePairStaticArrayILj256EEENS4_IS7_SB_NSE_INS_24placeholder_definition249multiMaskENSE_INS_24placeholder_definition2312localCellIdxESX_Li0EEELi0EEES1H_S1O_SU_NSE_INS_12NextFramePtrINSS_3argILi1EEEEENSE_INS_16PreviousFramePtrIS2L_EESU_Li0EEELi0EEEEEEEEELj2EED2Ev+0x11)[0x13dc041]
[1,0]<stderr>:[kepler002:02444] [ 7] /home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZN5pmacc10GridBufferINS_9SuperCellINS_5FrameINS_15ParticlesBufferINS_19ParticleDescriptionINS_4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEENS_4math2CT3IntILi16ELi16ELi2147483647EEEN5boost3mpl6v_itemIN8picongpu24placeholder_definition326probeEENSE_INSF_24placeholder_definition336probeBENSE_INSF_24placeholder_definition258positionINSF_24placeholder_definition2712position_picENS_24placeholder_definition2213pmacc_isAliasEEENSD_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENSE_INSF_24placeholder_definition4513interpolationINSF_28FieldToParticleInterpolationINSF_9particles6shapes3TSCENSF_30AssignedTrilinearInterpolationEEESP_EENSE_INSF_24placeholder_definition385shapeIS13_SP_EENSE_I[1,1]<stderr>:openmpi-sessions-9413@kepler002_0/6259/1/shared_mem_pool.kepler002 (deleted)
[1,1]<stderr>:2aaad8001000-2aaad8002000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaad8002000-2aaad8403000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaad8403000-2aaad8405000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaad8405000-2aaad8806000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaad8806000-2aaad8808000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaad8808000-2aaad8c09000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaad8c09000-2aaad8c0b000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaad8c0b000-2aaad900c000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaad900c000-2aaad900e000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaad900e000-2aaad940f000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaad940f000-2aaad9411000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaad9411000-2aaad9812000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaad9812000-2aaad9814000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaad9814000-2aaad9c15000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaad9c15000-2aaad9c17000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaad9c17000-2aaada018000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaada018000-2aaada019000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaada019000-2aaada01a000 ---p 00000000 00:00 0 
[1,1]<stderr>:2aaada01a000-2aaada21a000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaada21a000-2aaada21b000 ---p 00000000 00:00 0 
[1,1]<stderr>:2aaada21b000-2aaada41b000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaada41b000-2aaada41c000 ---p 00000000 00:00 0 
[1,1]<stderr>:2aaada41c000-2aaada61c000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaada61c000-2aaada61d000 ---p 00000000 00:00 0 
[1,1]<stderr>:2aaada61d000-2aaada81d000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaada81d000-2aaada85f000 r-xp 00000000 00:2a 21510345235                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_io_romio314.so
[1,1]<stderr>:2aaada85f000-2aaadaa5e000 ---p 00042000 00:2a 21510345235                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_io_romio314.so
[1,1]<stderr>:2aaadaa5e000-2aaadaa5f000 r--p 00041000 00:2a 21510345235                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_io_romio314.so
[1,1]<stderr>:2aaadaa5f000-2aaadaa61000 rw-p 00042000 00:2a 21510345235                /trinity/shared/pkg/mpi/openmpi/2.1.2-cuda100/gcc/7.3.0/lib/openmpi/mca_io_romio314.so
[1,1]<stderr>:2aaadaa61000-2aaadaa75000 r-xp 00000000 08:02 1185165                    /usr/lpp/mmfs/lib/libgpfs.so
[1,1]<stderr>:2aaadaa75000-2aaadac74000 ---p 00014000 08:02 1185165                    /usr/lpp/mmfs/lib/libgpfs.so
[1,1]<stderr>:2aaadac74000-2aaadac75000 r--p 00013000 08:02 1185165                    /usr/lpp/mmfs/lib/libgpfs.so
[1,1]<stderr>:2aaadac75000-2aaadac76000 rw-p 00014000 08:02 1185165                    /usr/lpp/mmfs/lib/libgpfs.so
[1,1]<stderr>:2aaadc000000-2aaadc021000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaadc021000-2aaae0000000 ---p 00000000 00:00 0 
[1,1]<stderr>:2aaae0000000-2aaae0021000 rw-p 00000000 00:00 0 
[1,1]<stderr>:2aaae0021000-2aaae4000000 ---p 00000000 00:00 0 
[1,1]<stderr>:7ffffffd6000-7ffffffff000 rw-p 00000000 00:00 0                          [stack]
[1,1]<stderr>:ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
[1,1]<stderr>:[kepler002:02445] *** Process received signal ***
[1,1]<stderr>:[kepler002:02445] Signal: Aborted (6)
[1,1]<stderr>:[kepler002:02445] Signal code:  (-6)
[1,1]<stderr>:[kepler002:02445] [ 0] /usr/lib64/libpthread.so.0(+0xf6d0)[0x2aaaaacde6d0]
[1,1]<stderr>:[kepler002:02445] [ 1] /usr/lib64/libc.so.6(gsignal+0x37)[0x2aaaae198277]
[1,1]<stderr>:[kepler002:02445] [ 2] /usr/lib64/libc.so.6(abort+0x148)[0x2aaaae199968]
[1,1]<stderr>:[kepler002:02445] [ 3] /usr/lib64/libc.so.6(+0x78d37)[0x2aaaae1dad37]
[1,1]<stderr>:[kepler002:02445] [ 4] /usr/lib64/libc.so.6(+0x81499)[0x2aaaae1e3499]
[1,1]<stderr>:[kepler002:02445] [ 5] /home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZN16cupla_cuda_async13cuplaFreeHostEPv+0xab)[0x153b71b]
[1,1]<stderr>:[kepler002:02445] [ 6] /home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZN5pmacc6BufferINS_9SuperCellINS_5FrameINS_15ParticlesBufferINS_19ParticleDescriptionINS_4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEENS_4math2CT3IntILi16ELi16ELi2147483647EEEN5boost3mpl6v_itemIN8picongpu24placeholder_definition326probeEENSE_INSF_24placeholder_definition336probeBENSE_INSF_24placeholder_definition258positionINSF_24placeholder_definition2712position_picENS_24placeholder_definition2213pmacc_isAliasEEENSD[1,0]<stderr>:NSF_24placeholder_definition3914particlePusherINS11_6pusher5ProbeESP_EESU_Li0EEELi0EEELi0EEENS_17HandleGuardRegionINS_9particles8policies17ExchangeParticlesENS11_8boundary29CallPluginsAndDeleteParticlesEEESU_SU_EESB_N8mallocMC9AllocatorINS1Q_16CreationPolicies7ScatterINSF_16DeviceHeapConfigENS1S_11ScatterConf27DefaultScatterHashingParamsEEENS1Q_20DistributionPolicies4NoopENS1Q_11OOMPolicies10ReturnNullENS1Q_19ReservePoolPolicies16SimpleCudaMallocENS1Q_17AlignmentPolicies6ShrinkINS24_12ShrinkConfig19DefaultShrinkConfigEEEEELj2EE29OperatorCreatePairStaticArrayILj256EEENS4_IS7_SB_NSE_INS_24placeholder_definition249multiMaskENSE_INS_24placeholder_definition2312localCellIdxESX_Li0EEELi0EEES1H_S1O_SU_NSE_INS_12NextFramePtrINSS_3argILi1EEEEENSE_INS_16PreviousFramePtrIS2L_EESU_Li0EEELi0EEEEEEEEELj2ES2T_ED1Ev+0xb7c)[0x13dd17c]
[1,0]<stderr>:[kepler002:02444] [ 8] /home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZThn32_N8picongpu9ParticlesIN5pmacc4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEEN5boost3mpl6v_itemINS_24placeholder_definition4513interpolationINS_28FieldToParticleInterpolationINS_9particles6shapes3TSCENS_30AssignedTrilinearInterpolationEEENS1_24placeholder_definition2213pmacc_isAliasEEENS7_INS_24placeholder_definition385shapeISD_SH_EENS7_INS_24placeholder_definition3914particlePusherINSB_6pusher5ProbeESH_EENS6_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENS7_INS_24placeholder_definition326probeEENS7_INS_24placeholder_definition336probeBENS7_INS_24placeholder_definition258positionINS_24placeholder_definition2712position_picESH_EESU_Li0EEELi0EEELi0EEEED0Ev+0x80)[0x13e6540]
[1,0]<stderr>:[kepler002:02444] [ 9] /home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZN5pmacc13DataConnectorD2Ev+0x152)[0x1360322]
[1,0]<stderr>:[kepler002:02444] [10] /usr/lib64/libc.so.6(+0x39bd9)[0x2aaaae19bbd9]
[1,0]<stderr>:[kepler002:02444] [11] /usr/lib64/libc.so.6(+0x39c27)[0x2aaaae19bc27]
[1,0]<stderr>:[kepler002:02444] [12] /usr/lib64/libc.so.6(__libc_start_main+0xfc)[0x2aaaae18444c]
[1,0]<stderr>:[kepler002:02444] [13] [1,1]<stderr>:_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENSE_INSF_24placeholder_definition4513interpolationINSF_28FieldToParticleInterpolationINSF_9particles6shapes3TSCENSF_30AssignedTrilinearInterpolationEEESP_EENSE_INSF_24placeholder_definition385shapeIS13_SP_EENSE_INSF_24placeholder_definition3914particlePusherINS11_6pusher5ProbeESP_EESU_Li0EEELi0EEELi0EEENS_17HandleGuardRegionINS_9particles8policies17ExchangeParticlesENS11_8boundary29CallPluginsAndDeleteParticlesEEESU_SU_EESB_N8mallocMC9AllocatorINS1Q_16CreationPolicies7ScatterINSF_16DeviceHeapConfigENS1S_11ScatterConf27DefaultScatterHashingParamsEEENS1Q_20DistributionPolicies4NoopENS1Q_11OOMPolicies10ReturnNullENS1Q_19ReservePoolPolicies16SimpleCudaMallocENS1Q_17AlignmentPolicies6ShrinkINS24_12ShrinkConfig19DefaultShrinkConfigEEEEELj2EE29OperatorCreatePairStaticArrayILj256EEENS4_IS7_SB_NSE_INS_24placeholder_definition249multiMaskENSE_INS_24placeholder_definition2312localCellIdxESX_Li0EEELi0EEES1H_S1O_SU_NSE_INS_12NextFramePtrINSS_3argILi1EEEEENSE_INS_16PreviousFramePtrIS2L_EESU_Li0EEELi0EEEEEEEEELj2EED2Ev+0x11)[0x13dc041]
[1,1]<stderr>:[kepler002:02445] [ 7] /home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZN5pmacc10GridBufferINS_9SuperCellINS_5FrameINS_15ParticlesBufferINS_19ParticleDescriptionINS_4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEENS_4math2CT3IntILi16ELi16ELi2147483647EEEN5boost3mpl6v_itemIN8picongpu24placeholder_definition326probeEENSE_INSF_24placeholder_definition336probeBENSE_INSF_24placeholder_definition258positionINSF_24placeholder_definition2712position_picENS_24placeholder_definition2213pmacc_isAliasEEENSD_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENSE_INSF_24placeholder_definition4513interpolationINSF_28FieldToParticleInterpolationINSF_9particles6shapes3TSCENSF_30AssignedTrilinearInterpolationEEESP_EENSE_INSF_24placeholder_definition385shapeIS13_SP_EENSE_INSF_24placeholder_definition3914particlePusherINS11_6pusher5ProbeESP_EESU_Li0EEELi0EEELi0EEENS_17HandleGuardRegionINS_9particles8policies17ExchangeParticlesENS11_8boundary29CallPluginsAndDeleteParticlesEEESU_SU_EESB_N8mallocMC9AllocatorINS1Q_16CreationPolicies7ScatterINSF_16DeviceHeapConfigENS1S_11ScatterConf27DefaultScatterHashingParamsEEENS1Q_20DistributionPolicies4NoopENS1Q_11OOMPolicies10ReturnNullENS1Q_19ReservePoolPolicies16SimpleCudaMallocENS1Q_17AlignmentPolicies6ShrinkINS24_12ShrinkConfig19DefaultShrinkConfigEEEEELj2EE29OperatorCreatePairStaticArrayILj256EEENS4_IS7_SB_NSE_INS_24placeholder_definition249multiMaskENSE_INS_24placeholder_definition2312localCellIdxESX_Li0EEELi0EEES1H_S1O_SU_NSE_INS_12NextFramePtrINSS_3argILi1EEEEENSE_INS_16PreviousFramePtrIS2L_EESU_Li0EEELi0EEEEEEEEELj2ES2T_ED1Ev+0xb7c)[0x13dd17c]
[1,1]<stderr>:[kepler002:02445] [ 8] /home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZThn32_N8picongpu9ParticlesIN5pmacc4meta6StringIJLc112ELc114ELc111ELc98ELc101ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0ELc0EEEEN5boost3mpl6v_itemINS_24placeholder_definition4513interpolationINS_28FieldToParticleInterpolationINS_9particles6shapes3TSCENS_30AssignedTrilinearInterpolationEEENS1_24placeholder_definition2213pmacc_isAliasEEENS7_INS_24placeholder_definition385shapeISD_SH_EENS7_INS_24placeholder_definition3914particlePusherINSB_6pusher5ProbeESH_EENS6_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEENS7_INS_24placeholder_definition326probeEENS7_INS_24placeholder_definition336probeBENS7_INS_24placeholder_definition258positionINS_24placeholder_definition2712position_picESH_EESU_Li0EEELi0EEELi0EEEED0Ev+0x80)[0x13e6540]
[1,1]<stderr>:[kepler002:02445] [ 9] /home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu(_ZN5pmacc13DataConnectorD2Ev+0x152)[0x1360322]
[1,1]<stderr>:[kepler002:02445] [10] /usr/lib64/lib[1,0]<stderr>:/home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu[0x11fc3df]
[1,0]<stderr>:[kepler002:02444] *** End of error message ***
[1,1]<stderr>:c.so.6(+0x39bd9)[0x2aaaae19bbd9]
[1,1]<stderr>:[kepler002:02445] [11] /usr/lib64/libc.so.6(+0x39c27)[0x2aaaae19bc27]
[1,1]<stderr>:[kepler002:02445] [12] /usr/lib64/libc.so.6(__libc_start_main+0xfc)[0x2aaaae18444c]
[1,1]<stderr>:[kepler002:02445] [13] /home/carste06/SCRATCH/sims/pullrequest/test21/input/bin/picongpu[0x11fc3df]
[1,1]<stderr>:[kepler002:02445] *** End of error message ***

finnolec on 30 Jul 2019

@finnolec thanks for your report. I'll try to reproduce and start investigating later today, right after our meeting.

sbastrakov on 30 Jul 2019

👍1

@psychocoderHPC has reproduced it and is also looking into.

sbastrakov on 30 Jul 2019

For unknown reason the call to the kernel KernelComputeSupercells with the error

cudaErrorInvalidDeviceFunction = 98
The requested device function does not exist or is not compiled for the proper device architecture.

I checked the ptx code and we had different versions for this code in. It is not 100% sure a special signature is missing. Since the names are mangled it is not easy to fiddle it out.

psychocoderHPC on 31 Jul 2019

I tested the current dev (1347ebb) with CUDA 9.2 and the error is still there. If I roll back all commits and jump back before the update alpaka/cupla (pull request #2958) than all runs fine under CUDA 9.2.
The old code can not be tested with CUDA 10.X since it is not compiling (old alpaka not support's CUDA 10.X)

psychocoderHPC on 31 Jul 2019

After a lot of tears @sbastrakov and I found the problem and know how to fix it. @sbastrakov will create a PR for the fix.

psychocoderHPC on 2 Aug 2019

👍1

And a lot of variations of WTF pronounced.

sbastrakov on 2 Aug 2019

😄1

@steindev If I remember correct you had also in the past the error invalid device function, the fix in #3015 could be interesting for you

psychocoderHPC on 5 Aug 2019

👍1

I checked that after #3015 is merged, the Foil example with 4.cfg successfully ran on k20 and k80. Closing the issue.

sbastrakov on 12 Aug 2019

👍3

Was this page helpful?

0 / 5 - 0 ratings

Related issues

travis-ci suggestions only valid for GNU sed

PrometheusPi · 3Comments

PIConGPU Development Visualization

berceanu · 3Comments

Doc: PMacc (Memory) Boxes

ax3l · 3Comments

Memory consumption

cbontoiu · 3Comments

Bash Completion for Most Important Scripts

ax3l · 4Comments