Wav2letter: WSJ Aborted

Created on 15 Aug 2019  路  6Comments  路  Source: flashlight/wav2letter

I pulled the latest docker of wavelatter and prepared the data. But the Training of WSJ will corrupt. My system is Ubuntu 18.4 and Nvidia-driver version is 410.48. Training command is CUDA_VISIBLE_DEVICES=0 ../../build/Train train --flagsfile configs/conv_glu/train.cfg

The error log is as follow:
root@e10f2f7f6c30:~/wav2letter/recipes/wsj# CUDA_VISIBLE_DEVICES=0 ../../build/Train train --flagsfile configs/conv_glu/train.cfg *** Error in../../build/Train': double free or corruption (!prev): 0x000000003bd34ec0 *
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f3db8a117e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7f3db8a1a37a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f3db8a1e53c]
/usr/lib/x86_64-linux-gnu/libcudnn.so.7(+0x965742)[0x7f3dc0dc0742]
/usr/lib/x86_64-linux-gnu/libcudnn.so.7(+0xa0f04b)[0x7f3dc0e6a04b]
/usr/lib/x86_64-linux-gnu/libcudnn.so.7(+0xa13cae)[0x7f3dc0e6ecae]
/usr/lib/x86_64-linux-gnu/libcudnn.so.7(+0xa782c)[0x7f3dc050282c]
/usr/lib/x86_64-linux-gnu/libcudnn.so.7(cudnnConvolutionBackwardFilter+0x678)[0x7f3dc0502fd8]
../../build/Train[0x6a1742]
../../build/Train[0x652674]
../../build/Train[0x652bf3]
../../build/Train[0x47dcb3]
../../build/Train[0x41ab54]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f3db89ba830]
../../build/Train[0x478d59]
======= Memory map: ========
00400000-0088e000 r-xp 00000000 08:01 7998743 /root/wav2letter/build/Train
00a8e000-00a91000 r--p 0048e000 08:01 7998743 /root/wav2letter/build/Train
00a91000-00a93000 rw-p 00491000 08:01 7998743 /root/wav2letter/build/Train
00a93000-00a9e000 rw-p 00000000 00:00 0
01de7000-3d475000 rw-p 00000000 00:00 0 [heap]
200000000-200200000 rw-s 00000000 00:06 423 /dev/nvidiactl
200200000-200400000 ---p 00000000 00:00 0
200400000-200404000 rw-s 00000000 00:06 423 /dev/nvidiactl
200404000-200600000 ---p 00000000 00:00 0
200600000-200a00000 rw-s 00000000 00:06 423 /dev/nvidiactl
200a00000-201600000 ---p 00000000 00:00 0
201600000-201800000 rw-s 00000000 00:06 423 /dev/nvidiactl
201800000-201804000 rw-s 00000000 00:06 423 /dev/nvidiactl
201804000-201a00000 ---p 00000000 00:00 0
201a00000-201e00000 rw-s 00000000 00:06 423 /dev/nvidiactl
201e00000-201e04000 rw-s 00000000 00:06 423 /dev/nvidiactl
201e04000-202000000 ---p 00000000 00:00 0
202000000-202400000 rw-s 00000000 00:06 423 /dev/nvidiactl
202400000-202404000 rw-s 00000000 00:06 423 /dev/nvidiactl
202404000-202600000 ---p 00000000 00:00 0
202600000-202a00000 rw-s 00000000 00:06 423 /dev/nvidiactl
202a00000-202a04000 rw-s 00000000 00:06 423 /dev/nvidiactl
202a04000-202c00000 ---p 00000000 00:00 0
202c00000-203000000 rw-s 00000000 00:06 423 /dev/nvidiactl
203000000-203004000 rw-s 00000000 00:06 423 /dev/nvidiactl
203004000-203200000 ---p 00000000 00:00 0
203200000-203600000 rw-s 00000000 00:06 423 /dev/nvidiactl
203600000-203604000 rw-s 00000000 00:06 423 /dev/nvidiactl
203604000-203800000 ---p 00000000 00:00 0
203800000-203c00000 rw-s 00000000 00:06 423 /dev/nvidiactl
203c00000-203c04000 rw-s 00000000 00:06 423 /dev/nvidiactl
203c04000-203e00000 ---p 00000000 00:00 0
203e00000-204200000 rw-s 00000000 00:06 423 /dev/nvidiactl
204200000-204204000 rw-s 00000000 00:06 423 /dev/nvidiactl
204204000-204400000 ---p 00000000 00:00 0
204400000-204800000 rw-s 00000000 00:06 423 /dev/nvidiactl
204800000-204804000 rw-s 00000000 00:06 423 /dev/nvidiactl
204804000-204a00000 ---p 00000000 00:00 0
204a00000-204e00000 rw-s 00000000 00:06 423 /dev/nvidiactl
204e00000-204e04000 rw-s 00000000 00:06 423 /dev/nvidiactl
204e04000-205000000 ---p 00000000 00:00 0
205000000-205400000 rw-s 00000000 00:06 423 /dev/nvidiactl
205400000-205404000 rw-s 00000000 00:06 423 /dev/nvidiactl
205404000-205600000 ---p 00000000 00:00 0
205600000-205a00000 rw-s 00000000 00:06 423 /dev/nvidiactl
205a00000-205a04000 rw-s 00000000 00:06 423 /dev/nvidiactl
205a04000-205c00000 ---p 00000000 00:00 0
205c00000-206000000 rw-s 00000000 00:06 423 /dev/nvidiactl
206000000-206004000 rw-s 00000000 00:06 423 /dev/nvidiactl
206004000-206200000 ---p 00000000 00:00 0
206200000-206600000 rw-s 00000000 00:06 423 /dev/nvidiactl
206600000-206604000 rw-s 00000000 00:06 423 /dev/nvidiactl
206604000-206800000 ---p 00000000 00:00 0
206800000-206c00000 rw-s 00000000 00:06 423 /dev/nvidiactl
206c00000-206c04000 rw-s 00000000 00:06 423 /dev/nvidiactl
206c04000-206e00000 ---p 00000000 00:00 0
206e00000-207200000 rw-s 00000000 00:06 423 /dev/nvidiactl
207200000-207400000 ---p 00000000 00:00 0
207400000-207600000 rw-s 00000000 00:06 423 /dev/nvidiactl
207600000-300200000 ---p 00000000 00:00 0
10000000000-10204000000 ---p 00000000 00:00 0
7f3bc8000000-7f3bc8021000 rw-p 00000000 00:00 0
7f3bc8021000-7f3bcc000000 ---p 00000000 00:00 0
7f3bce000000-7f3c6f400000 ---p 00000000 00:00 0
7f3c6f400000-7f3c6f600000 rw-s 00000000 00:05 4386150 /dev/zero (deleted)
7f3c6f600000-7f3c6f800000 ---p 00000000 00:00 0
7f3c6f800000-7f3c6fa00000 rw-s 00000000 00:05 4386151 /dev/zero (deleted)
7f3c6fa00000-7f3c70000000 ---p 00000000 00:00 0
7f3c70000000-7f3c7116d000 rw-p 00000000 00:00 0
7f3c7116d000-7f3c74000000 ---p 00000000 00:00 0
7f3c74631000-7f3c77de7000 r-xp 00000000 08:01 7480085 /opt/intel/compilers_and_libraries_2018.5.274/linux/mkl/lib/intel64_lin/libmkl_avx2.so
7f3c77de7000-7f3c77fe7000 ---p 037b6000 08:01 7480085 /opt/intel/compilers_and_libraries_2018.5.274/linux/mkl/lib/intel64_lin/libmkl_avx2.so
7f3c77fe7000-7f3c77fee000 r--p 037b6000 08:01 7480085 /opt/intel/compilers_and_libraries_2018.5.274/linux/mkl/lib/intel64_lin/libmkl_avx2.so
7f3c77fee000-7f3c77ffc000 rw-p 037bd000 08:01 7480085 /opt/intel/compilers_and_libraries_2018.5.274/linux/mkl/lib/intel64_lin/libmkl_avx2.so
7f3c77ffc000-7f3c78000000 rw-p 00000000 00:00 0
7f3c78000000-7f3c78021000 rw-p 00000000 00:00 0
7f3c78021000-7f3c7c000000 ---p 00000000 00:00 0
7f3c7c000000-7f3c7c021000 rw-p 00000000 00:00 0
7f3c7c021000-7f3c80000000 ---p 00000000 00:00 0
7f3c80000000-7f3c80021000 rw-p 00000000 00:00 0
7f3c80021000-7f3c84000000 ---p 00000000 00:00 0
7f3c84000000-7f3c84021000 rw-p 00000000 00:00 0
7f3c84021000-7f3c88000000 ---p 00000000 00:00 0
7f3c88000000-7f3c88021000 rw-p 00000000 00:00 0
7f3c88021000-7f3c8c000000 ---p 00000000 00:00 0
7f3c8c119000-7f3c8c9c9000 rw-p 00000000 00:00 0
7f3c8d279000-7f3c8d926000 rw-p 00000000 00:00 0
7f3c8dfd3000-7f3c8e3e7000 rw-p 00000000 00:00 0
7f3c8e7fb000-7f3c8e7fc000 ---p 00000000 00:00 0
7f3c8e7fc000-7f3c8effc000 rw-p 00000000 00:00 0
7f3c8effc000-7f3c8effd000 ---p 00000000 00:00 0
7f3c8effd000-7f3c8f3fd000 rw-p 00000000 00:00 0
7f3c8f3fd000-7f3c8f3fe000 ---p 00000000 00:00 0
7f3c8f3fe000-7f3c8f7fe000 rw-p 00000000 00:00 0
7f3c8f7fe000-7f3c8f7ff000 ---p 00000000 00:00 0
7f3c8f7ff000-7f3c8fbff000 rw-p 00000000 00:00 0
7f3c8fbff000-7f3c8fc00000 ---p 00000000 00:00 0
7f3c8fc00000-7f3c90000000 rw-p 00000000 00:00 0
7f3c90000000-7f3c90021000 rw-p 00000000 00:00 0
7f3c90021000-7f3c94000000 ---p 00000000 00:00 0
7f3c94000000-7f3c94021000 rw-p 00000000 00:00 0
7f3c94021000-7f3c98000000 ---p 00000000 00:00 0
7f3c98000000-7f3c98021000 rw-p 00000000 00:00 0
7f3c98021000-7f3c9c000000 ---p 00000000 00:00 0
7f3c9c000000-7f3c9c021000 rw-p 00000000 00:00 0
7f3c9c021000-7f3ca0000000 ---p 00000000 00:00 0
7f3ca0000000-7f3ca0021000 rw-p 00000000 00:00 0
7f3ca0021000-7f3ca4000000 ---p 00000000 00:00 0
7f3ca4000000-7f3ca4021000 rw-p 00000000 00:00 0
7f3ca4021000-7f3ca8000000 ---p 00000000 00:00 0
7f3ca8000000-7f3ca8021000 rw-p 00000000 00:00 0
7f3ca8021000-7f3cac000000 ---p 00000000 00:00 0
7f3cac000000-7f3cac021000 rw-p 00000000 00:00 0
7f3cac021000-7f3cb0000000 ---p 00000000 00:00 0
7f3cb0000000-7f3cb0021000 rw-p 00000000 00:00 0
7f3cb0021000-7f3cb4000000 ---p 00000000 00:00 0
7f3cb4000000-7f3cb4021000 rw-p 00000000 00:00 0
7f3cb4021000-7f3cb8000000 ---p 00000000 00:00 0
7f3cb8000000-7f3cb8021000 rw-p 00000000 00:00 0
7f3cb8021000-7f3cbc000000 ---p 00000000 00:00 0
7f3cbc000000-7f3cbc021000 rw-p 00000000 00:00 0
7f3cbc021000-7f3cc0000000 ---p 00000000 00:00 0
7f3cc0000000-7f3cc0021000 rw-p 00000000 00:00 0
7f3cc0021000-7f3cc4000000 ---p 00000000 00:00 0
7f3cc4000000-7f3cc4021000 rw-p 00000000 00:00 0
7f3cc4021000-7f3cc8000000 ---p 00000000 00:00 0
7f3cc8000000-7f3cc8021000 rw-p 00000000 00:00 0
7f3cc8021000-7f3ccc000000 ---p 00000000 00:00 0
7f3ccc3f1000-7f3ccc3f2000 ---p 00000000 00:00 0
7f3ccc3f2000-7f3ccc7f2000 rw-p 00000000 00:00 0
7f3ccc7f2000-7f3ccc7f3000 ---p 00000000 00:00 0
7f3ccc7f3000-7f3cccbf3000 rw-p 00000000 00:00 0
7f3cccbf3000-7f3cccbf4000 ---p 00000000 00:00 0
7f3cccbf4000-7f3cccff4000 rw-p 00000000 00:00 0
7f3cccff4000-7f3cccff5000 ---p 00000000 00:00 0
7f3cccff5000-7f3ccd3f5000 rw-p 00000000 00:00 0
7f3ccd3f5000-7f3ccd3f6000 ---p 00000000 00:00 0
7f3ccd3f6000-7f3ccd7f6000 rw-p 00000000 00:00 0
7f3ccd7f6000-7f3ccd7f7000 ---p 00000000 00:00 0
7f3ccd7f7000-7f3ccdbf7000 rw-p 00000000 00:00 0
7f3ccdbf7000-7f3ccdbf8000 ---p 00000000 00:00 0
7f3ccdbf8000-7f3ccdff8000 rw-p 00000000 00:00 0
7f3ccdff8000-7f3ccdff9000 ---p 00000000 00:00 0
7f3ccdff9000-7f3cce3f9000 rw-p 00000000 00:00 0
7f3cce3f9000-7f3cce3fa000 ---p 00000000 00:00 0
7f3cce3fa000-7f3cce7fa000 rw-p 00000000 00:00 0
7f3cce7fa000-7f3cce7fb000 ---p 00000000 00:00 0
7f3cce7fb000-7f3ccebfb000 rw-p 00000000 00:00 0
7f3ccebfb000-7f3ccebfc000 ---p 00000000 00:00 0
7f3ccebfc000-7f3cceffc000 rw-p 00000000 00:00 0
7f3cceffc000-7f3cceffd000 ---p 00000000 00:00 0
7f3cceffd000-7f3ccf3fd000 rw-p 00000000 00:00 0
7f3ccf3fd000-7f3ccf3fe000 ---p 00000000 00:00 0
7f3ccf3fe000-7f3ccf7fe000 rw-p 00000000 00:00 0
7f3ccf7fe000-7f3ccf7ff000 ---p 00000000 00:00 0
7f3ccf7ff000-7f3ccfbff000 rw-p 00000000 00:00 0
7f3ccfbff000-7f3ccfc00000 ---p 00000000 00:00 0
7f3ccfc00000-7f3cd0000000 rw-p 00000000 00:00 0
7f3cd0000000-7f3cd0021000 rw-p 00000000 00:00 0
7f3cd0021000-7f3cd4000000 ---p 00000000 00:00 0
7f3cd4000000-7f3cd4021000 rw-p 00000000 00:00 0
7f3cd4021000-7f3cd8000000 ---p 00000000 00:00 0
7f3cd8000000-7f3cd8021000 rw-p 00000000 00:00 0
7f3cd8021000-7f3cdc000000 ---p 00000000 00:00 0
7f3cdc000000-7f3cdc021000 rw-p 00000000 00:00 0
7f3cdc021000-7f3ce0000000 ---p 00000000 00:00 0
7f3ce0000000-7f3ce0021000 rw-p 00000000 00:00 0
7f3ce0021000-7f3ce4000000 ---p 00000000 00:00 0
7f3ce4000000-7f3ce4021000 rw-p 00000000 00:00 0
7f3ce4021000-7f3ce8000000 ---p 00000000 00:00 0
7f3ce8000000-7f3ce8021000 rw-p 00000000 00:00 0
7f3ce8021000-7f3cec000000 ---p 00000000 00:00 0
7f3cec000000-7f3cec021000 rw-p 00000000 00:00 0
7f3cec021000-7f3cf0000000 ---p 00000000 00:00 0
7f3cf0000000-7f3cf0021000 rw-p 00000000 00:00 0
7f3cf0021000-7f3cf4000000 ---p 00000000 00:00 0
7f3cf4000000-7f3cf4021000 rw-p 00000000 00:00 0
7f3cf4021000-7f3cf8000000 ---p 00000000 00:00 0
7f3cf8000000-7f3cf8021000 rw-p 00000000 00:00 0
7f3cf8021000-7f3cfc000000 ---p 00000000 00:00 0
7f3cfc000000-7f3cfc021000 rw-p 00000000 00:00 0
7f3cfc021000-7f3d00000000 ---p 00000000 00:00 0
7f3d00000000-7f3d00021000 rw-p 00000000 00:00 0
7f3d00021000-7f3d04000000 ---p 00000000 00:00 0
7f3d04000000-7f3d04021000 rw-p 00000000 00:00 0
7f3d04021000-7f3d08000000 ---p 00000000 00:00 0
7f3d08000000-7f3d08021000 rw-p 00000000 00:00 0
7f3d08021000-7f3d0c000000 ---p 00000000 00:00 0
7f3d0c3f1000-7f3d0c3f2000 ---p 00000000 00:00 0
7f3d0c3f2000-7f3d0c7f2000 rw-p 00000000 00:00 0
7f3d0c7f2000-7f3d0c7f3000 ---p 00000000 00:00 0
7f3d0c7f3000-7f3d0cbf3000 rw-p 00000000 00:00 0
7f3d0cbf3000-7f3d0cbf4000 ---p 00000000 00:00 0
7f3d0cbf4000-7f3d0cff4000 rw-p 00000000 00:00 0
7f3d0cff4000-7f3d0cff5000 ---p 00000000 00:00 0
7f3d0cff5000-7f3d0d3f5000 rw-p 00000000 00:00 0
7f3d0d3f5000-7f3d0d3f6000 ---p 00000000 00:00 0
7f3d0d3f6000-7f3d0d7f6000 rw-p 00000000 00:00 0
7f3d0d7f6000-7f3d0d7f7000 ---p 00000000 00:00 0
7f3d0d7f7000-7f3d0dbf7000 rw-p 00000000 00:00 0
7f3d0dbf7000-7f3d0dbf8000 ---p 00000000 00:00 0
7f3d0dbf8000-7f3d0dff8000 rw-p 00000000 00:00 0
7f3d0dff8000-7f3d0dff9000 ---p 00000000 00:00 0
7f3d0dff9000-7f3d0e3f9000 rw-p 00000000 00:00 0
7f3d0e3f9000-7f3d0e3fa000 ---p 00000000 00:00 0
7f3d0e3fa000-7f3d0e7fa000 rw-p 00000000 00:00 0
7f3d0e7fa000-7f3d0e7fb000 ---p 00000000 00:00 0
7f3d0e7fb000-7f3d0ebfb000 rw-p 00000000 00:00 0
7f3d0ebfb000-7f3d0ebfc000 ---p 00000000 00:00 0
7f3d0ebfc000-7f3d0effc000 rw-p 00000000 00:00 0
7f3d0effc000-7f3d0effd000 ---p 00000000 00:00 0
7f3d0effd000-7f3d0f3fd000 rw-p 00000000 00:00 0
7f3d0f3fd000-7f3d0f3fe000 ---p 00000000 00:00 0
7f3d0f3fe000-7f3d0f7fe000 rw-p 00000000 00:00 0
7f3d0f7fe000-7f3d0f7ff000 ---p 00000000 00:00 0
7f3d0f7ff000-7f3d0fbff000 rw-p 00000000 00:00 0
7f3d0fbff000-7f3d0fc00000 ---p 00000000 00:00 0
7f3d0fc00000-7f3d10000000 rw-p 00000000 00:00 0
7f3d10000000-7f3d10021000 rw-p 00000000 00:00 0
7f3d10021000-7f3d14000000 ---p 00000000 00:00 0
7f3d14000000-7f3d14021000 rw-p 00000000 00:00 0
7f3d14021000-7f3d18000000 ---p 00000000 00:00 0
7f3d18000000-7f3d18021000 rw-p 00000000 00:00 0
7f3d18021000-7f3d1c000000 ---p 00000000 00:00 0
7f3d1c000000-7f3d1c021000 rw-p 00000000 00:00 0
7f3d1c021000-7f3d20000000 ---p 00000000 00:00 0
7f3d20000000-7f3d20021000 rw-p 00000000 00:00 0
7f3d20021000-7f3d24000000 ---p 00000000 00:00 0
7f3d24000000-7f3d24021000 rw-p 00000000 00:00 0
7f3d24021000-7f3d28000000 ---p 00000000 00:00 0
7f3d28000000-7f3d28491000 rw-p 00000000 00:00 0
7f3d28491000-7f3d2c000000 ---p 00000000 00:00 0
7f3d2c000000-7f3d2c491000 rw-p 00000000 00:00 0
7f3d2c491000-7f3d30000000 ---p 00000000 00:00 0
7f3d30000000-7f3d3036e000 rw-p 00000000 00:00 0
7f3d3036e000-7f3d34000000 ---p 00000000 00:00 0
7f3d34000000-7f3d34491000 rw-p 00000000 00:00 0
7f3d34491000-7f3d38000000 ---p 00000000 00:00 0
7f3d38000000-7f3d3836e000 rw-p 00000000 00:00 0
7f3d3836e000-7f3d3c000000 ---p 00000000 00:00 0
7f3d3c000000-7f3d3c36e000 rw-p 00000000 00:00 0
7f3d3c36e000-7f3d40000000 ---p 00000000 00:00 0
7f3d40000000-7f3d40021000 rw-p 00000000 00:00 0
7f3d40021000-7f3d44000000 ---p 00000000 00:00 0
7f3d443f2000-7f3d443f3000 ---p 00000000 00:00 0
7f3d443f3000-7f3d447f3000 rw-p 00000000 00:00 0
7f3d447f3000-7f3d447f4000 ---p 00000000 00:00 0
7f3d447f4000-7f3d44bf4000 rw-p 00000000 00:00 0
7f3d44bf4000-7f3d44bf5000 ---p 00000000 00:00 0
7f3d44bf5000-7f3d44ff5000 rw-p 00000000 00:00 0
7f3d44ff5000-7f3d44ff6000 ---p 00000000 00:00 0
7f3d44ff6000-7f3d453f6000 rw-p 00000000 00:00 0
7f3d453f6000-7f3d453f7000 ---p 00000000 00:00 0
7f3d453f7000-7f3d457f7000 rw-p 00000000 00:00 0
7f3d457f7000-7f3d457f8000 ---p 00000000 00:00 0
7f3d457f8000-7f3d45bf8000 rw-p 00000000 00:00 0
7f3d45bf8000-7f3d45bf9000 ---p 00000000 00:00 0
7f3d45bf9000-7f3d45ff9000 rw-p 00000000 00:00 0
7f3d45ff9000-7f3d45ffa000 ---p 00000000 00:00 0
7f3d45ffa000-7f3d463fa000 rw-p 00000000 00:00 0
7f3d463fa000-7f3d463fb000 ---p 00000000 00:00 0
7f3d463fb000-7f3d467fb000 rw-p 00000000 00:00 0
7f3d467fb000-7f3d467fc000 ---p 00000000 00:00 0
7f3d467fc000-7f3d46bfc000 rw-p 00000000 00:00 0
7f3d46bfc000-7f3d46bfd000 ---p 00000000 00:00 0
7f3d46bfd000-7f3d46ffd000 rw-p 00000000 00:00 0
7f3d46ffd000-7f3d46ffe000 ---p 00000000 00:00 0
7f3d46ffe000-7f3d473fe000 rw-p 00000000 00:00 0
7f3d473fe000-7f3d473ff000 ---p 00000000 00:00 0
7f3d473ff000-7f3d477ff000 rw-p 00000000 00:00 0
7f3d477ff000-7f3d47800000 ---p 00000000 00:00 0
7f3d47800000-7f3d48000000 rw-p 00000000 00:00 0
7f3d48000000-7f3d80000000 ---p 00000000 00:00 0
7f3d80000000-7f3d80001000 rw-s 00000000 00:06 424 /dev/nvidia0
7f3d80001000-7f3d80002000 rw-s 00000000 00:06 424 /dev/nvidia0
7f3d80002000-7f3d80003000 rw-s 00000000 00:06 424 /dev/nvidia0
7f3d80003000-7f3d80004000 rw-s 00000000 00:06 424 /dev/nvidia0
7f3d80004000-7f3d80005000 rw-s 00000000 00:06 424 /dev/nvidia0
7f3d80005000-7f3d80006000 rw-s 00000000 00:06 424 /dev/nvidia0
7f3d80006000-7f3d80007000 rw-s 00000000 00:06 424 /dev/nvidia0
7f3d80007000-7f3d80008000 rw-s 00000000 00:06 424 /dev/nvidia0
7f3d80008000-7f3d80009000 rw-s 00000000 00:06 424 /dev/nvidia0
7f3d80009000-7f3d8000a000 rw-s 00000000 00:06 424 /dev/nvidia0
7f3d8000a000-7f3d8000b000 rw-s 00000000 00:06 424 /dev/nvidia0
7f3d8000b000-7f3d8000c000 rw-s 00000000 00:06 424 /dev/nvidia0
7f3d8000c000-7f3d8000d000 rw-s 00000000 00:06 424 /dev/nvidia0
7f3d8000d000-7f3d8000e000 rw-s 00000000 00:06 424 /dev/nvidia0
7f3d8000e000-7f3d8000f000 rw-s 00000000 00:06 424 /dev/nvidia0
7f3d8000f000-7f3d80010000 rw-s 00000000 00:06 424 /dev/nvidia0
7f3d80010000-7f3d90000000 ---p 00000000 00:00 0
7f3d903e9000-7f3d90e49000 r-xp 00000000 08:01 24913023 /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.410.48
7f3d90e49000-7f3d91048000 ---p 00a60000 08:01 24913023 /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.410.48
7f3d91048000-7f3d91179000 rw-p 00a5f000 08:01 24913023 /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.410.48
7f3d91179000-7f3d91182000 rw-p 00000000 00:00 0
7f3d91182000-7f3d914b7000 r-xp 00000000 08:01 6950381 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libnvrtc-builtins.so.9.2.148
7f3d914b7000-7f3d916b6000 ---p 00335000 08:01 6950381 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libnvrtc-builtins.so.9.2.148
7f3d916b6000-7f3d916b7000 rw-p 00334000 08:01 6950381 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libnvrtc-builtins.so.9.2.148
7f3d916b7000-7f3d92000000 rw-p 00000000 00:00 0
7f3d92000000-7f3d92200000 ---p 00000000 00:00 0
7f3d92200000-7f3d92400000 rw-s 00000000 00:05 3984996 /dev/zero (deleted)
7f3d92400000-7f3d92600000 rw-s 00000000 00:06 423 /dev/nvidiactl
7f3d92600000-7f3d92800000 rw-s 00000000 00:05 3984997 /dev/zero (deleted)
7f3d92800000-7f3d92a00000 rw-s 00000000 00:06 423 /dev/nvidiactl
7f3d92a00000-7f3d92c00000 ---p 00000000 00:00 0
7f3d92c00000-7f3d92ed6000 rw-s 00000000 00:06 423 /dev/nvidiactl
7f3d92ed6000-7f3d94000000 ---p 00000000 00:00 0
7f3d94000000-7f3d94021000 rw-p 00000000 00:00 0
7f3d94021000-7f3d98000000 ---p 00000000 00:00 0
7f3d982e0000-7f3d987bd000 rw-p 00000000 00:00 0
7f3d987bd000-7f3d987be000 ---p 00000000 00:00 0
7f3d987be000-7f3d98fbe000 rw-p 00000000 00:00 0
7f3d98fbe000-7f3d98fbf000 ---p 00000000 00:00 0
7f3d98fbf000-7f3d997bf000 rw-p 00000000 00:00 0
7f3d997bf000-7f3d9f7bf000 ---p 00000000 00:00 0
7f3d9f7bf000-7f3d9f7c8000 r-xp 00000000 08:01 7210782 /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.1
7f3d9f7c8000-7f3d9f9c7000 ---p 00009000 08:01 7210782 /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.1
7f3d9f9c7000-7f3d9f9c8000 r--p 00008000 08:01 7210782 /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.1
7f3d9f9c8000-7f3d9f9c9000 rw-p 00009000 08:01 7210782 /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.1
7f3d9f9c9000-7f3d9f9d3000 r-xp 00000000 08:01 7210800 /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
7f3d9f9d3000-7f3d9fbd2000 ---p 0000a000 08:01 7210800 /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
7f3d9fbd2000-7f3d9fbd3000 r--p 00009000 08:01 7210800 /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
7f3d9fbd3000-7f3d9fbd4000 rw-p 0000a000 08:01 7210800 /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
7f3d9fbd4000-7f3d9fbd6000 r-xp 00000000 08:01 6950604 /lib/x86_64-linux-gnu/libutil-2.23.so
7f3d9fbd6000-7f3d9fdd5000 ---p 00002000 08:01 6950604 /lib/x86_64-linux-gnu/libutil-2.23.so
7f3d9fdd5000-7f3d9fdd6000 r--p 00001000 08:01 6950604 /lib/x86_64-linux-gnu/libutil-2.23.so
7f3d9fdd6000-7f3d9fdd7000 rw-p 00002000 08:01 6950604 /lib/x86_64-linux-gnu/libutil-2.23.so
7f3d9fdd7000-7f3d9fe10000 r-xp 00000000 08:01 7210745 /usr/lib/x86_64-linux-gnu/libhwloc.so.5.6.8
7f3d9fe10000-7f3da000f000 ---p 00039000 08:01 7210745 /usr/lib/x86_64-linux-gnu/libhwloc.so.5.6.8
7f3da000f000-7f3da0010000 r--p 00038000 08:01 7210745 /usr/lib/x86_64-linux-gnu/libhwloc.so.5.6.8
7f3da0010000-7f3da0011000 rw-p 00039000 08:01 7210745 /usr/lib/x86_64-linux-gnu/libhwloc.so.5.6.8
7f3da0011000-7f3da004e000 r-xp 00000000 08:01 24913022 /usr/lib/x86_64-linux-gnu/libnvidia-fatbinaryloader.so.410.48
7f3da004e000-7f3da024e000 ---p 0003d000 08:01 24913022 /usr/lib/x86_64-linux-gnu/libnvidia-fatbinaryloader.so.410.48
7f3da024e000-7f3da0259000 rw-p 0003d000 08:01 24913022 /usr/lib/x86_64-linux-gnu/libnvidia-fatbinaryloader.so.410.48
7f3da0259000-7f3da025e000 rw-p 00000000 00:00 0
7f3da025e000-7f3da02eb000 r-xp 00000000 08:01 7739355 /usr/lib/x86_64-linux-gnu/libvorbisenc.so.2.0.11
7f3da02eb000-7f3da04ea000 ---p 0008d000 08:01 7739355 /usr/lib/x86_64-linux-gnu/libvorbisenc.so.2.0.11
7f3da04ea000-7f3da0506000 r--p 0008c000 08:01 7739355 /usr/lib/x86_64-linux-gnu/libvorbisenc.so.2.0.11
7f3da0506000-7f3da0507000 rw-p 000a8000 08:01 7739355 /usr/lib/x86_64-linux-gnu/libvorbisenc.so.2.0.11
7f3da0507000-7f3da0531000 r-xp 00000000 08:01 7739350 /usr/lib/x86_64-linux-gnu/libvorbis.so.0.4.8
7f3da0531000-7f3da0730000 ---p 0002a000 08:01 7739350 /usr/lib/x86_64-linux-gnu/libvorbis.so.0.4.8
7f3da0730000-7f3da0731000 r--p 00029000 08:01 7739350 /usr/lib/x86_64-linux-gnu/libvorbis.so.0.4.8
7f3da0731000-7f3da0732000 rw-p 0002a000 08:01 7739350 /usr/lib/x86_64-linux-gnu/libvorbis.so.0.4.8
7f3da0732000-7f3da0739000 r-xp 00000000 08:01 7739313 /usr/lib/x86_64-linux-gnu/libogg.so.0.8.2
7f3da0739000-7f3da0939000 ---p 00007000 08:01 7739313 /usr/lib/x86_64-linux-gnu/libogg.so.0.8.2
7f3da0939000-7f3da093a000 r--p 00007000 08:01 7739313 /usr/lib/x86_64-linux-gnu/libogg.so.0.8.2
7f3da093a000-7f3da093b000 rw-p 00008000 08:01 7739313 /usr/lib/x86_64-linux-gnu/libogg.so.0.8.2
7f3da093b000-7f3da09ae000 r-xp 00000000 08:01 7739102 /usr/lib/x86_64-linux-gnu/libFLAC.so.8.3.0
7f3da09ae000-7f3da0bae000 ---p 00073000 08:01 7739102 /usr/lib/x86_64-linux-gnu/libFLAC.so.8.3.0
7f3da0bae000-7f3da0baf000 r--p 00073000 08:01 7739102 /usr/lib/x86_64-linux-gnu/libFLAC.so.8.3.0
7f3da0baf000-7f3da0bb0000 rw-p 00074000 08:01 7739102 /usr/lib/x86_64-linux-gnu/libFLAC.so.8.3.0
7f3da0bb0000-7f3da0c40000 r-xp 00000000 08:01 7209281 /usr/lib/openmpi/lib/libopen-pal.so.13.0.2
7f3da0c40000-7f3da0e40000 ---p 00090000 08:01 7209281 /usr/lib/openmpi/lib/libopen-pal.so.13.0.2
7f3da0e40000-7f3da0e44000 r--p 00090000 08:01 7209281 /usr/lib/openmpi/lib/libopen-pal.so.13.0.2
7f3da0e44000-7f3da0e48000 rw-p 00094000 08:01 7209281 /usr/lib/openmpi/lib/libopen-pal.so.13.0.2
7f3da0e48000-7f3da0e4d000 rw-p 00000000 00:00 0
7f3da0e4d000-7f3da0ec2000 r-xp 00000000 08:01 7209283 /usr/lib/openmpi/lib/libopen-rte.so.12.0.2
7f3da0ec2000-7f3da10c1000 ---p 00075000 08:01 7209283 /usr/lib/openmpi/lib/libopen-rte.so.12.0.2
7f3da10c1000-7f3da10c2000 r--p 00074000 08:01 7209283 /usr/lib/openmpi/lib/libopen-rte.so.12.0.2
7f3da10c2000-7f3da10c5000 rw-p 00075000 08:01 7209283 /usr/lib/openmpi/lib/libopen-rte.so.12.0.2
7f3da10c5000-7f3da10c7000 rw-p 00000000 00:00 0
7f3da10c7000-7f3da10d4000 r-xp 00000000 08:01 7082433 /usr/lib/libibverbs.so.1.0.0
7f3da10d4000-7f3da12d4000 ---p 0000d000 08:01 7082433 /usr/lib/libibverbs.so.1.0.0
7f3da12d4000-7f3da12d5000 r--p 0000d000 08:01 7082433 /usr/lib/libibverbs.so.1.0.0
7f3da12d5000-7f3da12d6000 rw-p 0000e000 08:01 7082433 /usr/lib/libibverbs.so.1.0.0
7f3da12d6000-7f3da2052000 r-xp 00000000 08:01 24913017 /usr/lib/x86_64-linux-gnu/libcuda.so.410.48
7f3da2052000-7f3da2251000 ---p 00d7c000 08:01 24913017 /usr/lib/x86_64-linux-gnu/libcuda.so.410.48
7f3da2251000-7f3da23bc000 rw-p 00d7b000 08:01 24913017 /usr/lib/x86_64-linux-gnu/libcuda.so.410.48
7f3da23bc000-7f3da23cc000 rw-p 00000000 00:00 0
7f3da23cc000-7f3da64bd000 r-xp 00000000 08:01 6950346 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libcusparse.so.9.2.148
7f3da64bd000-7f3da66bc000 ---p 040f1000 08:01 6950346 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libcusparse.so.9.2.148
7f3da66bc000-7f3da66d6000 rw-p 040f0000 08:01 6950346 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libcusparse.so.9.2.148
7f3da66d6000-7f3da66e7000 rw-p 00000000 00:00 0
7f3da66e7000-7f3dad8b0000 r-xp 00000000 08:01 6950344 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libcusolver.so.9.2.148
7f3dad8b0000-7f3dadaaf000 ---p 071c9000 08:01 6950344 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libcusolver.so.9.2.148
7f3dadaaf000-7f3dadaef000 rw-p 071c8000 08:01 6950344 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libcusolver.so.9.2.148
7f3dadaef000-7f3dadb28000 rw-p 00000000 00:00 0
7f3dadb28000-7f3db2f28000 r-xp 00000000 08:01 6950338 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libcufft.so.9.2.148
7f3db2f28000-7f3db3127000 ---p 05400000 08:01 6950338 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libcufft.so.9.2.148
7f3db3127000-7f3db3135000 rw-p 053ff000 08:01 6950338 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libcufft.so.9.2.148
7f3db3135000-7f3db3182000 rw-p 00000000 00:00 0
7f3db3182000-7f3db6d03000 r-xp 00000000 08:01 6950336 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libcublas.so.9.2.174
7f3db6d03000-7f3db6f03000 ---p 03b81000 08:01 6950336 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libcublas.so.9.2.174
7f3db6f03000-7f3db6f3f000 rw-p 03b81000 08:01 6950336 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libcublas.so.9.2.174
7f3db6f3f000-7f3db6f50000 rw-p 00000000 00:00 0
7f3db6f50000-7f3db80f1000 r-xp 00000000 08:01 6950383 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libnvrtc.so.9.2.148
7f3db80f1000-7f3db82f0000 ---p 011a1000 08:01 6950383 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libnvrtc.so.9.2.148
7f3db82f0000-7f3db84a5000 rw-p 011a0000 08:01 6950383 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libnvrtc.so.9.2.148
7f3db84a5000-7f3db855d000 rw-p 00000000 00:00 0
7f3db855d000-7f3db8569000 r-xp 00000000 08:01 7739345 /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1
7f3db8569000-7f3db8768000 ---p 0000c000 08:01 7739345 /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1
7f3db8768000-7f3db8769000 r--p 0000b000 08:01 7739345 /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1
7f3db8769000-7f3db876a000 rw-p 0000c000 08:01 7739345 /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1
7f3db876a000-7f3db8778000 rw-p 00000000 00:00 0
7f3db8778000-7f3db8799000 r-xp 00000000 08:01 6953390 /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0
7f3db8799000-7f3db8998000 ---p 00021000 08:01 6953390 /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0
7f3db8998000-7f3db8999000 r--p 00020000 08:01 6953390 /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0
7f3db8999000-7f3db899a000 rw-p 00021000 08:01 6953390 /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0
7f3db899a000-7f3db8b5a000 r-xp 00000000 08:01 6950568 /lib/x86_64-linux-gnu/libc-2.23.so
7f3db8b5a000-7f3db8d5a000 ---p 001c0000 08:01 6950568 /lib/x86_64-linux-gnu/libc-2.23.so
7f3db8d5a000-7f3db8d5e000 r--p 001c0000 08:01 6950568 /lib/x86_64-linux-gnu/libc-2.23.so
7f3db8d5e000-7f3db8d60000 rw-p 001c4000 08:01 6950568 /lib/x86_64-linux-gnu/libc-2.23.so
7f3db8d60000-7f3db8d64000 rw-p 00000000 00:00 0
7f3db8d64000-7f3db8d7a000 r-xp 00000000 08:01 6817161 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f3db8d7a000-7f3db8f79000 ---p 00016000 08:01 6817161 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f3db8f79000-7f3db8f7a000 rw-p 00015000 08:01 6817161 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f3db8f7a000-7f3db9082000 r-xp 00000000 08:01 6950576 /lib/x86_64-linux-gnu/libm-2.23.so
7f3db9082000-7f3db9281000 ---p 00108000 08:01 6950576 /lib/x86_64-linux-gnu/libm-2.23.so
7f3db9281000-7f3db9282000 r--p 00107000 08:01 6950576 /lib/x86_64-linux-gnu/libm-2.23.so
7f3db9282000-7f3db9283000 rw-p 00108000 08:01 6950576 /lib/x86_64-linux-gnu/libm-2.23.so
7f3db9283000-7f3db93f5000 r-xp 00000000 08:01 6818043 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f3db93f5000-7f3db95f5000 ---p 00172000 08:01 6818043 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f3db95f5000-7f3db95ff000 r--p 00172000 08:01 6818043 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f3db95ff000-7f3db9601000 rw-p 0017c000 08:01 6818043 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f3db9601000-7f3db9605000 rw-p 00000000 00:00 0
7f3db9605000-7f3db967b000 r-xp 00000000 08:01 7739448 /usr/local/lib/libsndfile.so.1.0.29
7f3db967b000-7f3db987a000 ---p 00076000 08:01 7739448 /usr/local/lib/libsndfile.so.1.0.29
7f3db987a000-7f3db987d000 r--p 00075000 08:01 7739448 /usr/local/lib/libsndfile.so.1.0.29
7f3db987d000-7f3db987e000 rw-p 00078000 08:01 7739448 /usr/local/lib/libsndfile.so.1.0.29
7f3db987e000-7f3db9880000 rw-p 00000000 00:00 0
7f3db9880000-7f3db993e000 r-xp 00000000 08:01 7210802 /usr/lib/x86_64-linux-gnu/libomp.so.5
7f3db993e000-7f3db9b3d000 ---p 000be000 08:01 7210802 /usr/lib/x86_64-linux-gnu/libomp.so.5
7f3db9b3d000-7f3db9b3e000 r--p 000bd000 08:01 7210802 /usr/lib/x86_64-linux-gnu/libomp.so.5
7f3db9b3e000-7f3db9b43000 rw-p 000be000 08:01 7210802 /usr/lib/x86_64-linux-gnu/libomp.so.5
7f3db9b43000-7f3db9b4c000 rw-p 00000000 00:00 0
7f3db9b4c000-7f3dbd950000 r-xp 00000000 08:01 7480105 /opt/intel/compilers_and_libraries_2018.5.274/linux/mkl/lib/intel64_lin/libmkl_core.so
7f3dbd950000-7f3dbdb4f000 ---p 03e04000 08:01 7480105 /opt/intel/compilers_and_libraries_2018.5.274/linux/mkl/lib/intel64_lin/libmkl_core.so
7f3dbdb4f000-7f3dbdb56000 r--p 03e03000 08:01 7480105 /opt/intel/compilers_and_libraries_2018.5.274/linux/mkl/lib/intel64_lin/libmkl_core.so
7f3dbdb56000-7f3dbdb84000 rw-p 03e0a000 08:01 7480105 /opt/intel/compilers_and_libraries_2018.5.274/linux/mkl/lib/intel64_lin/libmkl_core.so
7f3dbdb84000-7f3dbdb99000 rw-p 00000000 00:00 0
7f3dbdb99000-7f3dbf110000 r-xp 00000000 08:01 7480112 /opt/intel/compilers_and_libraries_2018.5.274/linux/mkl/lib/intel64_lin/libmkl_gnu_thread.so
7f3dbf110000-7f3dbf30f000 ---p 01577000 08:01 7480112 /opt/intel/compilers_and_libraries_2018.5.274/linux/mkl/lib/intel64_lin/libmkl_gnu_thread.so
7f3dbf30f000-7f3dbf313000 r--p 01576000 08:01 7480112 /opt/intel/compilers_and_libraries_2018.5.274/linux/mkl/lib/intel64_lin/libmkl_gnu_thread.so
7f3dbf313000-7f3dbf32a000 rw-p 0157a000 08:01 7480112 /opt/intel/compilers_and_libraries_2018.5.274/linux/mkl/lib/intel64_lin/libmkl_gnu_thread.so
7f3dbf32a000-7f3dbfc19000 r-xp 00000000 08:01 7480110 /opt/intel/compilers_and_libraries_2018.5.274/linux/mkl/lib/intel64_lin/libmkl_gf_lp64.so
7f3dbfc19000-7f3dbfe19000 ---p 008ef000 08:01 7480110 /opt/intel/compilers_and_libraries_2018.5.274/linux/mkl/lib/intel64_lin/libmkl_gf_lp64.so
7f3dbfe19000-7f3dbfe1a000 r--p 008ef000 08:01 7480110 /opt/intel/compilers_and_libraries_2018.5.274/linux/mkl/lib/intel64_lin/libmkl_gf_lp64.so
7f3dbfe1a000-7f3dbfe2c000 rw-p 008f0000 08:01 7480110 /opt/intel/compilers_and_libraries_2018.5.274/linux/mkl/lib/intel64_lin/libmkl_gf_lp64.so
7f3dbfe2c000-7f3dbfe32000 rw-p 00000000 00:00 0
7f3dbfe32000-7f3dbfe39000 r-xp 00000000 08:01 6950600 /lib/x86_64-linux-gnu/librt-2.23.so
7f3dbfe39000-7f3dc0038000 ---p 00007000 08:01 6950600 /lib/x86_64-linux-gnu/librt-2.23.so
7f3dc0038000-7f3dc0039000 r--p 00006000 08:01 6950600 /lib/x86_64-linux-gnu/librt-2.23.so
7f3dc0039000-7f3dc003a000 rw-p 00007000 08:01 6950600 /lib/x86_64-linux-gnu/librt-2.23.so
7f3dc003a000-7f3dc003d000 r-xp 00000000 08:01 6950574 /lib/x86_64-linux-gnu/libdl-2.23.so
7f3dc003d000-7f3dc023c000 ---p 00003000 08:01 6950574 /lib/x86_64-linux-gnu/libdl-2.23.so
7f3dc023c000-7f3dc023d000 r--p 00002000 08:01 6950574 /lib/x86_64-linux-gnu/libdl-2.23.so
7f3dc023d000-7f3dc023e000 rw-p 00003000 08:01 6950574 /lib/x86_64-linux-gnu/libdl-2.23.so
7f3dc023e000-7f3dc0256000 r-xp 00000000 08:01 6950596 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f3dc0256000-7f3dc0455000 ---p 00018000 08:01 6950596 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f3dc0455000-7f3dc0456000 r--p 00017000 08:01 6950596 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f3dc0456000-7f3dc0457000 rw-p 00018000 08:01 6950596 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f3dc0457000-7f3dc045b000 rw-p 00000000 00:00 0
7f3dc045b000-7f3dd274b000 r-xp 00000000 08:01 7081452 /usr/lib/x86_64-linux-gnu/libcudnn.so.7.4.2
7f3dd274b000-7f3dd294a000 ---p 122f0000 08:01 7081452 /usr/lib/x86_64-linux-gnu/libcudnn.so.7.4.2
7f3dd294a000-7f3dd29c1000 rw-p 122ef000 08:01 7081452 /usr/lib/x86_64-linux-gnu/libcudnn.so.7.4.2
7f3dd29c1000-7f3dd2a3d000 rw-p 00000000 00:00 0
7f3dd2a3d000-7f3dd2aef000 r-xp 00000000 08:01 7209269 /usr/lib/openmpi/lib/libmpi.so.12.0.2
7f3dd2aef000-7f3dd2cef000 ---p 000b2000 08:01 7209269 /usr/lib/openmpi/lib/libmpi.so.12.0.2
7f3dd2cef000-7f3dd2cf0000 r--p 000b2000 08:01 7209269 /usr/lib/openmpi/lib/libmpi.so.12.0.2
7f3dd2cf0000-7f3dd2d00000 rw-p 000b3000 08:01 7209269 /usr/lib/openmpi/lib/libmpi.so.12.0.2
7f3dd2d00000-7f3dd2d13000 rw-p 00000000 00:00 0
7f3dd2d13000-7f3dd2d2b000 r-xp 00000000 08:01 7209271 /usr/lib/openmpi/lib/libmpi_cxx.so.1.1.3
7f3dd2d2b000-7f3dd2f2b000 ---p 00018000 08:01 7209271 /usr/lib/openmpi/lib/libmpi_cxx.so.1.1.3
7f3dd2f2b000-7f3dd2f2d000 r--p 00018000 08:01 7209271 /usr/lib/openmpi/lib/libmpi_cxx.so.1.1.3
7f3dd2f2d000-7f3dd2f2e000 rw-p 0001a000 08:01 7209271 /usr/lib/openmpi/lib/libmpi_cxx.so.1.1.3
7f3dd2f2e000-7f3ddaddc000 r-xp 00000000 08:01 6950322 /usr/lib/x86_64-linux-gnu/libnccl.so.2.4.2
7f3ddaddc000-7f3ddafdb000 ---p 07eae000 08:01 6950322 /usr/lib/x86_64-linux-gnu/libnccl.so.2.4.2
7f3ddafdb000-7f3ddafe2000 rw-p 07ead000 08:01 6950322 /usr/lib/x86_64-linux-gnu/libnccl.so.2.4.2
7f3ddafe2000-7f3ddafea000 rw-p 00000000 00:00 0
7f3ddafea000-7f3e01292000 r-xp 00000000 08:01 7211001 /usr/local/lib/libafcuda.so.3.6.4
7f3e01292000-7f3e01491000 ---p 262a8000 08:01 7211001 /usr/local/lib/libafcuda.so.3.6.4
7f3e01491000-7f3e014a0000 r--p 262a7000 08:01 7211001 /usr/local/lib/libafcuda.so.3.6.4
7f3e014a0000-7f3e014a4000 rw-p 262b6000 08:01 7211001 /usr/local/lib/libafcuda.so.3.6.4
7f3e014a4000-7f3e014d5000 rw-p 00000000 00:00 0
7f3e014d5000-7f3e014f5000 r-xp 00000000 08:01 7739272 /usr/lib/x86_64-linux-gnu/libgflags.so.2.1.2
7f3e014f5000-7f3e016f4000 ---p 00020000 08:01 7739272 /usr/lib/x86_64-linux-gnu/libgflags.so.2.1.2
7f3e016f4000-7f3e016f5000 r--p 0001f000 08:01 7739272 /usr/lib/x86_64-linux-gnu/libgflags.so.2.1.2
7f3e016f5000-7f3e016f6000 rw-p 00020000 08:01 7739272 /usr/lib/x86_64-linux-gnu/libgflags.so.2.1.2
7f3e016f6000-7f3e01714000 r-xp 00000000 08:01 7739282 /usr/lib/x86_64-linux-gnu/libglog.so.0.0.0
7f3e01714000-7f3e01913000 ---p 0001e000 08:01 7739282 /usr/lib/x86_64-linux-gnu/libglog.so.0.0.0
7f3e01913000-7f3e01914000 r--p 0001d000 08:01 7739282 /usr/lib/x86_64-linux-gnu/libglog.so.0.0.0
7f3e01914000-7f3e01915000 rw-p 0001e000 08:01 7739282 /usr/lib/x86_64-linux-gnu/libglog.so.0.0.0
7f3e01915000-7f3e01925000 rw-p 00000000 00:00 0
7f3e01925000-7f3e0193e000 r-xp 00000000 08:01 6817239 /lib/x86_64-linux-gnu/libz.so.1.2.8
7f3e0193e000-7f3e01b3d000 ---p 00019000 08:01 6817239 /lib/x86_64-linux-gnu/libz.so.1.2.8
7f3e01b3d000-7f3e01b3e000 r--p 00018000 08:01 6817239 /lib/x86_64-linux-gnu/libz.so.1.2.8
7f3e01b3e000-7f3e01b3f000 rw-p 00019000 08:01 6817239 /lib/x86_64-linux-gnu/libz.so.1.2.8
7f3e01b3f000-7f3e01b4e000 r-xp 00000000 08:01 6817139 /lib/x86_64-linux-gnu/libbz2.so.1.0.4
7f3e01b4e000-7f3e01d4d000 ---p 0000f000 08:01 6817139 /lib/x86_64-linux-gnu/libbz2.so.1.0.4
7f3e01d4d000-7f3e01d4e000 r--p 0000e000 08:01 6817139 /lib/x86_64-linux-gnu/libbz2.so.1.0.4
7f3e01d4e000-7f3e01d4f000 rw-p 0000f000 08:01 6817139 /lib/x86_64-linux-gnu/libbz2.so.1.0.4
7f3e01d4f000-7f3e01d70000 r-xp 00000000 08:01 6817171 /lib/x86_64-linux-gnu/liblzma.so.5.0.0
7f3e01d70000-7f3e01f6f000 ---p 00021000 08:01 6817171 /lib/x86_64-linux-gnu/liblzma.so.5.0.0
7f3e01f6f000-7f3e01f70000 r--p 00020000 08:01 6817171 /lib/x86_64-linux-gnu/liblzma.so.5.0.0
7f3e01f70000-7f3e01f71000 rw-p 00021000 08:01 6817171 /lib/x86_64-linux-gnu/liblzma.so.5.0.0
7f3e01f71000-7f3e0243b000 r-xp 00000000 08:01 8391456 /root/wav2letter/src/third_party/warpctc/build/libwarpctc.so
7f3e0243b000-7f3e0263b000 ---p 004ca000 08:01 8391456 /root/wav2letter/src/third_party/warpctc/build/libwarpctc.so
7f3e0263b000-7f3e0263e000 r--p 004ca000 08:01 8391456 /root/wav2letter/src/third_party/warpctc/build/libwarpctc.so
7f3e0263e000-7f3e0263f000 rw-p 004cd000 08:01 8391456 /root/wav2letter/src/third_party/warpctc/build/libwarpctc.so
7f3e0263f000-7f3e02640000 rw-p 00000000 00:00 0
7f3e02640000-7f3e02666000 r-xp 00000000 08:01 6950561 /lib/x86_64-linux-gnu/ld-2.23.so
7f3e02679000-7f3e0267a000 rw-p 00000000 00:00 0
7f3e0267a000-7f3e0267b000 ---p 00000000 00:00 0
7f3e0267b000-7f3e026c4000 rw-p 00000000 00:00 0
7f3e026c4000-7f3e026c5000 rw-s 00000000 00:06 423 /dev/nvidiactl
7f3e026c5000-7f3e026c6000 rw-s 00000000 00:06 423 /dev/nvidiactl
7f3e026c6000-7f3e026c7000 rw-s 00000000 00:06 423 /dev/nvidiactl
7f3e026c7000-7f3e02856000 rw-p 00000000 00:00 0
7f3e02856000-7f3e02857000 rw-s 00000000 00:06 423 /dev/nvidiactl
7f3e02857000-7f3e02858000 rw-s 00000000 00:06 423 /dev/nvidiactl
7f3e02858000-7f3e02859000 rw-s 00000000 00:06 423 /dev/nvidiactl
7f3e02859000-7f3e0285a000 rw-s 00000000 00:06 423 /dev/nvidiactl
7f3e0285a000-7f3e0285b000 rw-s 00000000 00:06 423 /dev/nvidiactl
7f3e0285b000-7f3e0285c000 rw-s 00000000 00:06 423 /dev/nvidiactl
7f3e0285c000-7f3e0285d000 rw-s 00000000 00:06 423 /dev/nvidiactl
7f3e0285d000-7f3e0285e000 rw-s 00000000 00:06 423 /dev/nvidiactl
7f3e0285e000-7f3e0285f000 rw-s 00000000 00:06 423 /dev/nvidiactl
7f3e0285f000-7f3e02860000 rw-s 00000000 00:06 423 /dev/nvidiactl
7f3e02860000-7f3e02861000 rw-s 00000000 00:06 423 /dev/nvidiactl
7f3e02861000-7f3e02862000 rw-s 00000000 00:06 423 /dev/nvidiactl
7f3e02862000-7f3e02863000 rw-s 00000000 00:06 423 /dev/nvidiactl
7f3e02863000-7f3e02864000 r--s 00000000 00:06 424 /dev/nvidia0
7f3e02864000-7f3e02865000 rw-p 00000000 00:00 0
7f3e02865000-7f3e02866000 r--p 00025000 08:01 6950561 /lib/x86_64-linux-gnu/ld-2.23.so
7f3e02866000-7f3e02867000 rw-p 00026000 08:01 6950561 /lib/x86_64-linux-gnu/ld-2.23.so
7f3e02867000-7f3e02868000 rw-p 00000000 00:00 0
7ffdca479000-7ffdca49a000 rw-p 00000000 00:00 0 [stack]
7ffdca5a9000-7ffdca5ac000 r--p 00000000 00:00 0 [vvar]
7ffdca5ac000-7ffdca5ae000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
 Aborted at 1565873911 (unix time) try "date -d @1565873911" if you are using GNU date
PC: @ 0x7f3db89cf428 gsignal
 SIGABRT (@0x461) received by PID 1121 (TID 0x7f3e02842600) from PID 1121; stack trace: **
@ 0x7f3dc024f390 (unknown)
@ 0x7f3db89cf428 gsignal
@ 0x7f3db89d102a abort
@ 0x7f3db8a117ea (unknown)
@ 0x7f3db8a1a37a (unknown)
@ 0x7f3db8a1e53c cfree
@ 0x7f3dc0dc0742 (unknown)
@ 0x7f3dc0e6a04b (unknown)
@ 0x7f3dc0e6ecae (unknown)
@ 0x7f3dc050282c (unknown)
@ 0x7f3dc0502fd8 cudnnConvolutionBackwardFilter
@ 0x6a1742 _ZZN2fl6conv2dERKNS_8VariableES2_S2_iiiiiiiENKUlRSt6vectorIS0_SaIS0_EES2_E_clES6_S2_
@ 0x652674 fl::Variable::backward()
@ 0x652bf3 fl::Variable::backward()
@ 0x47dcb3 _ZZ4mainENKUlSt10shared_ptrIN2fl6ModuleEES_IN3w2l17SequenceCriterionEES_INS3_10W2lDatasetEES_INS0_19FirstOrderOptimizerEES9_ddbiE3_clES2_S5_S7_S9_S9_ddbi.constprop.11263
@ 0x41ab54 main
@ 0x7f3db89ba830 __libc_start_main
@ 0x478d59 _start
@ 0x0 (unknown)
Aborted (core dumped)`

picture GaofengCheng

All 6 comments

Hi @GaofengCheng,

This is known issue, which happens only for WSJ training. Please have a look at https://github.com/facebookresearch/wav2letter/issues/146. It is a problem inside cudnn. They have fixed the issue, wait for a new release of cudnn where this will the fix will be provided.

For now, you could fix this in the following way inside docker:

vim /root/flashlight/flashlight/autograd/backend/cuda/Conv2D.cpp

insert instead of rows 354-374 here https://github.com/facebookresearch/flashlight/blob/42800fa666fe259ead405dd8699f0b4091fbb5ad/flashlight/autograd/backend/cuda/Conv2D.cpp#L354 the following code

      cudnnConvolutionBwdFilterAlgoPerf_t bwdFilterAlgoBestPerf;
      bwdFilterAlgoBestPerf.algo = kBwdFilterDefaultAlgo;
      CUDNN_CHECK_ERR(cudnnGetConvolutionBackwardFilterWorkspaceSize(
          hndl,
          iDesc.descriptor,
          oDesc.descriptor,
          cDesc.descriptor,
          wDesc.descriptor,
          bwdFilterAlgoBestPerf.algo,
          &bwdFilterAlgoBestPerf.memory));
      ws = af::array(bwdFilterAlgoBestPerf.memory, af::dtype::b8);

then run

cd /root/flashlight/build && make -j8 && make install
cd /root/wav2letter/build && make -j8
tlikhomanenko picture tlikhomanenko on 16 Aug 2019
👍1

@tlikhomanenko Thanks for your quick reply.
Following your guide, I commented ling 354-374, and inserted your codes.
But it fialed building as follows:

[  2%] Building CXX object CMakeFiles/flashlight.dir/flashlight/autograd/backend/cuda/Conv2D.cpp.o
/root/flashlight/flashlight/autograd/backend/cuda/Conv2D.cpp: In lambda function:
/root/flashlight/flashlight/autograd/backend/cuda/Conv2D.cpp:385:7: error: 'ws' was not declared in this scope
       ws = af::array(bwdFilterAlgoBestPerf.memory, af::dtype::b8);

It seems the ws is not defined, I found line 359 af::array ws; is commented.

GaofengCheng picture GaofengCheng on 16 Aug 2019
👍1

It works by adding

      af::array ws;
      cudnnConvolutionBwdFilterAlgoPerf_t bwdFilterAlgoBestPerf;
      bwdFilterAlgoBestPerf.algo = kBwdFilterDefaultAlgo;
      CUDNN_CHECK_ERR(cudnnGetConvolutionBackwardFilterWorkspaceSize(
          hndl,
          iDesc.descriptor,
          oDesc.descriptor,
          cDesc.descriptor,
          wDesc.descriptor,
          bwdFilterAlgoBestPerf.algo,
          &bwdFilterAlgoBestPerf.memory));
      ws = af::array(bwdFilterAlgoBestPerf.memory, af::dtype::b8);

waiting for the final results, thanks!

GaofengCheng picture GaofengCheng on 16 Aug 2019

I encountered the same problem but NOT for WSJ training, the above solution did not work for me. In addition, I followed these two steps which solved my problem:

  1. reinstalled cudnn v7.6.5
  2. re-make flashlight and wav2letter packages
00001101-xt picture 00001101-xt on 6 Nov 2019
🎉1

Hi @00001101-xt,

The bug we reported to NVIDIA (details see https://github.com/facebookresearch/wav2letter/issues/146) was solved in cudnn 7.6.3, so I could expect they also fixed something else too =).

Thanks for confirming that newest version of cudnn solves the issue!

tlikhomanenko picture tlikhomanenko on 6 Nov 2019

I encountered the same problem but NOT for WSJ training, the above solution did not work for me. In addition, I followed these two steps which solved my problem:

  1. reinstalled cudnn v7.6.5
  2. re-make flashlight and wav2letter packages

@tlikhomanenko However, the problem persists when training in distribution mode.

00001101-xt picture 00001101-xt on 8 Nov 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

JanX2 picture JanX2  路  5Comments
hajix picture hajix  路  4Comments
zhengqun picture zhengqun  路  5Comments
nutriver picture nutriver  路  3Comments
abhinavkulkarni picture abhinavkulkarni  路  3Comments