Wav2letter: Error in `build/Train': double free or corruption. Aborted (core dumped)

[smolendawid]

I have the following error.

user@server:~/wav2letter$ build/Train train --flagsfile=/home/user/wav2letter/recipes/name/configs/conv_glu/train.cfg
* Error in `build/Train': double free or corruption (!prev): 0x000000004da56e70 *
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f512072d7e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7f512073637a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f512073a53c]
/usr/lib/x86_64-linux-gnu/libcudnn.so.7(+0x965742)[0x7f5164b04742]
/usr/lib/x86_64-linux-gnu/libcudnn.so.7(+0xa0f04b)[0x7f5164bae04b]
/usr/lib/x86_64-linux-gnu/libcudnn.so.7(+0xa13cae)[0x7f5164bb2cae]
/usr/lib/x86_64-linux-gnu/libcudnn.so.7(+0xa782c)[0x7f516424682c]
/usr/lib/x86_64-linux-gnu/libcudnn.so.7(cudnnConvolutionBackwardFilter+0x678)[0x7f5164246fd8]
build/Train[0x6894e7]
build/Train[0x649cf4]
build/Train[0x64a273]
build/Train[0x455fd9]
build/Train[0x4135eb]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f51206d6830]
build/Train[0x451e19]
======= Memory map: ========
00400000-007a5000 r-xp 00000000 fc:00 5646164 /home/user/wav2letter/build/Train
009a5000-009a6000 r--p 003a5000 fc:00 5646164 /home/user/wav2letter/build/Train
009a6000-009ad000 rw-p 003a6000 fc:00 5646164 /home/user/wav2letter/build/Train
009ad000-009c7000 rw-p 00000000 00:00 0
00f84000-50e17000 rw-p 00000000 00:00 0 [heap]
200000000-200200000 rw-s 00000000 00:06 459 /dev/nvidiactl
200200000-200400000 ---p 00000000 00:00 0
200400000-200404000 rw-s 00000000 00:06 459 /dev/nvidiactl
200404000-200600000 ---p 00000000 00:00 0
200600000-200a00000 rw-s 00000000 00:06 459 /dev/nvidiactl
200a00000-201600000 ---p 00000000 00:00 0
201600000-201604000 rw-s 00000000 00:06 459 /dev/nvidiactl
201604000-201800000 ---p 00000000 00:00 0
201800000-201c00000 rw-s 00000000 00:06 459 /dev/nvidiactl
201c00000-201c04000 rw-s 00000000 00:06 459 /dev/nvidiactl
201c04000-201e00000 ---p 00000000 00:00 0
201e00000-202200000 rw-s 00000000 00:06 459 /dev/nvidiactl
202200000-202204000 rw-s 00000000 00:06 459 /dev/nvidiactl
202204000-202400000 ---p 00000000 00:00 0
202400000-202800000 rw-s 00000000 00:06 459 /dev/nvidiactl
202800000-202804000 rw-s 00000000 00:06 459 /dev/nvidiactl
202804000-202a00000 ---p 00000000 00:00 0
202a00000-202e00000 rw-s 00000000 00:06 459 /dev/nvidiactl
202e00000-202e04000 rw-s 00000000 00:06 459 /dev/nvidiactl
202e04000-203000000 ---p 00000000 00:00 0
203000000-203400000 rw-s 00000000 00:06 459 /dev/nvidiactl
203400000-203404000 rw-s 00000000 00:06 459 /dev/nvidiactl
203404000-203600000 ---p 00000000 00:00 0
203600000-203a00000 rw-s 00000000 00:06 459 /dev/nvidiactl
203a00000-203a04000 rw-s 00000000 00:06 459 /dev/nvidiactl
203a04000-203c00000 ---p 00000000 00:00 0
203c00000-204000000 rw-s 00000000 00:06 459 /dev/nvidiactl
204000000-204004000 rw-s 00000000 00:06 459 /dev/nvidiactl
204004000-204200000 ---p 00000000 00:00 0
204200000-204600000 rw-s 00000000 00:06 459 /dev/nvidiactl
204600000-204604000 rw-s 00000000 00:06 459 /dev/nvidiactl
204604000-204800000 ---p 00000000 00:00 0
204800000-204c00000 rw-s 00000000 00:06 459 /dev/nvidiactl
204c00000-204c04000 rw-s 00000000 00:06 459 /dev/nvidiactl
204c04000-204e00000 ---p 00000000 00:00 0
204e00000-205200000 rw-s 00000000 00:06 459 /dev/nvidiactl
205200000-205204000 rw-s 00000000 00:06 459 /dev/nvidiactl
205204000-205400000 ---p 00000000 00:00 0
205400000-205800000 rw-s 00000000 00:06 459 /dev/nvidiactl
205800000-205804000 rw-s 00000000 00:06 459 /dev/nvidiactl
205804000-205a00000 ---p 00000000 00:00 0
205a00000-205e00000 rw-s 00000000 00:06 459 /dev/nvidiactl
205e00000-205e04000 rw-s 00000000 00:06 459 /dev/nvidiactl
205e04000-206000000 ---p 00000000 00:00 0
206000000-206400000 rw-s 00000000 00:06 459 /dev/nvidiactl
206400000-206404000 rw-s 00000000 00:06 459 /dev/nvidiactl
206404000-206600000 ---p 00000000 00:00 0
206600000-206a00000 rw-s 00000000 00:06 459 /dev/nvidiactl
206a00000-206a04000 rw-s 00000000 00:06 459 /dev/nvidiactl
206a04000-206c00000 ---p 00000000 00:00 0
206c00000-207000000 rw-s 00000000 00:06 459 /dev/nvidiactl
207000000-207200000 ---p 00000000 00:00 0
207200000-207400000 rw-s 00000000 00:06 459 /dev/nvidiactl
207400000-500200000 ---p 00000000 00:00 0
10000000000-10404000000 ---p 00000000 00:00 0
7f4fa4000000-7f5004000000 ---p 00000000 00:00 0
7f5004000000-7f5004067000 rw-p 00000000 00:00 0
7f5004067000-7f5008000000 ---p 00000000 00:00 0
7f500a000000-7f500c000000 ---p 00000000 00:00 0
7f500c000000-7f500c076000 rw-p 00000000 00:00 0
7f500c076000-7f5010000000 ---p 00000000 00:00 0
7f5010000000-7f5010068000 rw-p 00000000 00:00 0
7f5010068000-7f5014000000 ---p 00000000 00:00 0
7f5014000000-7f5014068000 rw-p 00000000 00:00 0
7f5014068000-7f5018000000 ---p 00000000 00:00 0
7f5018000000-7f5018068000 rw-p 00000000 00:00 0
7f5018068000-7f501c000000 ---p 00000000 00:00 0
7f501c000000-7f501c068000 rw-p 00000000 00:00 0
7f501c068000-7f5020000000 ---p 00000000 00:00 0
7f5020000000-7f5020068000 rw-p 00000000 00:00 0
7f5020068000-7f5024000000 ---p 00000000 00:00 0
7f5024000000-7f5024052000 rw-p 00000000 00:00 0
7f5024052000-7f5028000000 ---p 00000000 00:00 0
7f5028000000-7f5028052000 rw-p 00000000 00:00 0
7f5028052000-7f502c000000 ---p 00000000 00:00 0
7f502c000000-7f502c052000 rw-p 00000000 00:00 0
7f502c052000-7f5030000000 ---p 00000000 00:00 0
7f5030000000-7f5030052000 rw-p 00000000 00:00 0
7f5030052000-7f5034000000 ---p 00000000 00:00 0
7f5034000000-7f5034052000 rw-p 00000000 00:00 0
7f5034052000-7f5038000000 ---p 00000000 00:00 0
7f5038000000-7f5038051000 rw-p 00000000 00:00 0
7f5038051000-7f503c000000 ---p 00000000 00:00 0
7f503e000000-7f503f400000 ---p 00000000 00:00 0
7f503f400000-7f503f600000 rw-s 00000000 00:05 36678 /dev/zero (deleted)
7f503f600000-7f503f800000 ---p 00000000 00:00 0
7f503f800000-7f503fa00000 rw-s 00000000 00:05 36679 /dev/zero (deleted)
7f503fa00000-7f5040000000 ---p 00000000 00:00 0
7f5040000000-7f5040b28000 rw-p 00000000 00:00 0
7f5040b28000-7f5044000000 ---p 00000000 00:00 0
7f5044000000-7f5044052000 rw-p 00000000 00:00 0
7f5044052000-7f5048000000 ---p 00000000 00:00 0
7f5048000000-7f5048a4f000 rw-p 00000000 00:00 0
7f5048a4f000-7f504c000000 ---p 00000000 00:00 0
7f504c000000-7f504cd0d000 rw-p 00000000 00:00 0
7f504cd0d000-7f5050000000 ---p 00000000 00:00 0
7f5050000000-7f5050821000 rw-p 00000000 00:00 0
7f5050821000-7f5054000000 ---p 00000000 00:00 0
7f5054000000-7f5054e3f000 rw-p 00000000 00:00 0
7f5054e3f000-7f5058000000 ---p 00000000 00:00 0
7f5058000000-7f5058b94000 rw-p 00000000 00:00 0
7f5058b94000-7f505c000000 ---p 00000000 00:00 0
7f505eff3000-7f505eff4000 ---p 00000000 00:00 0
7f505eff4000-7f505f7f4000 rw-p 00000000 00:00 0
7f505f7f4000-7f505f7f5000 ---p 00000000 00:00 0
7f505f7f5000-7f505fff5000 rw-p 00000000 00:00 0
7f5060ff7000-7f5060ff8000 ---p 00000000 00:00 0
7f5060ff8000-7f50617f8000 rw-p 00000000 00:00 0
7f50617f8000-7f5063df6000 r-xp 00000000 fc:00 5771769 /home/user/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64_lin/libmkl_def.so
7f5063df6000-7f5063ff5000 ---p 025fe000 fc:00 5771769 /home/user/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64_lin/libmkl_def.so
7f5063ff5000-7f5063ff8000 r--p 025fd000 fc:00 5771769 /home/user/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64_lin/libmkl_def.so
7f5063ff8000-7f5063ffe000 rw-p 02600000 fc:00 5771769 /home/user/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64_lin/libmkl_def.so
7f5063ffe000-7f5064000000 rw-p 00000000 00:00 0
7f5064000000-7f5064021000 rw-p 00000000 00:00 0
7f5064021000-7f5068000000 ---p 00000000 00:00 0
7f5068fc0000-7f5068fc1000 ---p 00000000 00:00 0
7f5068fc1000-7f50697c1000 rw-p 00000000 00:00 0
7f50697c1000-7f5069de4000 r-xp 00000000 fc:00 5771782 /home/user/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64_lin/libmkl_vml_def.so
7f5069de4000-7f5069fe3000 ---p 00623000 fc:00 5771782 /home/user/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64_lin/libmkl_vml_def.so
7f5069fe3000-7f5069fe6000 r--p 00622000 fc:00 5771782 /home/user/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64_lin/libmkl_vml_def.so
7f5069fe6000-7f5069ffb000 rw-p 00625000 fc:00 5771782 /home/user/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64_lin/libmkl_vml_def.so
7f5069ffb000-7f5069ffc000 rw-p 00000000 00:00 0
7f5069ffc000-7f5069ffd000 ---p 00000000 00:00 0
7f5069ffd000-7f506a7fd000 rw-p 00000000 00:00 0
7f506a7fd000-7f506a7fe000 ---p 00000000 00:00 0
7f506a7fe000-7f506affe000 rw-p 00000000 00:00 0
7f506affe000-7f506afff000 ---p 00000000 00:00 0
7f506afff000-7f506b7ff000 rw-p 00000000 00:00 0
7f506b7ff000-7f506b800000 ---p 00000000 00:00 0
7f506b800000-7f506c000000 rw-p 00000000 00:00 0
7f506c000000-7f506c021000 rw-p 00000000 00:00 0
7f506c021000-7f5070000000 ---p 00000000 00:00 0
7f50707f9000-7f50707fa000 ---p 00000000 00:00 0
7f50707fa000-7f5070ffa000 rw-p 00000000 00:00 0
7f5070ffa000-7f5070ffb000 ---p 00000000 00:00 0
7f5070ffb000-7f50717fb000 rw-p 00000000 00:00 0
7f50717fb000-7f50717fc000 ---p 00000000 00:00 0
7f50717fc000-7f5071ffc000 rw-p 00000000 00:00 0
7f5071ffc000-7f5071ffd000 ---p 00000000 00:00 0
7f5071ffd000-7f50727fd000 rw-p 00000000 00:00 0
7f50727fd000-7f50727fe000 ---p 00000000 00:00 0
7f50727fe000-7f5072ffe000 rw-p 00000000 00:00 0
7f5072ffe000-7f5072fff000 ---p 00000000 00:00 0
7f5072fff000-7f50737ff000 rw-p 00000000 00:00 0
7f50737ff000-7f5073800000 ---p 00000000 00:00 0
7f5073800000-7f5074000000 rw-p 00000000 00:00 0
7f5074000000-7f5074021000 rw-p 00000000 00:00 0
7f5074021000-7f5078000000 ---p 00000000 00:00 0
7f5078000000-7f5078021000 rw-p 00000000 00:00 0
7f5078021000-7f507c000000 ---p 00000000 00:00 0
7f507c000000-7f507c021000 rw-p 00000000 00:00 0
7f507c021000-7f5080000000 ---p 00000000 00:00 0
7f5080000000-7f5080021000 rw-p 00000000 00:00 0
7f5080021000-7f5084000000 ---p 00000000 00:00 0
7f5084000000-7f5084021000 rw-p 00000000 00:00 0
7f5084021000-7f5088000000 ---p 00000000 00:00 0
7f5088000000-7f5088021000 rw-p 00000000 00:00 0
7f5088021000-7f508c000000 ---p 00000000 00:00 0
7f508c000000-7f508c021000 rw-p 00000000 00:00 0
7f508c021000-7f5090000000 ---p 00000000 00:00 0
7f5090000000-7f5090021000 rw-p 00000000 00:00 0
7f5090021000-7f5094000000 ---p 00000000 00:00 0
7f5094000000-7f5094021000 rw-p 00000000 00:00 0
7f5094021000-7f5098000000 ---p 00000000 00:00 0
7f5098000000-7f5098021000 rw-p 00000000 00:00 0
7f5098021000-7f509c000000 ---p 00000000 00:00 0
7f509c000000-7f509c021000 rw-p 00000000 00:00 0
7f509c021000-7f50a0000000 ---p 00000000 00:00 0
7f50a0000000-7f50a0021000 rw-p 00000000 00:00 0
7f50a0021000-7f50a4000000 ---p 00000000 00:00 0
7f50a4000000-7f50a4021000 rw-p 00000000 00:00 0
7f50a4021000-7f50a8000000 ---p 00000000 00:00 0
7f50a87ed000-7f50a87ee000 ---p 00000000 00:00 0
7f50a87ee000-7f50a8fee000 rw-p 00000000 00:00 0
7f50a8fee000-7f50a8fef000 ---p 00000000 00:00 0
7f50a8fef000-7f50a97ef000 rw-p 00000000 00:00 0
7f50a97ef000-7f50a97f0000 ---p 00000000 00:00 0
7f50a97f0000-7f50a9ff0000 rw-p 00000000 00:00 0
7f50a9ff0000-7f50a9ff1000 ---p 00000000 00:00 0
7f50a9ff1000-7f50aa7f1000 rw-p 00000000 00:00 0
7f50aa7f1000-7f50aa7f2000 ---p 00000000 00:00 0
7f50aa7f2000-7f50aaff2000 rw-p 00000000 00:00 0
7f50aaff2000-7f50aaff3000 ---p 00000000 00:00 0
7f50aaff3000-7f50ab7f3000 rw-p 00000000 00:00 0
7f50ab7f3000-7f50ab7f4000 ---p 00000000 00:00 0
7f50ab7f4000-7f50abff4000 rw-p 00000000 00:00 0
7f50abff4000-7f50abff5000 ---p 00000000 00:00 0
7f50abff5000-7f50ac7f5000 rw-p 00000000 00:00 0
7f50ac7f5000-7f50ac7f6000 ---p 00000000 00:00 0
7f50ac7f6000-7f50acff6000 rw-p 00000000 00:00 0
7f50acff6000-7f50acff7000 ---p 00000000 00:00 0
7f50acff7000-7f50ad7f7000 rw-p 00000000 00:00 0
7f50ad7f7000-7f50ad7f8000 ---p 00000000 00:00 0
7f50ad7f8000-7f50adff8000 rw-p 00000000 00:00 0
7f50adff8000-7f50adff9000 ---p 00000000 00:00 0
7f50adff9000-7f50ae7f9000 rw-p 00000000 00:00 0
7f50ae7f9000-7f50ae7fa000 ---p 00000000 00:00 0
7f50ae7fa000-7f50aeffa000 rw-p 00000000 00:00 0
7f50aeffa000-7f50aeffb000 ---p 00000000 00:00 0
7f50aeffb000-7f50af7fb000 rw-p 00000000 00:00 0
7f50af7fb000-7f50af7fc000 ---p 00000000 00:00 0
7f50af7fc000-7f50afffc000 rw-p 00000000 00:00 0
7f50afffc000-7f50afffd000 ---p 00000000 00:00 0
7f50afffd000-7f50b07fd000 rw-p 00000000 00:00 0
7f50b07fd000-7f50b07fe000 ---p 00000000 00:00 0
7f50b07fe000-7f50b0ffe000 rw-p 00000000 00:00 0
7f50b0ffe000-7f50b0fff000 ---p 00000000 00:00 0
7f50b0fff000-7f50b17ff000 rw-p 00000000 00:00 0
7f50b17ff000-7f50b1800000 ---p 00000000 00:00 0
7f50b1800000-7f50b2000000 rw-p 00000000 00:00 0
7f50b2000000-7f50d8000000 ---p 00000000 00:00 0
7f50d8053000-7f50da000000 rw-p 00000000 00:00 0
7f50da000000-7f50e4000000 ---p 00000000 00:00 0
7f50e4000000-7f50e4021000 rw-p 00000000 00:00 0
7f50e4021000-7f50e8000000 ---p 00000000 00:00 0
7f50e8000000-7f50e8021000 rw-p 00000000 00:00 0
7f50e8021000-7f50ec000000 ---p 00000000 00:00 0
7f50ec7fd000-7f50ec7fe000 ---p 00000000 00:00 0
7f50ec7fe000-7f50ecffe000 rw-p 00000000 00:00 0
7f50ecffe000-7f50ecfff000 ---p 00000000 00:00 0
7f50ecfff000-7f50ed7ff000 rw-p 00000000 00:00 0
7f50ed7ff000-7f50ed800000 ---p 00000000 00:00 0
7f50ed800000-7f50ee000000 rw-p 00000000 00:00 0
7f50ee000000-7f50ee200000 ---p 00000000 00:00 0
7f50ee200000-7f50ee400000 rw-s 00000000 00:05 37415 /dev/zero (deleted)
7f50ee400000-7f50ee600000 rw-s 00000000 00:06 459 /dev/nvidiactl
7f50ee600000-7f50ee800000 rw-s 00000000 00:05 33443 /dev/zero (deleted)
7f50ee800000-7f50eea00000 rw-s 00000000 00:06 459 /dev/nvidiactl
7f50eea00000-7f50eec00000 ---p 00000000 00:00 0
7f50eec00000-7f50eeed6000 rw-s 00000000 00:06 459 /dev/nvidiactl
7f50eeed6000-7f50f0000000 ---p 00000000 00:00 0
7f50f0000000-7f50f0001000 rw-s 00000000 00:06 476 /dev/nvidia1
7f50f0001000-7f50f0002000 rw-s 00000000 00:06 476 /dev/nvidia1
7f50f0002000-7f50f0003000 rw-s 00000000 00:06 476 /dev/nvidia1
7f50f0003000-7f50f0004000 rw-s 00000000 00:06 476 /dev/nvidia1
7f50f0004000-7f50f0005000 rw-s 00000000 00:06 476 /dev/nvidia1
7f50f0005000-7f50f0006000 rw-s 00000000 00:06 476 /dev/nvidia1
7f50f0006000-7f50f0007000 rw-s 00000000 00:06 476 /dev/nvidia1
7f50f0007000-7f50f0008000 rw-s 00000000 00:06 476 /dev/nvidia1
7f50f0008000-7f50f0009000 rw-s 00000000 00:06 476 /dev/nvidia1
7f50f0009000-7f50f000a000 rw-s 00000000 00:06 476 /dev/nvidia1
7f50f000a000-7f50f000b000 rw-s 00000000 00:06 476 /dev/nvidia1
7f50f000b000-7f50f000c000 rw-s 00000000 00:06 476 /dev/nvidia1
7f50f000c000-7f50f000d000 rw-s 00000000 00:06 476 /dev/nvidia1
7f50f000d000-7f50f000e000 rw-s 00000000 00:06 476 /dev/nvidia1
7f50f000e000-7f50f000f000 rw-s 00000000 00:06 476 /dev/nvidia1
7f50f000f000-7f50f0010000 rw-s 00000000 00:06 476 /dev/nvidia1
7f50f0010000-7f5100000000 ---p 00000000 00:00 0
7f5100676000-7f5100f7a000 r-xp 00000000 fc:00 19666909 /usr/lib/nvidia-396/libnvidia-ptxjitcompiler.so.396.37
7f5100f7a000-7f510117a000 ---p 00904000 fc:00 19666909 /usr/lib/nvidia-396/libnvidia-ptxjitcompiler.so.396.37
7f510117a000-7f5101296000 rw-p 00904000 fc:00 19666909 /usr/lib/nvidia-396/libnvidia-ptxjitcompiler.so.396.37
7f5101296000-7f510129f000 rw-p 00000000 00:00 0
7f510129f000-7f51015d4000 r-xp 00000000 fc:00 19403919 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libnvrtc-builtins.so.9.2.148
7f51015d4000-7f51017d3000 ---p 00335000 fc:00 19403919 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libnvrtc-builtins.so.9.2.148
7f51017d3000-7f51017d4000 rw-p 00334000 fc:00 19403919 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libnvrtc-builtins.so.9.2.148
7f51017d4000-7f5101844000 rw-p 00000000 00:00 0
7f5101844000-7f5101845000 ---p 00000000 00:00 0
7f5101845000-7f51020c6000 rw-p 00000000 00:00 0
7f51020c6000-7f51020c7000 ---p 00000000 00:00 0
7f51020c7000-7f51028c7000 rw-p 00000000 00:00 0
7f51028c7000-7f51028c8000 ---p 00000000 00:00 0
7f51028c8000-7f51030c8000 rw-p 00000000 00:00 0
7f51030c8000-7f51090c8000 ---p 00000000 00:00 0
7f51090c8000-7f5109105000 r-xp 00000000 fc:00 19666881 /usr/lib/nvidia-396/libnvidia-fatbinaryloader.so.396.37
7f5109105000-7f5109304000 ---p 0003d000 fc:00 19666881 /usr/lib/nvidia-396/libnvidia-fatbinaryloader.so.396.37
7f5109304000-7f510930f000 rw-p 0003c000 fc:00 19666881 /usr/lib/nvidia-396/libnvidia-fatbinaryloader.so.396.37
7f510930f000-7f5109314000 rw-p 00000000 00:00 0
7f5109314000-7f5109f1a000 r-xp 00000000 fc:00 19285740 /usr/lib/x86_64-linux-gnu/libcuda.so.396.37
7f5109f1a000-7f510a11a000 ---p 00c06000 fc:00 19285740 /usr/lib/x86_64-linux-gnu/libcuda.so.396.37
7f510a11a000-7f510a280000 rw-p 00c06000 fc:00 19285740 /usr/lib/x86_64-linux-gnu/libcuda.so.396.37
7f510a280000-7f510a28f000 rw-p 00000000 00:00 0
7f510a28f000-7f510e380000 r-xp 00000000 fc:00 19407292 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libcusparse.so.9.2.148
7f510e380000-7f510e57f000 ---p 040f1000 fc:00 19407292 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libcusparse.so.9.2.148
7f510e57f000-7f510e599000 rw-p 040f0000 fc:00 19407292 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libcusparse.so.9.2.148
7f510e599000-7f510e5aa000 rw-p 00000000 00:00 0
7f510e5aa000-7f5115773000 r-xp 00000000 fc:00 19403931 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libcusolver.so.9.2.148
7f5115773000-7f5115972000 ---p 071c9000 fc:00 19403931 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libcusolver.so.9.2.148
7f5115972000-7f51159b2000 rw-p 071c8000 fc:00 19403931 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libcusolver.so.9.2.148
7f51159b2000-7f51159eb000 rw-p 00000000 00:00 0
7f51159eb000-7f511adeb000 r-xp 00000000 fc:00 19407236 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libcufft.so.9.2.148
7f511adeb000-7f511afea000 ---p 05400000 fc:00 19407236 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libcufft.so.9.2.148
7f511afea000-7f511aff8000 rw-p 053ff000 fc:00 19407236 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libcufft.so.9.2.148
7f511aff8000-7f511b045000 rw-p 00000000 00:00 0
7f511b045000-7f511e848000 r-xp 00000000 fc:00 19407206 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libcublas.so.9.2.148
7f511e848000-7f511ea47000 ---p 03803000 fc:00 19407206 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libcublas.so.9.2.148
7f511ea47000-7f511ea83000 rw-p 03802000 fc:00 19407206 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libcublas.so.9.2.148
7f511ea83000-7f511ea93000 rw-p 00000000 00:00 0
7f511ea93000-7f511fc34000 r-xp 00000000 fc:00 19403918 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libnvrtc.so.9.2.148
7f511fc34000-7f511fe33000 ---p 011a1000 fc:00 19403918 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libnvrtc.so.9.2.148
7f511fe33000-7f511ffe8000 rw-p 011a0000 fc:00 19403918 /usr/local/cuda-9.2/targets/x86_64-linux/lib/libnvrtc.so.9.2.148
7f511ffe8000-7f51200a0000 rw-p 00000000 00:00 0
7f51200a0000-7f51200a8000 r-xp 00000000 fc:00 19274385 /usr/lib/libcr.so.0.5.5
7f51200a8000-7f51202a7000 ---p 00008000 fc:00 19274385 /usr/lib/libcr.so.0.5.5
7f51202a7000-7f51202a8000 r--p 00007000 fc:00 19274385 /usr/lib/libcr.so.0.5.5
7f51202a8000-7f51202a9000 rw-p 00008000 fc:00 19274385 /usr/lib/libcr.so.0.5.5
7f51202a9000-7f51202aa000 rw-p 00000000 00:00 0
7f51202aa000-7f51202b1000 r-xp 00000000 fc:00 15208897 /lib/x86_64-linux-gnu/librt-2.23.so
7f51202b1000-7f51204b0000 ---p 00007000 fc:00 15208897 /lib/x86_64-linux-gnu/librt-2.23.so
7f51204b0000-7f51204b1000 r--p 00006000 fc:00 15208897 /lib/x86_64-linux-gnu/librt-2.23.so
7f51204b1000-7f51204b2000 rw-p 00007000 fc:00 15208897 /lib/x86_64-linux-gnu/librt-2.23.so
7f51204b2000-7f51204b5000 r-xp 00000000 fc:00 15208881 /lib/x86_64-linux-gnu/libdl-2.23.so
7f51204b5000-7f51206b4000 ---p 00003000 fc:00 15208881 /lib/x86_64-linux-gnu/libdl-2.23.so
7f51206b4000-7f51206b5000 r--p 00002000 fc:00 15208881 /lib/x86_64-linux-gnu/libdl-2.23.so
7f51206b5000-7f51206b6000 rw-p 00003000 fc:00 15208881 /lib/x86_64-linux-gnu/libdl-2.23.so
7f51206b6000-7f5120876000 r-xp 00000000 fc:00 15208879 /lib/x86_64-linux-gnu/libc-2.23.so
7f5120876000-7f5120a76000 ---p 001c0000 fc:00 15208879 /lib/x86_64-linux-gnu/libc-2.23.so
7f5120a76000-7f5120a7a000 r--p 001c0000 fc:00 15208879 /lib/x86_64-linux-gnu/libc-2.23.so
7f5120a7a000-7f5120a7c000 rw-p 001c4000 fc:00 15208879 /lib/x86_64-linux-gnu/libc-2.23.so
7f5120a7c000-7f5120a80000 rw-p 00000000 00:00 0
7f5120a80000-7f5120a96000 r-xp 00000000 fc:00 15204877 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f5120a96000-7f5120c95000 ---p 00016000 fc:00 15204877 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f5120c95000-7f5120c96000 rw-p 00015000 fc:00 15204877 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f5120c96000-7f5120cb7000 r-xp 00000000 fc:00 19271062 /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0
7f5120cb7000-7f5120eb6000 ---p 00021000 fc:00 19271062 /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0
7f5120eb6000-7f5120eb7000 r--p 00020000 fc:00 19271062 /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0
7f5120eb7000-7f5120eb8000 rw-p 00021000 fc:00 19271062 /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0
7f5120eb8000-7f512102a000 r-xp 00000000 fc:00 19273801 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f512102a000-7f512122a000 ---p 00172000 fc:00 19273801 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f512122a000-7f5121234000 r--p 00172000 fc:00 19273801 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f5121234000-7f5121236000 rw-p 0017c000 fc:00 19273801 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f5121236000-7f512123a000 rw-p 00000000 00:00 0
7f512123a000-7f5163f57000 r-xp 00000000 fc:00 5644454 /home/user/wav2letter/build/arrayfire-no-gl/lib/libafcuda.so.3.6.1
7f5163f57000-7f5164157000 ---p 42d1d000 fc:00 5644454 /home/user/wav2letter/build/arrayfire-no-gl/lib/libafcuda.so.3.6.1
7f5164157000-7f516416d000 rw-p 42d1d000 fc:00 5644454 /home/user/wav2letter/build/arrayfire-no-gl/lib/libafcuda.so.3.6.1
7f516416d000-7f516419f000 rw-p 00000000 00:00 0
7f516419f000-7f517648f000 r-xp 00000000 fc:00 19285901 /usr/lib/x86_64-linux-gnu/libcudnn.so.7.4.2
7f517648f000-7f517668e000 ---p 122f0000 fc:00 19285901 /usr/lib/x86_64-linux-gnu/libcudnn.so.7.4.2
7f517668e000-7f5176705000 rw-p 122ef000 fc:00 19285901 /usr/lib/x86_64-linux-gnu/libcudnn.so.7.4.2
7f5176705000-7f5176781000 rw-p 00000000 00:00 0
7f5176781000-7f51769c3000 r-xp 00000000 fc:00 19285959 /usr/lib/x86_64-linux-gnu/libmpich.so.12.1.0
7f51769c3000-7f5176bc3000 ---p 00242000 fc:00 19285959 /usr/lib/x86_64-linux-gnu/libmpich.so.12.1.0
7f5176bc3000-7f5176bd0000 r--p 00242000 fc:00 19285959 /usr/lib/x86_64-linux-gnu/libmpich.so.12.1.0
7f5176bd0000-7f5176bd2000 rw-p 0024f000 fc:00 19285959 /usr/lib/x86_64-linux-gnu/libmpich.so.12.1.0
7f5176bd2000-7f5176c0c000 rw-p 00000000 00:00 0
7f5176c0c000-7f51821c3000 r-xp 00000000 fc:00 19277924 /usr/lib/libnccl.so.2.1.4
7f51821c3000-7f51823c3000 ---p 0b5b7000 fc:00 19277924 /usr/lib/libnccl.so.2.1.4
7f51823c3000-7f51823c8000 rw-p 0b5b7000 fc:00 19277924 /usr/lib/libnccl.so.2.1.4
7f51823c8000-7f51823e3000 rw-p 00000000 00:00 0
7f51823e3000-7f5182403000 r-xp 00000000 fc:00 19285776 /usr/lib/x86_64-linux-gnu/libgflags.so.2.1.2
7f5182403000-7f5182602000 ---p 00020000 fc:00 19285776 /usr/lib/x86_64-linux-gnu/libgflags.so.2.1.2
7f5182602000-7f5182603000 r--p 0001f000 fc:00 19285776 /usr/lib/x86_64-linux-gnu/libgflags.so.2.1.2
7f5182603000-7f5182604000 rw-p 00020000 fc:00 19285776 /usr/lib/x86_64-linux-gnu/libgflags.so.2.1.2
7f5182604000-7f518261d000 r-xp 00000000 fc:00 15204974 /lib/x86_64-linux-gnu/libz.so.1.2.8
7f518261d000-7f518281c000 ---p 00019000 fc:00 15204974 /lib/x86_64-linux-gnu/libz.so.1.2.8
7f518281c000-7f518281d000 r--p 00018000 fc:00 15204974 /lib/x86_64-linux-gnu/libz.so.1.2.8
7f518281d000-7f518281e000 rw-p 00019000 fc:00 15204974 /lib/x86_64-linux-gnu/libz.so.1.2.8
7f518281e000-7f518282d000 r-xp 00000000 fc:00 15204850 /lib/x86_64-linux-gnu/libbz2.so.1.0.4
7f518282d000-7f5182a2c000 ---p 0000f000 fc:00 15204850 /lib/x86_64-linux-gnu/libbz2.so.1.0.4
7f5182a2c000-7f5182a2d000 r--p 0000e000 fc:00 15204850 /lib/x86_64-linux-gnu/libbz2.so.1.0.4
7f5182a2d000-7f5182a2e000 rw-p 0000f000 fc:00 15204850 /lib/x86_64-linux-gnu/libbz2.so.1.0.4
7f5182a2e000-7f5182a4f000 r-xp 00000000 fc:00 15204891 /lib/x86_64-linux-gnu/liblzma.so.5.0.0
7f5182a4f000-7f5182c4e000 ---p 00021000 fc:00 15204891 /lib/x86_64-linux-gnu/liblzma.so.5.0.0
7f5182c4e000-7f5182c4f000 r--p 00020000 fc:00 15204891 /lib/x86_64-linux-gnu/liblzma.so.5.0.0
7f5182c4f000-7f5182c50000 rw-p 00021000 fc:00 15204891 /lib/x86_64-linux-gnu/liblzma.so.5.0.0
7f5182c50000-7f5182c57000 r-xp 00000000 fc:00 19274791 /usr/lib/x86_64-linux-gnu/libogg.so.0.8.2
7f5182c57000-7f5182e57000 ---p 00007000 fc:00 19274791 /usr/lib/x86_64-linux-gnu/libogg.so.0.8.2
7f5182e57000-7f5182e58000 r--p 00007000 fc:00 19274791 /usr/lib/x86_64-linux-gnu/libogg.so.0.8.2
7f5182e58000-7f5182e59000 rw-p 00008000 fc:00 19274791 /usr/lib/x86_64-linux-gnu/libogg.so.0.8.2
7f5182e59000-7f5182ecc000 r-xp 00000000 fc:00 19275462 /usr/lib/x86_64-linux-gnu/libFLAC.so.8.3.0
7f5182ecc000-7f51830cc000 ---p 00073000 fc:00 19275462 /usr/lib/x86_64-linux-gnu/libFLAC.so.8.3.0
7f51830cc000-7f51830cd000 r--p 00073000 fc:00 19275462 /usr/lib/x86_64-linux-gnu/libFLAC.so.8.3.0
7f51830cd000-7f51830ce000 rw-p 00074000 fc:00 19275462 /usr/lib/x86_64-linux-gnu/libFLAC.so.8.3.0
7f51830ce000-7f51830f8000 r-xp 00000000 fc:00 19275284 /usr/lib/x86_64-linux-gnu/libvorbis.so.0.4.8
7f51830f8000-7f51832f7000 ---p 0002a000 fc:00 19275284 /usr/lib/x86_64-linux-gnu/libvorbis.so.0.4.8
7f51832f7000-7f51832f8000 r--p 00029000 fc:00 19275284 /usr/lib/x86_64-linux-gnu/libvorbis.so.0.4.8
7f51832f8000-7f51832f9000 rw-p 0002a000 fc:00 19275284 /usr/lib/x86_64-linux-gnu/libvorbis.so.0.4.8
7f51832f9000-7f5183386000 r-xp 00000000 fc:00 19274612 /usr/lib/x86_64-linux-gnu/libvorbisenc.so.2.0.11
7f5183386000-7f5183585000 ---p 0008d000 fc:00 19274612 /usr/lib/x86_64-linux-gnu/libvorbisenc.so.2.0.11
7f5183585000-7f51835a1000 r--p 0008c000 fc:00 19274612 /usr/lib/x86_64-linux-gnu/libvorbisenc.so.2.0.11
7f51835a1000-7f51835a2000 rw-p 000a8000 fc:00 19274612 /usr/lib/x86_64-linux-gnu/libvorbisenc.so.2.0.11
7f51835a2000-7f51836aa000 r-xp 00000000 fc:00 15204769 /lib/x86_64-linux-gnu/libm-2.23.so
7f51836aa000-7f51838a9000 ---p 00108000 fc:00 15204769 /lib/x86_64-linux-gnu/libm-2.23.so
7f51838a9000-7f51838aa000 r--p 00107000 fc:00 15204769 /lib/x86_64-linux-gnu/libm-2.23.so
7f51838aa000-7f51838ab000 rw-p 00108000 fc:00 15204769 /lib/x86_64-linux-gnu/libm-2.23.so
7f51838ab000-7f51877c4000 r-xp 00000000 fc:00 5771768 /home/user/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64_lin/libmkl_core.so
7f51877c4000-7f51879c3000 ---p 03f19000 fc:00 5771768 /home/user/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64_lin/libmkl_core.so
7f51879c3000-7f51879ca000 r--p 03f18000 fc:00 5771768 /home/user/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64_lin/libmkl_core.so
7f51879ca000-7f51879f8000 rw-p 03f1f000 fc:00 5771768 /home/user/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64_lin/libmkl_core.so
7f51879f8000-7f5187a37000 rw-p 00000000 00:00 0
7f5187a37000-7f5189070000 r-xp 00000000 fc:00 5771905 /home/user/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64_lin/libmkl_gnu_thread.so
7f5189070000-7f518926f000 ---p 01639000 fc:00 5771905 /home/user/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64_lin/libmkl_gnu_thread.so
7f518926f000-7f5189273000 r--p 01638000 fc:00 5771905 /home/user/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64_lin/libmkl_gnu_thread.so
7f5189273000-7f518928b000 rw-p 0163c000 fc:00 5771905 /home/user/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64_lin/libmkl_gnu_thread.so
7f518928b000-7f5189bb8000 r-xp 00000000 fc:00 5773997 /home/user/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64_lin/libmkl_gf_lp64.so
7f5189bb8000-7f5189db8000 ---p 0092d000 fc:00 5773997 /home/user/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64_lin/libmkl_gf_lp64.so
7f5189db8000-7f5189db9000 r--p 0092d000 fc:00 5773997 /home/user/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64_lin/libmkl_gf_lp64.so
7f5189db9000-7f5189dcb000 rw-p 0092e000 fc:00 5773997 /home/user/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64_lin/libmkl_gf_lp64.so
7f5189dcb000-7f5189dd1000 rw-p 00000000 00:00 0
7f5189dd1000-7f5189de9000 r-xp 00000000 fc:00 15208878 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f5189de9000-7f5189fe8000 ---p 00018000 fc:00 15208878 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f5189fe8000-7f5189fe9000 r--p 00017000 fc:00 15208878 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f5189fe9000-7f5189fea000 rw-p 00018000 fc:00 15208878 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f5189fea000-7f5189fee000 rw-p 00000000 00:00 0
7f5189fee000-7f518a4e3000 r-xp 00000000 fc:00 6166455 /home/user/wav2letter/src/third_party/warpctc/build/libwarpctc.so
7f518a4e3000-7f518a6e2000 ---p 004f5000 fc:00 6166455 /home/user/wav2letter/src/third_party/warpctc/build/libwarpctc.so
7f518a6e2000-7f518a6e5000 r--p 004f4000 fc:00 6166455 /home/user/wav2letter/src/third_party/warpctc/build/libwarpctc.so
7f518a6e5000-7f518a6e6000 rw-p 004f7000 fc:00 6166455 /home/user/wav2letter/src/third_party/warpctc/build/libwarpctc.so
7f518a6e6000-7f518a6e7000 rw-p 00000000 00:00 0
7f518a6e7000-7f518a70d000 r-xp 00000000 fc:00 15208877 /lib/x86_64-linux-gnu/ld-2.23.so
7f518a73f000-7f518a8f2000 rw-p 00000000 00:00 0
7f518a8fa000-7f518a8fb000 rw-p 00000000 00:00 0
7f518a8fb000-7f518a8fc000 rw-s 00000000 00:06 459 /dev/nvidiactl
7f518a8fc000-7f518a8fd000 rw-s 00000000 00:06 459 /dev/nvidiactl
7f518a8fd000-7f518a8fe000 rw-s 00000000 00:06 459 /dev/nvidiactl
7f518a8fe000-7f518a8ff000 rw-s 00000000 00:06 459 /dev/nvidiactl
7f518a8ff000-7f518a900000 rw-s 00000000 00:06 459 /dev/nvidiactl
7f518a900000-7f518a901000 rw-s 00000000 00:06 459 /dev/nvidiactl
7f518a901000-7f518a902000 rw-s 00000000 00:06 459 /dev/nvidiactl
7f518a902000-7f518a903000 rw-s 00000000 00:06 459 /dev/nvidiactl
7f518a903000-7f518a904000 rw-s 00000000 00:06 459 /dev/nvidiactl
7f518a904000-7f518a905000 rw-s 00000000 00:06 459 /dev/nvidiactl
7f518a905000-7f518a906000 rw-s 00000000 00:06 459 /dev/nvidiactl
7f518a906000-7f518a907000 rw-s 00000000 00:06 459 /dev/nvidiactl
7f518a907000-7f518a908000 rw-s 00000000 00:06 459 /dev/nvidiactl
7f518a908000-7f518a909000 rw-s 00000000 00:06 459 /dev/nvidiactl
7f518a909000-7f518a90a000 rw-s 00000000 00:06 459 /dev/nvidiactl
7f518a90a000-7f518a90b000 rw-s 00000000 00:06 459 /dev/nvidiactl
7f518a90b000-7f518a90c000 rw-p 00000000 00:00 0
7f518a90c000-7f518a90d000 r--p 00025000 fc:00 15208877 /lib/x86_64-linux-gnu/ld-2.23.so
7f518a90d000-7f518a90e000 rw-p 00026000 fc:00 15208877 /lib/x86_64-linux-gnu/ld-2.23.so
7f518a90e000-7f518a90f000 rw-p 00000000 00:00 0
7ffeb9f3d000-7ffeb9f5e000 rw-p 00000000 00:00 0 [stack]
7ffeb9fcb000-7ffeb9fce000 r--p 00000000 00:00 0 [vvar]
7ffeb9fce000-7ffeb9fd0000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Aborted (core dumped)

It's hard for me to understand it. I'm using WSJ recipe and memory seems to be around 2.5 GB. logs are empty.

[tlikhomanenko]

Hi @smolendawid,

We have faced the same issue and found out that for some specific shapes of matrices (not a corner cases) there is a bug in cudnn during cudnnConvolutionBackwardFilter call. This bug is already reported (if you can get the access to it, details here https://developer.nvidia.com/nvidia_bug/2483750)

The quick fix to avoid this error in cudnn is to set the algorithm for backward to CUDNN_CONVOLUTION_BWD_FILTER_ALGO_0 in
https://github.com/facebookresearch/flashlight/blob/master/flashlight/autograd/backend/cuda/Conv2D.cpp#L190: change this row to

cudnnConvolutionBwdFilterAlgo_t bwd_filter_algo = CUDNN_CONVOLUTION_BWD_FILTER_ALGO_0;

and comment the next line (where the algorithm for backward is set by cudnn call).

[smolendawid]

What I've done precisely was:
changing line 190:
cudnnConvolutionBwdFilterAlgo_t bwd_filter_algo;
to
cudnnConvolutionBwdFilterAlgo_t bwd_filter_algo = CUDNN_CONVOLUTION_BWD_FILTER_ALGO_0;
and
in flashlight
_make clean_
_make_

and in wav2letter
_make clean_
_make j16_

no errors occurred during make. Unfortunately, I face the same error during training. Did I miss something? Maybe I could change the cudnn version?

[jacobkahn]

@smolendawid — you also need to comment out Conv2D.cpp#L191-L199, or your default value will be overwritten by the call to cudnnGetConvolutionBackwardFilterAlgorithm. Give that a try?

[smolendawid]

Thank you for the help, however, I still can't overcome this error.

Just like suggested, I replaced line 190 and commented lines <191-199>. Then I run make clean and make in both flashlight and wav2letter. After that, command:

user@server:~/wav2letter$ build/Train train --flagsfile=/home/user/wav2letter/recipes/my_recipe/configs/conv_glu/train.cfg

returns the similar error to above one.

Can using docker image or changing the version of cudnn help?

[tlikhomanenko]

@smolendawid Did you also run make install for flashlight?

[smolendawid]

I forgot about that, excuse me. Thanks!