Wav2letter: Can't make this architecture run

[julenmerchan]

Hi, I'm trying to build a 1D Conv net based on this paper https://openreview.net/pdf?id=BkUDvt5gg, this is my binary file for the architecture:

V -1 NFEAT 1 0
AC NFEAT 250 250 160 0 0 0
R
AC 250 250 48 2 0 0 0
R
AC 250 250 7 1 0 0 0
R
AC 250 250 7 1 0 0 0
R
AC 250 250 7 1 0 0 0
R
AC 250 250 7 1 0 0 0
R
AC 250 250 7 1 0 0 0
R
AC 250 250 7 1 0 0 0
R
AC 250 250 7 1 0 0 0
R
AC 250 2000 32 1 0 0 0
R
AC 2000 2000 1 1 0 0 0
R
AC 2000 NLABEL 1 1 0 0 0

But every time I try to train de model I get this error:

terminate called after throwing an instance of 'std::invalid_argument'
what(): CUDNN_STATUS_BAD_PARAM
* Aborted at 1586109531 (unix time) try "date -d @1586109531" if you are using GNU date
PC: @ 0x7fe6bcb4e428 gsignal
SIGABRT (@0x16) received by PID 22 (TID 0x7fe70a332600) from PID 22; stack trace: *
@ 0x7fe70920e390 (unknown)
@ 0x7fe6bcb4e428 gsignal
@ 0x7fe6bcb5002a abort
@ 0x7fe6bd49184d __gnu_cxx::__verbose_terminate_handler()
@ 0x7fe6bd48f6b6 (unknown)
@ 0x7fe6bd48f701 std::terminate()
@ 0x7fe6bd48f919 __cxa_throw
@ 0x70fa6e fl::ConvDescriptor::ConvDescriptor()
@ 0x70d4b6 fl::conv2d()
@ 0x700a9b fl::AsymmetricConv1D::forward()
@ 0x6e353f fl::UnaryModule::forward()
@ 0x6d3282 fl::Sequential::forward()
@ 0x49352f _ZZ4mainENKUlSt10shared_ptrIN2fl6ModuleEES_IN3w2l17SequenceCriterionEES_INS3_10W2lDatasetEES_INS0_19FirstOrderOptimizerEES9_ddbiE3_clES2_S5_S7_S9_S9_ddbi.constprop.12666
@ 0x41c3f0 main
@ 0x7fe6bcb39830 __libc_start_main
@ 0x48e3d9 _start
@ 0x0 (unknown)
Aborted (core dumped)

I've tried to train models with other architectures with the same dataset and it's worked every time, so I assume there is a problem with the architecture above.

Can someone help me? thx.

[julenmerchan]

I've modified the arch to:

V -1 1 NFEAT 0
AC NFEAT 250 250 160 -1 0 1
R
AC 250 250 48 2 -1 0 1
R
AC 250 250 7 1 -1 0 1
R
AC 250 250 7 1 -1 0 1
R
AC 250 250 7 1 -1 0 1
R
AC 250 250 7 1 -1 0 1
R
AC 250 250 7 1 -1 0 1
R
AC 250 250 7 1 -1 0 1
R
AC 250 250 7 1 -1 0 1
R
AC 250 2000 32 1 -1 0 1
R
AC 2000 2000 1 1 -1 0 1
R
AC 2000 NLABEL 1 1 -1 0 1

Now I'm getting this error:
terminate called after throwing an instance of 'af::exception'
what(): ArrayFire Exception (Invalid input argument:202):
In function af_err af_index(void*, af_array, unsigned int, const af_seq)
In file src/api/c/index.cpp:80
Invalid argument at index 3
Expected: (indices_[i].begin >= 0. && indices_[i].end >= 0.)

In function af::array::array_proxy::operator af::array()
In file src/api/cpp/array.cpp:649
* Aborted at 1586167473 (unix time) try "date -d @1586167473" if you are using GNU date
PC: @ 0x7f98d3d88428 gsignal
SIGABRT (@0xe21) received by PID 3617 (TID 0x7f992156c600) from PID 3617; stack trace: *
@ 0x7f9920448390 (unknown)
@ 0x7f98d3d88428 gsignal
@ 0x7f98d3d8a02a abort
@ 0x7f98d46cb84d __gnu_cxx::__verbose_terminate_handler()
@ 0x7f98d46c96b6 (unknown)
@ 0x7f98d46c9701 std::terminate()
@ 0x7f98d46c9919 __cxa_throw
@ 0x7f98f43d4b7a af::array::array_proxy::operator ?()
@ 0x6a41fd fl::Variable::rows()
@ 0x700ae1 fl::AsymmetricConv1D::forward()
@ 0x6e353f fl::UnaryModule::forward()
@ 0x6d3282 fl::Sequential::forward()
@ 0x49352f _ZZ4mainENKUlSt10shared_ptrIN2fl6ModuleEES_IN3w2l17SequenceCriterionEES_INS3_10W2lDatasetEES_INS0_19FirstOrderOptimizerEES9_ddbiE3_clES2_S5_S7_S9_S9_ddbi.constprop.12666
@ 0x41c3f0 main
@ 0x7f98d3d73830 __libc_start_main
@ 0x48e3d9 _start
@ 0x0 (unknown)
Aborted (core dumped)

[avidov]

Arrayfire released a new version recently. Please pull the latests version of everything to see if it is simply that.

[vineelpratap]

Hi, futurePart_ should be 0.5 so that we do symmetric padding ==> AC NFEAT 250 250 160 -1 0.5 1 .

Also, make sure you do not use -mfsc or -mfcc or -powspec so that it operates on raw audio as mentioned in the paper.

[julenmerchan]

Hi, futurePart_ should be 0.5 so that we do symmetric padding ==> AC NFEAT 250 250 160 -1 0.5 1 .

Also, make sure you do not use -mfsc or -mfcc or -powspec so that it operates on raw audio as mentioned in the paper.

I've tried your solution but now I'm getting new error:

terminate called after throwing an instance of 'af::exception'
what(): CTC: Incorrect dimensions for input. Expected dim4(N, T, B)
* Aborted at 1586246651 (unix time) try "date -d @1586246651" if you are using GNU date
PC: @ 0x7f9c75781428 gsignal
SIGABRT (@0x1e3) received by PID 483 (TID 0x7f9cc2f65600) from PID 483; stack trace: *
@ 0x7f9cc1e41390 (unknown)
@ 0x7f9c75781428 gsignal
@ 0x7f9c7578302a abort
@ 0x7f9c760c484d __gnu_cxx::__verbose_terminate_handler()
@ 0x7f9c760c26b6 (unknown)
@ 0x7f9c760c2701 std::terminate()
@ 0x7f9c760c2919 __cxa_throw
@ 0x5d710b w2l::ConnectionistTemporalClassificationCriterion::validate()
@ 0x5cf86b w2l::ConnectionistTemporalClassificationCriterion::forward()
@ 0x49374c _ZZ4mainENKUlSt10shared_ptrIN2fl6ModuleEES_IN3w2l17SequenceCriterionEES_INS3_10W2lDatasetEES_INS0_19FirstOrderOptimizerEES9_ddbiE3_clES2_S5_S7_S9_S9_ddbi.constprop.12666
@ 0x41c3f0 main
@ 0x7f9c7576c830 __libc_start_main
@ 0x48e3d9 _start
@ 0x0 (unknown)
Aborted (core dumped)

The architecture looks like this now:

V -1 1 NFEAT 0
AC NFEAT 250 250 160 -1 0.5 1
R
AC 250 250 48 2 -1 0.5 1
R
AC 250 250 7 1 -1 0.5 1
R
AC 250 250 7 1 -1 0.5 1
R
AC 250 250 7 1 -1 0.5 1
R
AC 250 250 7 1 -1 0.5 1
R
AC 250 250 7 1 -1 0.5 1
R
AC 250 250 7 1 -1 0.5 1
R
AC 250 250 7 1 -1 0.5 1
R
AC 250 2000 32 1 -1 0.5 1
R
AC 2000 2000 1 1 -1 0.5 1
R
AC 2000 NLABEL 1 1 -1 0.5 1

msfc and mfcc are disabled, I've tried to change V layer to:

V -1 NFEAT 0 1

but I get this error:

terminate called after throwing an instance of 'af::exception'
what(): CTC: Batchsize mismatch for input and target
* Aborted at 1586246625 (unix time) try "date -d @1586246625" if you are using GNU date
PC: @ 0x7f78e54ae428 gsignal
SIGABRT (@0x1cc) received by PID 460 (TID 0x7f7932c92600) from PID 460; stack trace: *
@ 0x7f7931b6e390 (unknown)
@ 0x7f78e54ae428 gsignal
@ 0x7f78e54b002a abort
@ 0x7f78e5df184d __gnu_cxx::__verbose_terminate_handler()
@ 0x7f78e5def6b6 (unknown)
@ 0x7f78e5def701 std::terminate()
@ 0x7f78e5def919 __cxa_throw
@ 0x5d710b w2l::ConnectionistTemporalClassificationCriterion::validate()
@ 0x5cf86b w2l::ConnectionistTemporalClassificationCriterion::forward()
@ 0x49374c _ZZ4mainENKUlSt10shared_ptrIN2fl6ModuleEES_IN3w2l17SequenceCriterionEES_INS3_10W2lDatasetEES_INS0_19FirstOrderOptimizerEES9_ddbiE3_clES2_S5_S7_S9_S9_ddbi.constprop.12666
@ 0x41c3f0 main
@ 0x7f78e5499830 __libc_start_main
@ 0x48e3d9 _start
@ 0x0 (unknown)
Aborted (core dumped)

I'm going to try to update all as @avidov said and I'll comment if that works.

[julenmerchan]

Hi everyone, I've tried to update flashlight and I'm getting this error:

CMake Error at CMakeLists.txt:65 (message):
ArrayFire versions < 3.7.1 are no longer supported with flashlight. To
build flashlight with a version of ArrayFire < 3.7.1, use commit <=
5518d91b7f4fd5b400cbc802cfbecc0df57836bd.

I've installed ArrayFire 3.7.1 following the tutorial http://arrayfire.org/docs/installing.html but for some reason Flashlight doesn't detect ArrayFire 3.7.1

[julenmerchan]

Hi everyone, I've tried to update flashlight and I'm getting this error:

CMake Error at CMakeLists.txt:65 (message):
ArrayFire versions < 3.7.1 are no longer supported with flashlight. To
build flashlight with a version of ArrayFire < 3.7.1, use commit <=
5518d91b7f4fd5b400cbc802cfbecc0df57836bd.

I've installed ArrayFire 3.7.1 following the tutorial http://arrayfire.org/docs/installing.html but for some reason Flashlight doesn't detect ArrayFire 3.7.1

This problem is solved adding -DArrayFire_DIR=/opt/arrayfire/share/ArrayFire/cmake/ option.

[vineelpratap]

Make sure to add RO 2 0 3 1 as the final line in your arch file so that inputs are sent in the right order to CTC.

[julenmerchan]

Make sure to add RO 2 0 3 1 as the final line in your arch file so that inputs are sent in the right order to CTC.

Many thanks now it works.