Opennmt-py: error when trying to do speech to text preprocessing

[Nouraaaawwad]

when i try to do the example speech to text, when i run this command ::

!python OpenNMT-py/preprocess.py -data_type audio -src_dir OpenNMT-py/data/speech/an4_dataset -train_src OpenNMT-py/data/speech/src-train.txt -train_tgt OpenNMT-py/data/speech/tgt-train.txt -valid_src OpenNMT-py/data/speech/src-val.txt -valid_tgt OpenNMT-py/data/speech/tgt-val.txt -save_data OpenNMT-py/data/speech/demo

i got this error ::

[2020-03-03 20:09:24,327 INFO] Extracting features...
[2020-03-03 20:09:24,327 INFO] * number of source features: 0.
[2020-03-03 20:09:24,327 INFO] * number of target features: 0.
[2020-03-03 20:09:24,327 INFO] Building Fields object...
[2020-03-03 20:09:24,327 INFO] Building & saving training data...
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(args, *kwds))
File "/content/OpenNMT-py/onmt/bin/preprocess.py", line 52, in process_one_shard
assert len(src_shard) == len(tgt_shard)
AssertionError
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "OpenNMT-py/preprocess.py", line 6, in
main()
File "/content/OpenNMT-py/onmt/bin/preprocess.py", line 293, in main
preprocess(opt)
File "/content/OpenNMT-py/onmt/bin/preprocess.py", line 273, in preprocess
'train', fields, src_reader, tgt_reader, align_reader, opt)
File "/content/OpenNMT-py/onmt/bin/preprocess.py", line 200, in build_save_dataset
for sub_counter in p.imap(func, shard_iter):
File "/usr/lib/python3.6/multiprocessing/pool.py", line 735, in next
raise value
AssertionError

i am using google colab to implement it.

[francoishernandez]

You probably have an issue with the data. Check that your source and target have the same length.

[Nouraaaawwad]

your were right i just fix the file and add a new one, and i am certain it is the same length and rerun it and i got this error : ( ::

[2020-03-03 18:52:48,552 INFO] Extracting features...
[2020-03-03 18:52:48,552 INFO] * number of source features: 0.
[2020-03-03 18:52:48,552 INFO] * number of target features: 0.
[2020-03-03 18:52:48,552 INFO] Building Fields object...
[2020-03-03 18:52:48,553 INFO] Building & saving training data...
[2020-03-03 18:52:48,561 INFO] Building shard 0.
0% 0/276 [00:00

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(args, *kwds))
File "/content/OpenNMT-py/onmt/bin/preprocess.py", line 64, in process_one_shard
filter_pred=filter_pred
File "/content/OpenNMT-py/onmt/inputters/dataset_base.py", line 121, in __init__
for ex_dict in starmap(_join_dicts, zip(*read_iters)):
File "/content/OpenNMT-py/onmt/inputters/audio_dataset.py", line 127, in read
spect = self.extract_features(audio_path)
File "/content/OpenNMT-py/onmt/inputters/audio_dataset.py", line 64, in extract_features
sound, sample_rate_ = torchaudio.load(audio_path)
File "/usr/local/lib/python3.6/dist-packages/torchaudio/__init__.py", line 86, in load
filetype=filetype,
File "/usr/local/lib/python3.6/dist-packages/torchaudio/_sox_backend.py", line 47, in load
filetype
RuntimeError: Error opening audio file
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/content/OpenNMT-py/preprocess.py", line 6, in
main()
File "/content/OpenNMT-py/onmt/bin/preprocess.py", line 293, in main
preprocess(opt)
File "/content/OpenNMT-py/onmt/bin/preprocess.py", line 273, in preprocess
'train', fields, src_reader, tgt_reader, align_reader, opt)
File "/content/OpenNMT-py/onmt/bin/preprocess.py", line 200, in build_save_dataset
for sub_counter in p.imap(func, shard_iter):
File "/usr/lib/python3.6/multiprocessing/pool.py", line 735, in next
raise value
RuntimeError: Error opening audio file

i don't know why it can't open the audio file.

[francoishernandez]

The audio file is probably not in the right format.

2020-03-03 18:52:48,561 INFO] Building shard 0.
0% 0/276 [00:00<?, ?it/s]formats: can't open input file `/content/OpenNMT-py/data/speech/an4_dataset/train/wav/2.wav': WAVE: RIFF header not found

[Nouraaaawwad]

i just figured out it raise this error to almost half of the audio collection.

i don't know where is the problem or why are some of the audio has a correct header and some not !

is there a way to solve this problem ? does it have to do with converting the audio from .mp3 to .wav ?

I feel my head is gonna explode from this project T-T

[francoishernandez]

You probably need to convert your files to wav 16k.
Something like
ffmpeg -i your_audio_file -acodec pcm_s16le -ar 16000 -ac 1 your_converted_file.wav

[Nouraaaawwad]

yes it worked very smoothly ! thank you soo much francois.

[Nouraaaawwad]

l just face this issue, but i wasn't sure if i need to open a new issue for it. :/
i did run this command ::

!python /content/OpenNMT-py/train.py -model_type audio -enc_rnn_size 512 -dec_rnn_size 512 -audio_enc_pooling 1,2 -dropout 0 -enc_layers 2 -dec_layers 1 -rnn_type LSTM -data /content/OpenNMT-py/data/speech/demo -save_model demo-model -global_attention mlp -gpu_ranks 0 -batch_size 8 -optim adam -max_grad_norm 100 -learning_rate 0.0003 -learning_rate_decay 0.8 -train_steps 2000

and i got this error ::

_[2020-03-04 21:03:57,891 INFO] * tgt vocab size = 15
[2020-03-04 21:03:57,892 INFO] Building model...
[2020-03-04 21:04:02,067 INFO] NMTModel(
(encoder): AudioEncoder(
(W): Linear(in_features=512, out_features=512, bias=False)
(batchnorm_0): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(rnn_0): LSTM(161, 512)
(pool_0): MaxPool1d(kernel_size=1, stride=1, padding=0, dilation=1, ceil_mode=False)
(rnn_1): LSTM(512, 512)
(pool_1): MaxPool1d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
(batchnorm_1): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(decoder): InputFeedRNNDecoder(
(embeddings): Embeddings(
(make_embedding): Sequential(
(emb_luts): Elementwise(
(0): Embedding(15, 500, padding_idx=1)
)
)
)
(dropout): Dropout(p=0.0, inplace=False)
(rnn): StackedLSTM(
(dropout): Dropout(p=0.0, inplace=False)
(layers): ModuleList(
(0): LSTMCell(1012, 512)
)
)
(attn): GlobalAttention(
(linear_context): Linear(in_features=512, out_features=512, bias=False)
(linear_query): Linear(in_features=512, out_features=512, bias=True)
(v): Linear(in_features=512, out_features=1, bias=False)
(linear_out): Linear(in_features=1024, out_features=512, bias=True)
)
)
(generator): Sequential(
(0): Linear(in_features=512, out_features=15, bias=True)
(1): Cast()
(2): LogSoftmax()
)
)
[2020-03-04 21:04:02,067 INFO] encoder: 3747840
[2020-03-04 21:04:02,067 INFO] decoder: 4190555
[2020-03-04 21:04:02,067 INFO] * number of parameters: 7938395
[2020-03-04 21:04:02,068 INFO] Starting training on GPU: [0]
[2020-03-04 21:04:02,068 INFO] Start training loop and validate every 10000 steps...
[2020-03-04 21:04:02,069 INFO] Loading dataset from /content/OpenNMT-py/data/speech/demo.train.0.pt
[2020-03-04 21:04:02,070 INFO] number of examples: 15
Traceback (most recent call last):
File "/content/OpenNMT-py/train.py", line 6, in
main()
File "/content/OpenNMT-py/onmt/bin/train.py", line 204, in main
train(opt)
File "/content/OpenNMT-py/onmt/bin/train.py", line 88, in train
single_main(opt, 0)
File "/content/OpenNMT-py/onmt/train_single.py", line 143, in main
valid_steps=opt.valid_steps)
File "/content/OpenNMT-py/onmt/trainer.py", line 244, in train
report_stats)
File "/content/OpenNMT-py/onmt/trainer.py", line 365, in _gradient_accumulation
with_align=self.with_align)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
result = self.forward(input, *kwargs)
File "/content/OpenNMT-py/onmt/models/model.py", line 45, in forward
enc_state, memory_bank, lengths = self.encoder(src, lengths)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
result = self.forward(input, *kwargs)
File "/content/OpenNMT-py/onmt/encoders/audio_encoder.py", line 119, in forward
memory_bank = pool(memory_bank)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
result = self.forward(input, *kwargs)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/pooling.py", line 76, in forward
self.return_indices)
File "/usr/local/lib/python3.6/dist-packages/torch/_jit_internal.py", line 181, in fn
return if_false(args, *kwargs)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 457, in _max_pool1d
input, kernel_size, stride, padding, dilation, ceil_mode)
RuntimeError: Given input size: (7x1x1). Calculated output size: (7x1x0). Output size is too small_

I think the problem is in the pooling size but i don't know how to fix it.
it is my first time working with this amazing open source OpenNMT so i don't have much experience T-T.

[francoishernandez]

Not sure about this. Maybe one of your audio files is too short.

[Nouraaaawwad]

the audio duration varying between 5s to 1s, does all the audio have to be with the same length ?
i tried to make the audio files similar to the ones in the speech to text example :/ , but with another language

** Update:
i removed all the 2s and 1s audio files, but still getting the same error.
i changed the -audio_enc_pooling from -audio_enc_pooling 1,2 to -audio_enc_pooling 1 and the -enc_layers 2 the same as before, and it did work fine.
but does that affect the accuracy of the training ? is it better to have -audio_enc_pooling 1,2 ?

[Nouraaaawwad]

since i changed the audio_enc_pooling to 1, the recognition accuracy is very bad. i think the problem is in the preprocessing results.
how to read the demo.vocab.pt file ?
i tried this command :

import torch
vocab = torch.load("OpenNMT-py/data/Tutor/demo.vocab.pt")
print(vocab) [ ('src', ),
('tgt', )]

but it gives me a syntax error : /