Tts: Torch.jit, onnx again. Is it convertable after all?

[vcjob]

Hello everyone!

I spent a few days trying to convert a tacotron2 model to torch.jit.script and onnx.

1. I did convert it to onnx, thought I couldn't convert the onnx model to TensorRT after all due to some errors.
2. I can only do tracing with torch.jit, but is works, obviously, for the same input length. I gave up changing the code for jit.script, even to script only the Decoder. No success so far.
   I read on the Nvidia's Tacotron2 repo, they say it must be not hard to export in jit.script.
   Did anyone successfully converted the model to jit.script? Maybe to onnx?
   If so, please, give some hints!!
   @ergol, any plans for special branch maybe for jit/onnx?

Thanks for any suggestions!

[erogol]

We had done some preliminary work on that. I was able to convert it with torch script but never tried onnx. However, I've lost the branch at some point.

@reuben has even written C based inference code for TTS. He probably has more to say about this.

If you are willing to take this effort forward, I'd be happy to help.

[reuben]

I was able to get a basic setup working by making TTS + Griffin-Lim work with TorchScript. Then from C++ I could get the PCM samples directly as output. You can see the changes here: https://github.com/mozilla/TTS/compare/master...reuben:native_inference_1.2_rebased_dev

It's outdated now. Maybe PyTorch 1.4 has made things easier, I haven't tried again. What I saw at the time is that it is not any faster than the Python code, so I stopped spending time on that.

[vcjob]

I was able to get a basic setup working by making TTS + Griffin-Lim work with TorchScript. Then from C++ I could get the PCM samples directly as output. You can see the changes here: master...reuben:native_inference_1.2_rebased_dev

It's outdated now. Maybe PyTorch 1.4 has made things easier, I haven't tried again. What I saw at the time is that it is not any faster than the Python code, so I stopped spending time on that.

Thank you!
I've tried to do the same changes, it didn't work, had errors with _ValueError: Unknown type annotation: 'typing_extensions.Final[float]_ for some reason.
Anyway, I tried just trace complete model (making it input invariant), though that lead to increasing the inference time. I expected it to improve speed like 1.5 times at least or more. I find it strange. Maybe I did something wrong.

[erogol]

What I saw at the time is that it is not any faster than the Python code, so I stopped spending time on that.

[Kotsiopoulos]

Hello everyone!

I spent a few days trying to convert a tacotron2 model to torch.jit.script and onnx.

1. I did convert it to onnx, thought I couldn't convert the onnx model to TensorRT after all due to some errors.

2. I can only do tracing with torch.jit, but is works, obviously, for the same input length. I gave up changing the code for jit.script, even to script only the Decoder. No success so far.
   I read on the Nvidia's Tacotron2 repo, they say it must be not hard to export in jit.script.
   Did anyone successfully converted the model to jit.script? Maybe to onnx?
   If so, please, give some hints!!
   @ergol, any plans for special branch maybe for jit/onnx?

Thanks for any suggestions!

Can you elaborate on how you successfully converted tacontron2 to onnx?
I am trying to export tacotron2 checkpoint_261000 with:

text_cleaner = [CONFIG.text_cleaner]
if CONFIG.use_phonemes:
seq = np.asarray(
phoneme_to_sequence(text, text_cleaner, CONFIG.phoneme_language, enable_eos_bos_chars),
dtype=np.int32)
else:
seq = np.asarray(text_to_sequence(text, text_cleaner), dtype=np.int32)
chars_var = torch.from_numpy(seq).unsqueeze(0)
# synthesize voice
if use_cuda:
chars_var = chars_var.cuda()
if truncated:
decoder_output, postnet_output, alignments, stop_tokens = model.inference_truncated(
chars_var.long())

else:
    decoder_output, postnet_output, alignments, stop_tokens = model.inference(chars_var.long())
    print('attempt to save model to onnx')
    torch.onnx.export(model, chars_var.long(),location_to_save_model)

but i get the error:
KeyError: '_thnn_fused_lstm_cell'

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. You might also look our discourse page for further help. https://discourse.mozilla.org/c/tts

[hengyeliu]

Hello everyone!
I spent a few days trying to convert a tacotron2 model to torch.jit.script and onnx.

1. I did convert it to onnx, thought I couldn't convert the onnx model to TensorRT after all due to some errors.

2. I can only do tracing with torch.jit, but is works, obviously, for the same input length. I gave up changing the code for jit.script, even to script only the Decoder. No success so far.
   I read on the Nvidia's Tacotron2 repo, they say it must be not hard to export in jit.script.
   Did anyone successfully converted the model to jit.script? Maybe to onnx?
   If so, please, give some hints!!
   @ergol, any plans for special branch maybe for jit/onnx?

Thanks for any suggestions!

Can you elaborate on how you successfully converted tacontron2 to onnx?
I am trying to export tacotron2 checkpoint_261000 with:

text_cleaner = [CONFIG.text_cleaner]
if CONFIG.use_phonemes:
seq = np.asarray(
phoneme_to_sequence(text, text_cleaner, CONFIG.phoneme_language, enable_eos_bos_chars),
dtype=np.int32)
else:
seq = np.asarray(text_to_sequence(text, text_cleaner), dtype=np.int32)
chars_var = torch.from_numpy(seq).unsqueeze(0)

synthesize voice

if use_cuda:
chars_var = chars_var.cuda()
if truncated:
decoder_output, postnet_output, alignments, stop_tokens = model.inference_truncated(
chars_var.long())

else:
    decoder_output, postnet_output, alignments, stop_tokens = model.inference(chars_var.long())
    print('attempt to save model to onnx')
    torch.onnx.export(model, chars_var.long(),location_to_save_model)

but i get the error:
KeyError: '_thnn_fused_lstm_cell'

hello，have you solved this error? I got same error.