Tensorflowtts: Deployment of frozen models with tf-addons

[sujeendran]

Hi,
For freezing the graph and deploying on target device, is it necessary to have Tensorflow-Addons installed on the device? Or will it be possible for just tensorflow 2.x package to perform inference from the frozen graph. Sorry but I have never used Tensorflow-Addons before and dont have much knowledge about it.

[dathudeptrai]

Hi, can you refer the code to freeze entire tacotron 2 from notebooks dir? I think only tf dependency is enough, can you try?

[sujeendran]

@dathudeptrai Tested FastSpeech saved model inference in environment without tensorflow-addons. Works!
Will update once I check other models.
How can I save mb-melgan in the same way? Is there a way to combine both the mb_melgan module and PQMF module into one?

[dathudeptrai]

@sujeendran i will add notebook for mb-melgan saved model inference asap.

Is there a way to combine both the mb_melgan module and PQMF module into one?

ofc, you can :))), i will do it for you :)).

[dathudeptrai]

@sujeendran https://github.com/dathudeptrai/TensorflowTTS/blob/master/notebooks/multiband_melgan_inference.ipynb. I will close issue now.

[sujeendran]

Hi @dathudeptrai,
Thanks for the update. I followed your notebook to save the model and later load it from another script for inference. Sorry to reopen this issue, but I'm facing an issue.
I noticed it takes twice as much time for inference compared to another script that builds the model and load weights manually like in your notebooks. I ran both scripts to generate 4 sentences in a loop using FastSpeech and mb-melgan and the difference in performance is clearly evident. Ignoring the warm-up run, the script that builds model and loads weights takes just ~2 seconds while the script that directly loads saved model takes ~4 seconds for each sentence.
Here is my script to run inference using saved model:

import os
#To run on CPU uncomment below line
#os.environ["CUDA_VISIBLE_DEVICES"]="-1"
import numpy as np
import tensorflow as tf
import yaml
import soundfile as sf
import time
from tensorflow_tts.processor.ljspeech import LJSpeechProcessor
from tensorflow_tts.processor.ljspeech import symbols, _symbol_to_id
processor = LJSpeechProcessor(None, "english_cleaners")
sentences = ["Recent research at Harvard has shown meditating for as little as 8 weeks, can actually increase the grey matter in the parts of the brain responsible for emotional regulation, and learning.","Something doesn't seem right.","Is this barrel of the rifle?","What can it do now?"]
fastspeech = tf.saved_model.load("./fsfreezed")
mbm = tf.saved_model.load("./mbmfreezed")
for i,sentence in enumerate(sentences):
    start = time.time()
    input_ids = tf.expand_dims(tf.convert_to_tensor(processor.text_to_sequence(sentence), dtype=tf.int32), 0)
    mel_before, mel_after, duration_outputs = fastspeech.inference(
        input_ids=input_ids,
        attention_mask=tf.math.not_equal(input_ids, 0),
        speaker_ids=tf.convert_to_tensor([0], dtype=tf.int32),
        speed_ratios=tf.convert_to_tensor([1.0], dtype=tf.float32),
    )
    audio_after = mbm(tf.convert_to_tensor(mel_after,dtype=tf.float32))[0, :,0]     #Tried to convert to tensor to avoid retracing. Didn't help either
    end = time.time()
    # save to file
    sf.write(f'frozen_test_{i}.wav', audio_after, 22050, "PCM_16")
    print(f"Time taken: {end-start}s")

Output:

2020-06-29 17:26:03.962096: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-06-29 17:26:20.135192: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-06-29 17:26:20.162452: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:949] ARM64 does not support NUMA - returning NUMA node zero
2020-06-29 17:26:20.162651: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.87GiB deviceMemoryBandwidth: 194.55MiB/s
2020-06-29 17:26:20.162733: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-06-29 17:26:20.196368: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-06-29 17:26:20.215041: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-06-29 17:26:20.255546: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-06-29 17:26:20.274772: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-06-29 17:26:20.293345: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-06-29 17:26:20.293954: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-06-29 17:26:20.294285: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:949] ARM64 does not support NUMA - returning NUMA node zero
2020-06-29 17:26:20.294607: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:949] ARM64 does not support NUMA - returning NUMA node zero
2020-06-29 17:26:20.294693: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-06-29 17:26:20.297638: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:949] ARM64 does not support NUMA - returning NUMA node zero
2020-06-29 17:26:20.297778: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.87GiB deviceMemoryBandwidth: 194.55MiB/s
2020-06-29 17:26:20.297849: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-06-29 17:26:20.297933: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-06-29 17:26:20.297999: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-06-29 17:26:20.298057: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-06-29 17:26:20.298113: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-06-29 17:26:20.298163: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-06-29 17:26:20.298211: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-06-29 17:26:20.298414: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:949] ARM64 does not support NUMA - returning NUMA node zero
2020-06-29 17:26:20.298637: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:949] ARM64 does not support NUMA - returning NUMA node zero
2020-06-29 17:26:20.298704: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-06-29 17:26:20.298795: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-06-29 17:26:24.034187: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-06-29 17:26:24.034267: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0
2020-06-29 17:26:24.034300: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N
2020-06-29 17:26:24.034776: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:949] ARM64 does not support NUMA - returning NUMA node zero
2020-06-29 17:26:24.035076: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:949] ARM64 does not support NUMA - returning NUMA node zero
2020-06-29 17:26:24.035389: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:949] ARM64 does not support NUMA - returning NUMA node zero
2020-06-29 17:26:24.035516: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1457 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2020-06-29 17:27:55.581187: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-06-29 17:27:58.583955: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
Time taken: 33.52933359146118s
2020-06-29 17:28:25.711613: W tensorflow/core/common_runtime/bfc_allocator.cc:246] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.21GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-06-29 17:28:25.804403: W tensorflow/core/common_runtime/bfc_allocator.cc:246] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.22GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-06-29 17:28:29.056533: W tensorflow/core/common_runtime/bfc_allocator.cc:246] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.34GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
Time taken: 4.630234956741333s
2020-06-29 17:28:30.552505: W tensorflow/core/common_runtime/bfc_allocator.cc:246] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.12GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-06-29 17:28:30.615705: W tensorflow/core/common_runtime/bfc_allocator.cc:246] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.09GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-06-29 17:28:33.352142: W tensorflow/core/common_runtime/bfc_allocator.cc:246] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.34GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
Time taken: 4.438023090362549s
2020-06-29 17:28:34.797070: W tensorflow/core/common_runtime/bfc_allocator.cc:246] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.21GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-06-29 17:28:34.876055: W tensorflow/core/common_runtime/bfc_allocator.cc:246] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.22GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-06-29 17:28:37.587992: W tensorflow/core/common_runtime/bfc_allocator.cc:246] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.83GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-06-29 17:28:38.049835: W tensorflow/core/common_runtime/bfc_allocator.cc:246] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.34GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
WARNING:tensorflow:5 out of the last 8 calls to <function recreate_function.<locals>.restored_function_body at 0x7ed72fd510> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
Time taken: 4.414276361465454s

Due to the warning at the end of the output i tried to add the experimental_relax_shapes option in the tf.function for saving mb_melgan, but it still occurs:

@tf.function(experimental_relax_shapes=True,input_signature=[tf.TensorSpec(shape=[None, None, 80], dtype=tf.float32)])
    def call(self, mels):
        mb_audios = super().call(mels)
        audios = self.pqmf.synthesis(mb_audios)
        return audios

Could you please check and verify if you have the same behaviour?

[dathudeptrai]

@sujeendran can you make a colab for this ?, including a code for downloading pretrained model, saved it into pb and above code, so i can easy to check and help :D.

[sujeendran]

@dathudeptrai Hi! Sure. Here you go: https://colab.research.google.com/drive/1q065tbqWevKleMjiAJkp58cjR7LWJIj_?usp=sharing

I don't see the tf.function retracing warnings in colab, maybe because of some environment variable set. But the performance difference is as expected. Saved model is 2x slower.

[dathudeptrai]

@sujeendran @tf.function and saved model should be a same because @tf.function will convert a code to graph as saved model does. Here saved model 2x slower is the bug of Tensorflow :))). Let say that saved model tensorflow will be slow with the shape of input that saved_model nerver seen before :)). In ur notebook, if you follow bellow code you will see that saved_model run as fast as @tf.function

for j in range(5):
    for i,sentence in enumerate(sentences):
        start = time.time()
        input_ids = tf.expand_dims(tf.convert_to_tensor(processor.text_to_sequence(sentence), dtype=tf.int32), 0)

        mel_before, mel_after, duration_outputs = fastspeech.inference(
            input_ids=input_ids,
            attention_mask=tf.math.not_equal(input_ids, 0),
            speaker_ids=tf.convert_to_tensor([0], dtype=tf.int32),
            speed_ratios=tf.convert_to_tensor([1.0], dtype=tf.float32),
        )
        audio_after = mbm(tf.convert_to_tensor(mel_after,dtype=tf.float32))[0, :,0]
        end = time.time()
        # save to file
        #sf.write(f'frozen_test_{i}.wav', audio_after, 22050, "PCM_16")
        print(f"Time taken: {end-start}s")

So in the real depoyment scenario, you should warm up both fastspeech and melgan by dynamic fake-inputs has length shape from 1->1000 for example :)). I think that is ok, just take more time to warm-up model. Pls refer this issue https://github.com/tensorflow/tensorflow/issues/29075. Maybe you can create issue on tensorflow repo to ask about this bug :)).

[sujeendran]

@dathudeptrai Hmm you are right, the time does reduce from the second j loop. This is really a very bad issue with tf2 when it comes to deployment point of view. On lower-power machines the GPU overhead to load the model itself takes a long time, add to that warmup for 1000 fake samples, it will take a really long time to start any useful inferencing. I will have to see how it will behave with tflite if the conversion works in first place for fastspeech and mbmelgan. Have you tested the same?
PS. Thanks for the link to tensorflow issue!

[dathudeptrai]

@sujeendran tflite is ok, no problem but you need convert to tflite by urself first :)) it's not ez :v. For server, we can use @tf.function :))) rather than saved model ^^.