Tensorflowtts: TypeError: Can't instantiate abstract class Processor with abstract methods setup_eos_token when conbverting Fastspeech2 to Tflite

[Zak-SA]

I tried to convert fastspeech2 to tflite, first I got this error:

from tensorflow_tts.processor.ZAKspeech import symbols, _symbol_to_id
ImportError: cannot import name 'symbols' from 'tensorflow_tts.processor.ZAKspeech' (/home/zak/venv/lib/python3.8/site-packages/tensorflow_tts/processor/ZAKspeech.py)

and I solved it by adding import ZAKSPEECH_SYMBOLS as symbols. but had the issue with _symbol_to_id and it's not defined in my processor so I removed it (I don't know if this will cause issues)

now I am having this Typeerror:
line 40, in
processor = ZAKSpeechProcessor(None, "arabic_cleaners")
TypeError: Can't instantiate abstract class ZAKSpeechProcessor with abstract methods setup_eos_token

any suggestions?

appreciate the help.

[Zak-SA]

above problem solved by modifying my processor

[Zak-SA]

after fixing the above issue I am getting this error, I will try to debug it but if anyone have a suggestion it would be appreciated :
processor = ZAKSpeechProcessor(None, "arabic_cleaners")
File "", line 15, in __init__
File "/home/zak/venv/lib/python3.8/site-packages/tensorflow_tts/processor/base_processor.py", line 69, in __post_init__
self.add_symbol(
File "/home/zak/venv/lib/python3.8/site-packages/tensorflow_tts/processor/base_processor.py", line 125, in add_symbol
self.symbols.append(symbol)
AttributeError: 'str' object has no attribute 'append'

[machineko]

At first it looks like you are overwriting symbols so it is not list of symbols but just a string check if its true.

If not send a minimal code example, so I can follow this later.

[tekinek]

@Zak-SA
It is likely the symbols in your custom processor was not exported as list but a str. In case of LJSpeech processor, the code blow combines all the supported symbols into a list variable, LJSPEECH_SYMBOLS, which is referenced by self.symbols.

Your error message says self.symbolsis a str but not a list, so the underlying cause might be your corresponding YOUR_SYMBOLS is a str.

Try to change it by:
YOUR_SYMBOLS = list(YOUR_SYMBOLS)

# Export all symbols:
LJSPEECH_SYMBOLS = (
    [_pad] + list(_special) + list(_punctuation) + list(_letters) + _arpabet + [_eos]
)

[Zak-SA]

@tekinek
@machineko
my processor is almost exactly same as LJSpeech processor (except the letters)

[machineko]

@tekinek
@machineko
my processor is almost exactly same as LJSpeech processor (except the letters)

Again if u show us minimal code example we can follow your process else i can't help you

[Zak-SA]

This is the code I am using to Inference from TFlite:

import numpy as np
import yaml
import tensorflow as tf

from tensorflow_tts.processor import ZAKSpeechProcessor
from tensorflow_tts.processor.ZAKspeech import ZAKSPEECH_SYMBOLS

from tensorflow_tts.configs import FastSpeechConfig, FastSpeech2Config
from tensorflow_tts.configs import MultiBandMelGANGeneratorConfig

from tensorflow_tts.models import TFFastSpeech, TFFastSpeech2
from tensorflow_tts.models import TFMBMelGANGenerator

from IPython.display import Audio

Load the TFLite model and allocate tensors.

interpreter = tf.lite.Interpreter(model_path='fastspeech2_quant.tflite')

Get input and output tensors.

input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

Prepare input data.

def prepare_input(input_ids):
input_ids = tf.expand_dims(tf.convert_to_tensor(input_ids, dtype=tf.int32), 0)
return (input_ids,
tf.convert_to_tensor([0], tf.int32),
tf.convert_to_tensor([1.0], dtype=tf.float32),
tf.convert_to_tensor([1.0], dtype=tf.float32),
tf.convert_to_tensor([1.0], dtype=tf.float32))

Test the model on random input data.

def infer(input_text):
for x in input_details:
print(x)
for x in output_details:
print(x)
processor = ZAKSpeechProcessor(None, "arabic_cleaners")
input_ids = processor.text_to_sequence(input_text.lower())
interpreter.resize_tensor_input(input_details[0]['index'],
[1, len(input_ids)])
interpreter.resize_tensor_input(input_details[1]['index'],
[1])
interpreter.resize_tensor_input(input_details[2]['index'],
[1])
interpreter.resize_tensor_input(input_details[3]['index'],
[1])
interpreter.resize_tensor_input(input_details[4]['index'],
[1])
interpreter.allocate_tensors()
input_data = prepare_input(input_ids)
for i, detail in enumerate(input_details):
input_shape = detail['shape']
interpreter.set_tensor(detail['index'], input_data[i])

interpreter.invoke()

return (interpreter.get_tensor(output_details[0]['index']),
      interpreter.get_tensor(output_details[1]['index']))

initialize mb_melgan model

with open('../examples/multiband_melgan/conf/multiband_melgan.v1.yaml') as f:
mb_melgan_config = yaml.load(f, Loader=yaml.Loader)
mb_melgan_config = MultiBandMelGANGeneratorConfig(**mb_melgan_config["multiband_melgan_generator_params"])
mb_melgan = TFMBMelGANGenerator(config=mb_melgan_config, name='mb_melgan_generator')
mb_melgan._build()
mb_melgan.load_weights("../examples/multiband_melgan/exp/train.multiband_melgan.v1/checkpoints/generator-1000000.h5")

input_text = ""

decoder_output_tflite, mel_output_tflite = infer(input_text)
audio_before_tflite = mb_melgan(decoder_output_tflite)[0, :, 0]
audio_after_tflite = mb_melgan(mel_output_tflite)[0, :, 0]

---

And the processor is similar to LJSpeech processor

[machineko]

@Zak-SA If you want help you need to send your processor class and how you create it cause this is the problem not your TFlite conversion code.

[ZDisket]

@Zak-SA The error code contains a clue (I experienced it too)

Can't instantiate abstract class Processor with abstract methods setup_eos_token

Add a complete definition of setup_eos_tokento your processor like this returning a EOS token of your choosing (or return Noneif you don't use)
https://github.com/TensorSpeech/TensorFlowTTS/blob/0f02377f679d176ce1223c4c41750985d2d4e8c5/tensorflow_tts/processor/ljspeech.py#L158-L159

[Zak-SA]

@ZDisket
I already fixed the eos issue as I mentioned earlier, my current problem is with
AttributeError: 'str' object has no attribute 'append'

@machineko

the processor is same as LjSpeech

import os
import re

import numpy as np
import soundfile as sf
from dataclasses import dataclass
from tensorflow_tts.processor import BaseProcessor
from tensorflow_tts.utils import cleaners

valid_symbols = [

"ZH",

]

_pad = "pad"
_eos = "eos"
_punctuation = "!'(),.:;?؟؛.!"
_special = "-"
_letters = ""

Prepend "@" to ARPAbet symbols to ensure uniqueness (some are the same as uppercase letters):

_arpabet = ["@" + s for s in valid_symbols]

Export all symbols:

ZAKSPEECH_SYMBOLS = (
[_pad] + list(_special) + list(_punctuation) + list(_letters) + _arpabet + [_eos]
)

Regular expression matching text enclosed in curly braces:

_curly_re = re.compile(r"(.?){(.+?)}(.)")

@dataclass
class ZAKSpeechProcessor(BaseProcessor):
"""ZAKSpeech processor."""

cleaner_names: str = "arabic_cleaners"
positions = {
    "wave_file": 0,
    "text": 1,
    "text_norm": 2,
}
train_f_name: str = "metadata.csv"

def create_items(self):
    if self.data_dir:
        with open(
            os.path.join(self.data_dir, self.train_f_name), encoding="utf-8"
        ) as f:
            self.items = [self.split_line(self.data_dir, line, "|") for line in f]

def split_line(self, data_dir, line, split):
    parts = line.strip().split(split)
    wave_file = parts[self.positions["wave_file"]]
    text_norm = parts[self.positions["text_norm"]]
    wav_path = os.path.join(data_dir, "wavs", f"{wave_file}.wav")
    speaker_name = "ZAKspeech"
    return text_norm, wav_path, speaker_name


def setup_eos_token(self):
    return _eos

def get_one_sample(self, item):
    text, wav_path, speaker_name = item

    # normalize audio signal to be [-1, 1], soundfile already norm.
    audio, rate = sf.read(wav_path)
    audio = audio.astype(np.float32)

    # convert text to ids
    text_ids = np.asarray(self.text_to_sequence(text), np.int32)

    sample = {
        "raw_text": text,
        "text_ids": text_ids,
        "audio": audio,
        "utt_id": os.path.split(wav_path)[-1].split(".")[0],
        "speaker_name": speaker_name,
        "rate": rate,
    }

    return sample

def text_to_sequence(self, text):
    sequence = []
    # Check for curly braces and treat their contents as ARPAbet:
    while len(text):
        m = _curly_re.match(text)
        if not m:
            sequence += self._symbols_to_sequence(
                self._clean_text(text, [self.cleaner_names])
            )
            break
        sequence += self._symbols_to_sequence(
            self._clean_text(m.group(1), [self.cleaner_names])
        )
        sequence += self._arpabet_to_sequence(m.group(2))
        text = m.group(3)

def _clean_text(self, text, cleaner_names):
    for name in cleaner_names:
        cleaner = getattr(cleaners, name)
        if not cleaner:
            raise Exceptiadd ("Unknown cleaner: %s" % name)
        text = cleaner(text)
    return text

def _symbols_to_sequence(self, symbols):
    return [self.symbol_to_id[s] for s in symbols if self._should_keep_symbol(s)]

def _arpabet_to_sequence(self, text):
    return self._symbols_to_sequence(["@" + s for s in text.split()])

def _should_keep_symbol(self, s):
    return s in self.symbol_to_id and s != "_" and s != "~"

[ZDisket]

@Zak-SA

ZAKSPEECH_SYMBOLS = (
[_pad] + list(_special) + list(_punctuation) + list(_letters) + _arpabet + [_eos]
)

Have you tried wrapping _arpabet in list like this?

ZAKSPEECH_SYMBOLS = (
[_pad] + list(_special) + list(_punctuation) + list(_letters) +  list(_arpabet) + [_eos]
)

[Zak-SA]

@ZDisket I will try it now. Thanks

[Zak-SA]

@ZDisket Same error
Traceback (most recent call last):
File "ionference_tflite.py", line 76, in
decoder_output_tflite, mel_output_tflite = infer(input_text)
File "ionference_tflite.py", line 39, in infer
processor = ZAKSpeechProcessor(None, "arabic_cleaners")
File "", line 15, in __init__
File "/home/zak/venv/lib/python3.8/site-packages/tensorflow_tts/processor/base_processor.py", line 69, in __post_init__
self.add_symbol(
File "/home/zak/venv/lib/python3.8/site-packages/tensorflow_tts/processor/base_processor.py", line 125, in add_symbol
self.symbols.append(symbol)
AttributeError: 'str' object has no attribute 'append'

[dathudeptrai]

@Zak-SA after create ur processor, can you print (processor.symbols) and pass it here ?

[Zak-SA]

@dathudeptrai sorry for the stupid question, but how to do so? the only output I have is the mapper

[dathudeptrai]

@dathudeptrai sorry for the stupid question, but how to do so? the only output I have is the mapper

processor = ZAKProcessor(data_dir=None, loaded_mapper_path="ur_mapper.json")
print(processor.symbols)

[Zak-SA]

@dathudeptrai got it. Will print it and share it here. Thanks

[machineko]

You have a bug in creating symbols you never pass them to the class and dont load them from a file
your code

processor = ZAKSpeechProcessor(None, "arabic_cleaners")

#should be 
processor = ZAKSpeechProcessor(data_dir=None, symbols=ZAKSPEECH_SYMBOLS, cleaner_names="arabic_cleaners")

[Zak-SA]

@machineko that worked, the append to str error is gone, now I am getting a different error

RuntimeError: tensorflow/lite/kernels/reshape.cc:55 stretch_dim != -1 (0 != -1)Node number 83 (RESHAPE) failed to prepare.

[machineko]

@Zak-SA Nice i'm not sure about other errors as I don't write TFLite code (@dathudeptrai) but you should ask this question in another issue and close this so people will find response for same problems easier :)

[dathudeptrai]

@machineko that worked, the append to str error is gone, now I am getting a different error

RuntimeError: tensorflow/lite/kernels/reshape.cc:55 stretch_dim != -1 (0 != -1)Node number 83 (RESHAPE) failed to prepare.

can you create a new issue, if you can, please provide us ur colab to reproduce ur bug :)).

[Zak-SA]

@dathudeptrai I will, thanks to everyone here for all the help, you are doing a great job. The synthesis results are amazing and probable the best quality Ive ever heard.

I have two more steps to get great TTS, verify the conversion to TFlite and use the existing Android samply to build the android APK.
wouldn't get this far without all of you guys helping.

:+1:

[dathudeptrai]

@dathudeptrai I will, thanks to everyone here for all the help, you are doing a great job. The synthesis results are amazing and probable the best quality Ive ever heard.

I have two more steps to get great TTS, verify the conversion to TFlite and use the existing Android samply to build the android APK.
wouldn't get this far without all of you guys helping.

so let give us a star and share our work with ur friends :)).

BTW, i'm not a person write tflite code or android sample :). other contributor did it :)), i do not know if i can help you but i will try :)).

[Zak-SA]

@dathudeptrai you are definitely getting many stars from me. I am recommending your work to organizations and many linguistic professionals who already admiring your work.
Keep the great work. 👍