Tts: Is there a plan to develop a more robust front end?

[dacson]

[erogol]

font-end meaning?

[nmstoker]

@dacson could you give a little more colour about what your concerns are with the existing set up and what you'd want to see?
I've got a few simple enhancements in mind with sentence splitting and number handling in the demo server - is this was the kind of thing you were after?

[benbucksch]

I think this issue ticket is not useful, because it defines neither "robust" nor "front end". Less crashes of the server? More predictable results in terms of response speed? More consistent quality of the generated speech?

sentence splitting and number handling

Both of these would be great! +1 for that!

[m-toman]

Traditionally the text analysis part of a TTS system Is called frontend. Guess this is what's meant here.
(sorry, sent too early)
Espeak handles a good number of cases and the g2p works reasonably well - it can handle "I will read" vs "I have read", which many deep learning based systems out there can't, because most of them are trained on Individual tokens.
Füll solutions also seem to be rare, mostly Festival and flite.

There is a number of normalization solutions out there. For example https://github.com/EFord36/normalise

[nmstoker]

@m-toman thanks for mentioning Normalise - I had been looking for something like that. I'll compare it to the solution I came up with using num2words and PySBD

[erogol]

If the front-end is being the front end of a TTS system here are my comments.

We need to get rid of espeak especially for lower-end devices since it is slow and even slower our model in most cases. So for instance, if you train a model with only characters things are almost 2 times faster.

My suggestions are

• Replace phoneme extraction with another network by training it with the outputs of espeak.
• If anyone knows any faster alternative, let's switch to that.

[nmstoker]

@erogol is that speed impact in training really all attributable to espeak though? I thought the phonemes for the training dataset were worked out in the first epoch and then it re-uses them from the cache?

Perhaps I'm clutching at straws here, but could some of the reduction in training speed with phonemes relate to it using roughly twice as many symbols compared to plain text? (I forget the exact number of phonemes for English but isn't it said to be around 40-something)

I did some very basic testing of code I've written which runs sentences through espeak (but then does additional processing for improved heteronyms handling) and it came out at around 13ms per sentence and that's with the overhead of my (_non-optimised!_) heteronym code on top and independent calls to espeak each time. That kind of timing looks like it would be dwarfed by the rest of the TTS times, wouldn't it?

That said, I'm not against exploring better options, I suppose a couple of points to bear in mind with solutions are:

• How adaptable it is to different languages/accents (assuming the aim is to make a replacement that's still reasonably easy to plug in for basically any language)
• How custom words could be handled (presumably you'd want some mechanism so one didn't need to retrain just to add the latest inconsistently spoken words!)
• The quality of the output and how flaws could be handled effectively

Your first point about using espeak to generate the training sounds like it would address the language aspects above. And maybe for the second point it's just a matter of substituting the relevant set of phoneme tokens in the output with the custom phonemes (that's effectively how my heteronym code works once it has decided which of the variants is most likely to apply).

With my last point, I've generally found espeak pretty reasonable but it has some definite weak points, so anything which used data from it wholesale would just be importing those weak points. But maybe using it as a source is the best option for now and that's something to think about for later? :slightly_smiling_face:

[erogol]

yes for training it is not a problems but for inference it computes phonemes too. This is the real problem for the run-time factor. It is toooooo slow.

Interesting maybe my phonemization code is bad. 13 ms seems like a dream to me :). Did you compile espeak from the source?

• How adaptable it is to different languages/accents (assuming the aim is to make a replacement that's still reasonably easy to plug in for basically any language)

• not adaptable at all. We need to train new models for each lang.

• How custom words could be handled (presumably you'd want some mechanism so one didn't need to retrain just to add the latest inconsistently spoken words!)

• if we train models, phoneme side is a blackbox any more but it is a good counter-argument.

• The quality of the output and how flaws could be handled effectively

• you can write custom rules to recover model errors. But again it is not clear.

[m-toman]

Espeak(-ng) for me typically runs in 10ms as well but I recently installed ion Ubuntu 20 and it takes 3 seconds to run. Haven't figured out the issue yet - it's 3 seconds just for - - help as well. It does basically nothing before that so I assume it has to do with the shared libraries.

[nmstoker]

I am using espeak-ng (I should've been clearer above :slightly_smiling_face: ). I compiled it locally because I wanted to try customising words (following Josh Meyer's instructions and the docs in their repo). The times quoted are for my laptop which is an i7-7500U (several years old). I'm running Arch but I have another laptop I can try with Ubuntu soon to see if I get similar speed issues with 20.4.

For certain words I've added them to the espeak-ng dictionary, but that approach gets kind of manual and hard to deploy for TTS. My custom heteronym and punctuation code does need a custom character in the espeak-ng dictionary which it uses to tag items that'll be replaced (based on ideas here: https://github.com/espeak-ng/espeak-ng/issues/275) and it certainly seems easier to fix words when you can mark them in an advance rather than if you just get phonemes out a model and then have to retrofit the corrections (hope that makes sense!)

[m-toman]

For me this happened with the apt version as well as my own compiled version. Not sure if it's because of Ubuntu 20 or some issue with the machine. I found flite to be even faster and the code a bit easier to work with but more often delivers wrong results.
Moreover adding new languages is a pain... Did it a couple years ago and described the process here
https://sourceforge.net/p/at-flite/wiki/AddingNewLanguage/

[erogol]

my system is ubuntu 18.04 and installed by apt-get. I also have the speed issues. I'll try to compile it locally.

[benbucksch]

it's 3 seconds just for - - help as well.

• System: Ubuntu 20.04, SSD
• Version: $ apt show espeak -> Version: 1.48.04+dfsg-8build1 -- $ whereis espeak -> espeak: /usr/bin/espeak
• Speed: $ time espeak --help -> real 0m0.004s

FYI, it's 4 milli seconds for me, with the Ubuntu distro package.

[benbucksch]

Did you try check whereis espeak and ldd /usr/bin/espeak and /usr/bin/espeak --help?

[m-toman]

Suddenly on my other machine it went from a couple ms to > 3s without any modification to my espeak build.

time ./espeak-ng --path . -q --ipa -x "this is a test"
ð(0)ɪs ɪ(5)z ɐ(8) t(10)ˈɛst

real    0m3.522s
user    0m0.005s
sys     0m0.008s

time ./espeak-ng --path . -q --ipa -x --help
[...]
real    0m3.510s
user    0m0.000s
sys     0m0.006s

It links a lot of audio libs we would not need...

 ldd /usr/bin/espeak-ng
        linux-vdso.so.1 (0x00007ffce3562000)
        libespeak-ng.so.1 => /usr/lib/x86_64-linux-gnu/libespeak-ng.so.1 (0x00007f1bf2b58000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1bf2767000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1bf2548000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1bf21aa000)
        libpcaudio.so.0 => /usr/lib/x86_64-linux-gnu/libpcaudio.so.0 (0x00007f1bf1fa6000)
        libsonic.so.0 => /usr/lib/x86_64-linux-gnu/libsonic.so.0 (0x00007f1bf1da0000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f1bf3006000)
        libasound.so.2 => /usr/lib/x86_64-linux-gnu/libasound.so.2 (0x00007f1bf1a99000)
        libpulse-simple.so.0 => /usr/lib/x86_64-linux-gnu/libpulse-simple.so.0 (0x00007f1bf1894000)
        libpulse.so.0 => /usr/lib/x86_64-linux-gnu/libpulse.so.0 (0x00007f1bf1644000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1bf1440000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f1bf1238000)
        libpulsecommon-11.1.so => /usr/lib/x86_64-linux-gnu/pulseaudio/libpulsecommon-11.1.so (0x00007f1bf0fba000)
        libdbus-1.so.3 => /lib/x86_64-linux-gnu/libdbus-1.so.3 (0x00007f1bf0d6d000)
        libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1 (0x00007f1bf0b45000)
        libsystemd.so.0 => /lib/x86_64-linux-gnu/libsystemd.so.0 (0x00007f1bf08c1000)
        libwrap.so.0 => /lib/x86_64-linux-gnu/libwrap.so.0 (0x00007f1bf06b7000)
        libsndfile.so.1 => /usr/lib/x86_64-linux-gnu/libsndfile.so.1 (0x00007f1bf043e000)
        libasyncns.so.0 => /usr/lib/x86_64-linux-gnu/libasyncns.so.0 (0x00007f1bf0238000)
        libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007f1bf0034000)
        libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007f1befe2e000)
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f1befc08000)
        liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1 (0x00007f1bef9ec000)
        libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f1bef6d0000)
        libnsl.so.1 => /lib/x86_64-linux-gnu/libnsl.so.1 (0x00007f1bef4b6000)
        libFLAC.so.8 => /usr/lib/x86_64-linux-gnu/libFLAC.so.8 (0x00007f1bef23f000)
        libogg.so.0 => /usr/lib/x86_64-linux-gnu/libogg.so.0 (0x00007f1bef036000)
        libvorbis.so.0 => /usr/lib/x86_64-linux-gnu/libvorbis.so.0 (0x00007f1beee0b000)
        libvorbisenc.so.2 => /usr/lib/x86_64-linux-gnu/libvorbisenc.so.2 (0x00007f1beeb62000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f1bee947000)
        libbsd.so.0 => /lib/x86_64-linux-gnu/libbsd.so.0 (0x00007f1bee732000)
        libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f1bee51d000)

[benbucksch]

That's a good point. It may be that one of the audio libs tries to enumerate audio devices, and depending on your setup (or even an USB audio device that you connect), that may take longer. (I'm just speculating.)

[m-toman]

Found the issue with strace, would have never found that otherwise... at least it solved it for me.
I recently enabled X11 forwarding for this connection.
Likely wants to forward audio output to me and gets stuck there:

openat(AT_FDCWD, "/home/markus/.Xauthority", O_RDONLY) = 6
fstat(6, {st_mode=S_IFREG|0600, st_size=162, ...}) = 0
read(6, "\1\0\0\10gershwin\0\00212\0\22MIT-MAGIC-COOK"..., 4096) = 162
read(6, "", 4096)                       = 0
close(6)                                = 0
getsockname(5, {sa_family=AF_INET, sin_port=htons(35586), sin_addr=inet_addr("127.0.0.1")}, [124->16]) = 0
fcntl(5, F_GETFL)                       = 0x2 (flags O_RDWR)
fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK)    = 0
fcntl(5, F_SETFD, FD_CLOEXEC)           = 0
poll([{fd=5, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=5, revents=POLLOUT}])
writev(5, [{iov_base="l\0\v\0\0\0\22\0\20\0\0\0", iov_len=12}, {iov_base="", iov_len=0}, {iov_base="MIT-MAGIC-COOKIE-1", iov_len=18}, {iov_base="\0\0", iov_len=2}, {iov_base="oi\314\271\0276\200\225\270;\315n\33>:\214", iov_len=16}, {iov_base="", iov_len=0}], 6) = 48
recvfrom(5, 0x56143c231c80, 8, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=5, events=POLLIN}], 1, -1

Reconnected without it and:

time ./espeak-ng -x --ipa -q --path . "this is a test"
ð(0)ɪs ɪ(5)z ɐ(8) t(10)ˈɛst

real    0m0.009s
user    0m0.000s
sys     0m0.009s

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. You might also look our discourse page for further help. https://discourse.mozilla.org/c/tts