Tacotron2: teach tacotron 2 to sing?

[BluePrintRandom]

can tacotron 2 be taught to sing?

can tacotron 2 be taught to read a second line of text - emotion? ie angry etc?

[marshgammon]

I think thats not how its been designed to run, so it just wont work. If you're interested in singing synthesizers check out Yamaha's Vocaloid.

You could also look at putting the audio output of tacotron through melodyne, which is an vocal pitch manipulation program, though I'm not sure how well that will work.

With the different emotions, as far as I know (and I'm a beginner at this) you would need training data for each type of emotion and this would take to much time to be feasibly possible.

[big-penguin]

I know nothing about Tacotron 2 or ML in general but I've been around singing synths for years, so here's my $0.02:

As of now, the only synthesis method for human vocals (i.e. singing) that matters is concatentive synthesis; essentially a glorified sampler. Large collections of phoneme chunks are recorded, which are then picked, pitch-adjusted, and spliced together at runtime - exactly what mainstream TTS does. There are one or two HMM-based singing synthesizers, but they're practically irrelevant.

I will stop right here to say that if you're looking for a "plug-n-chug" solution where you can, say, have a script throw in lyrics + a MIDI file into an engine and have comprehensible audio in return, forget it. All current vocal/singing synthesizers require extensive editing and tweaking - known as "tuning" in the synth community - to be palatable. This can take 40+ hours just for a single song, and learning the art is just as hard as mastering the piano or violin. The latest version of Vocaloid does reduce this and try to do some of the "tuning" by itself, but you still have to know what you're doing to get top-notch results.

On Tacotron 2: converting it into a singing generator may or may not be possible. As I see it, the problem essentially boils down to this: in addition to being conditioned (is that the right word?) with the desired text to be spoken, you now also need to condition the pitch and length of each note as well. A usable dataset would need to be of consistent singing (much harder to achieve than consistent speech) covering a wide enough domain so that Tacotron "knows" what to do for everything you throw at it. Where I see this becoming a problem is that while now it needs to learn how to pronounce word X, how it should sound in relation to it's position in a sentence, etc., with singing it would have to learn that for a C pitch at 240ms, D at 240ms, F at 480ms... A much larger problem to solve. A dataset large enough to cover all these situations would be _huuuuuuge._

Perhaps a more reasonable option would be to make a hybrid between Tacotron 2 and a traditional concatentive synth; the NN takes care of the pronunciations of words and such, then spits out a monotonous, uniform-timed sample. Then traditional signal processing would take over and apply the requisite pitch- and time-shifting in order to create the final output. One could perhaps use NNs to generate the final pitch curve used within the aforementioned signal processing as well, since pitch in human singing is a complicated matter. However, I have my doubts with such a method because extensively manipulating audio in such a manner always hurts the quality. With so much traditional processing in the pipeline, the setbacks of using neural nets will most likely outweigh the gain, if any, in output quality.

I don't think it's fair to rule out ML-based singing synthesis as plausible, however. It may very well be possible to create a machine learning program which solves the task of singing effectively - It would just probably take something to motivate the teams at places like Google Deepmind to do it, which is highly unlikely.

[BluePrintRandom]

what about a sample

Sing the word "Love" - while angry

Sing the world "Love" - while happy

Sing .... while....

collect tons of samples
[label the training set]

then first try to see if the pre proccessor can listen to music and
identify a overall tone from words / tempo ques etc.

try and give it a input (text)

and it uses a GAN to generate output that matches the tone of the text
???

it is very complex but it's very imporant.

imagine if every video game ever in the future could use 1 module to
generate any voice needed, and not pack huge data sets.

(a speech could be distilled into a paragraph)

On Thu, Sep 13, 2018 at 8:30 PM qazxswedcvfrtgbnhyujmkiolp <
[email protected]> wrote:

I know nothing about Tacotron 2 or ML in general but I've been around
singing synths for years, so here's my $0.02:

As of now, the only synthesis method for human vocals (i.e. singing) that
matters is concatentive synthesis; essentially a glorified sampler. Large
collections of phoneme chunks are recorded, which are then picked,
pitch-adjusted, and spliced together at runtime - exactly what mainstream
TTS does. There are one or two HMM-based singing synthesizers, but they're
practically irrelevant.

I will stop right here to say that if you're looking for a "plug-n-chug"
solution where you can, say, have a script throw in lyrics + a MIDI file
into an engine and have comprehensible audio in return, forget it. All
current vocal/singing synthesizers require extensive editing and tweaking -
known as "tuning" in the synth community - to be palatable. This can take
40+ hours just for a single song, and learning the art is just as hard as
mastering the piano or violin. The latest version of Vocaloid does reduce
this and try to do some of the "tuning" by itself, but you still have to
know what you're doing to get top-notch results.

On Tacotron 2: converting it into a singing generator may or may not be
possible. As I see it, the problem essentially boils down to this: in
addition to being conditioned (is that the right word?) with the desired
text to be spoken, you now also need to condition the pitch and length of
each note as well. A usable dataset would need to be of consistent singing
(much harder to achieve than consistent speech) covering a wide enough
domain so that Tacotron "knows" what to do for everything you throw at it.
Where I see this becoming a problem is that while now it needs to learn how
to pronounce word X, how it should sound in relation to it's position in a
sentence, etc., with singing it would have to learn that for a C pitch at
240ms, D at 240ms, F at 480ms... A much larger problem to solve. A dataset
large enough to cover all these situations would be huuuuuuge.

Perhaps a more reasonable option would be to make a hybrid between
Tacotron 2 and a traditional concatentive synth; the NN takes care of the
pronunciations of words and such, then spits out a monotonous,
uniform-timed sample. Then traditional signal processing would take over
and apply the requisite pitch- and time-shifting in order to create the
final output. One could perhaps use NNs to generate the final pitch curve
used within the aforementioned signal processing as well, since pitch in
human singing is a complicated matter. However, I have my doubts with such
a method because extensively manipulating audio in such a manner always
hurts the quality. With so much traditional processing in the pipeline, the
setbacks of using neural nets will most likely outweigh the gain, if any,
in output quality.

I don't think it's fair to rule out ML-based singing synthesis as
plausible, however. It may very well be possible to create a machine
learning program which solves the task of singing effectively - It would
just probably take something to motivate the teams at places like Google
Deepmind to do it, which is highly unlikely.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/NVIDIA/tacotron2/issues/71#issuecomment-421220252,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AG25WQBi9ExLeq2fPHD6MftQqIFBJRYvks5uayLrgaJpZM4V9cE7
.

[will-rice]

I believe style transfer/style tokens could be a solution for this problem. This repo out of the box isn't built for this, but this is definitely possible with some modification.

Here are some papers that address similar topics:
https://arxiv.org/abs/1808.01410v1
https://arxiv.org/abs/1803.09017v1
https://arxiv.org/abs/1603.08155

[ljuvela]

I think the duration and pitch information available from the music score should actually make learning easier. There's a line of recent work on "neural parametric singing synthesis" that uses a WaveNet-like model to generate vocoder features http://www.dtic.upf.edu/~mblaauw/NPSS/

Of course this and Tacotron both would benefit from a more HiFi waveform generation method. Hybrid concatenative unit selection is also a good option, and has been mainstream in text-to-speech for a while.

[aleksas]

Was thinking of same. Using tacotron and wavenet to train singing. To use midi and lyrics as input and vocal as target result. Sure some modifications would be necessary, but to me that sounds plausible.
Midi can be generated from vocal using audio_to_midi_melodia python library built on top of Melody Extraction vamp plug-in.

[mrgloom]

Is there any concatenative TTS system available with pretrained model for test?

[rafaelvalle]

We taught Tacotron 2 how to sing: https://github.com/NVIDIA/mellotron. It comes with a pre-trained model trained on LibriTTS, which has 100+ speakers.

[BluePrintRandom]

\O/

[loretoparisi]

There are interesting developments with the WGANSing
Some cool examples here https://github.com/MTG/singing-synthesis-demos