Usdx: [Feature Request] Record tone lengths and match syllables

[natowi]

Variant B to #378 Record and apply note pitches in editor

Record tone length to file similar to Yass but instead of using keys to "tap", sing/ (hum/whistle)

Start USDX -> Tools -> Editor -> -> load song-folder with lyric file without timing or just the audio file.

Start the playback and sing/hum/whistle the length of the sung words (similar to Yass).
The lengths of the tones are being recorded.
The resulting timed and pitched lyric file could then be synced with the syllabification text.

Separating pitch and tone length recording might produce better results.

[natowi]

For the Syllabification using the own voice recording USDX could make use of speech recognition, segmentation and transcription projects like SPPAS http://www.sppas.org/tutorial_114_align.html#2.0
https://github.com/nassosoassos/sail_align
https://github.com/pettarin/forced-alignment-tools
or
https://github.com/synalp/jtrans

About text-to-speech alignment
Text-to-speech" alignment is a speech processing task that is related to the well-known "speech recognition" task. However, unlike speech recognition, text-to-speech alignment requires the text that is pronounced to be already known. All that remains to do is to align the words onto the speech signal, i.e., find the milliseconds at which every word starts and ends in the audio file.
This is very useful for example to synchronize the lips animation of a cartoon character with the recording of an actor, or to build "Karaoke"-like applications, or yet to quickly find where an utterance occurs in a video database.
Text-to-speech alignment is technically easier to achieve than speech recognition, thanks to the additional information provided by the (approximate) transcription. Therefore, this technology is more precise and more efficient than speech recognition for many corpora where the sound quality is not good enough and where speech recognition fails.

[basisbit]

The mentioned/linked projects can't be used because of license issues, and don't work for all languages.
Honestly, I think this feature request ifs by far too big and out of focus of this game. Of course, others may feel free to create a feture branch for this and work on implementing this.

[natowi]

I understand that this might be too big, but at least I wanted to point out the possibilities.
Implementing parts of speech recognition/segmentation/transcription software into a karaoke software would be really neat. Maybe some student will find this post and picks this idea for his thesis -you never know...

For now, we can stay with testing out the possibilities of the existing audio-engine of USDX.

[natowi]

I close this issue as there are now other tools available.
https://github.com/UltraStar-Deluxe/USDX/issues/378