This is not an issue and is more of a discussion. I read the WaveGrad paper today (which may be found here) and listened to the samples here, which sound very good. There seems to be an open source implementation already here with great progress. Has anyone read the paper or used this implementation?
george-roussos
All 71 comments
in the repo it says 2 days of training is enough for convergence which is a huge time reduction, however, it is unclear how fast it is on a CPU.
I skim the paper quickly. Probability diffusion and Lavegin dynamics are topics that I am not really familiar but the main idea looks similar to normalizing flows with its iterations having a resemblance to the flow steps.
erogol
on 17 Sep 2020
I saw the same! 2 days then it seems that it should be even faster than the GANs and no finetuning needed if you want less iterations for faster inference. I don't think it can use a CPU though, because the repo mentions grid search on CPU is slow, so I would imagine even 6 iterations (so lowest quality) would take longer on CPU. I also tried to see if it works with spectrograms synthesized by mozilla TTS but didn't get far. But even the 6 iters sample sounds nice
george-roussos
on 17 Sep 2020
CPU is a blocker for me to consider it since MB-MelGAN is already faster than real-time on a CPU. But 2 days of training is really impressive for a vocoder. Now I wait who will write the paper using GAN with this model and report better results :)
erogol
on 17 Sep 2020
I agree. They also mention a 2080ti for inference, so I guess it is just too expensive.
george-roussos
on 17 Sep 2020
@freds0 and I supported the TTS audio configuration on this fork here . We adapted the upsample factors of the model to our hop_lenght. We are training the model on the train-clean-100 and 360 subsets of the LibriTTS dataset and we hope to have good results in the coming days. We intend to provide the checkpoint. On the other hand, we are also trying to train WaveGlow with the TTS audio configuration. So we would have 2 SOTA vocoders.
Edresson
on 19 Sep 2020
Have you tried inference with 6 iterations and a number between 25 and 50? How does it sound? Is it fast?
george-roussos
on 21 Sep 2020
Have you tried inference with 6 iterations and a number between 25 and 50? How does it sound? Is it fast?
I tested an 11-second audio on Google Colab with 6 interactions:
GPU: 0.41608357429504395 seconds
CPU: 77.3947184085846 seconds
Here is a sample of the model trained for 1 day, in the TTS audio configuration
Edresson
on 21 Sep 2020
It sounds nice for 6. And the other repo is also updated to show inference on CPU for 6 iterations.
Is it plugged in TTS the same way as WaveRNN?
george-roussos
on 21 Sep 2020
@george-roussos Yes, integration is very simple. I used this implementation and not the one that is more active because it provided a multispeaker model for transfer learning.
I synthesized some new samples with our best multispeaker model. These voices are voices generated with random vectors, that is, they are artificial voices. The other samples were from a model that was not very good.
Edresson
on 21 Sep 2020
Do you have a notebook somewhere that you run it?
george-roussos
on 21 Sep 2020
You can download the current WaveGrad checkpoint at the link: https://drive.google.com/uc?id=1Te-9YaUirTGOa2syq2uQwBNU1vbDlUN0 (gdown https://drive.google.com/uc?id=1Te-9YaUirTGOa2syq2uQwBNU1vbDlUN0)
You must install WaveGrad with the following command: pip install git+https://github.com/freds0/wavegrad.git
Then, just use the wavegrad_predict function by passing the transposed Mel spectrogram (spec.T) and the device.
```import numpy as np
import os
import torch
import torchaudio
import time
from tqdm import tqdm
from argparse import ArgumentParser
from wavegrad.params import AttrDict, params as WAVEGRAD_PARAMS
from wavegrad.model import WaveGrad
wavegrad
WAVEGRAD_VOCODER_PATH = './'
if os.path.exists(f'{WAVEGRAD_VOCODER_PATH}/weights.pt'):
checkpoint = torch.load(f'{WAVEGRAD_VOCODER_PATH}/weights.pt')
else:
checkpoint = torch.load(WAVEGRAD_VOCODER_PATH)
WAVEGRAD_MODEL = WaveGrad(AttrDict(WAVEGRAD_PARAMS))
WAVEGRAD_MODEL.load_state_dict(checkpoint['model'])
WAVEGRAD_MODEL.eval()
print("WaveGrad load !")
WAVEGRAD_PARAMS['noise_schedule'] = [4.47739327e-06, 4.47739327e-05, 9.49513587e-04, 9.49513587e-03, 9.49513587e-02, 4.47739327e-01] # get this with noise_schedule search
def wavegrad_predict(spectrogram, device=torch.device('cuda')):
start = time.time()
# Lazy load model.
model = WAVEGRAD_MODEL.to(device)
model.params.override(WAVEGRAD_PARAMS)
with torch.no_grad():
beta = np.array(model.params.noise_schedule)
alpha = 1 - beta
alpha_cum = np.cumprod(alpha)
# Expand rank 2 tensors by adding a batch dimension.
if len(spectrogram.shape) == 2:
spectrogram = spectrogram.unsqueeze(0)
spectrogram = spectrogram.to(device)
audio = torch.randn(spectrogram.shape[0], model.params.hop_length * spectrogram.shape[-1], device=device)
noise_scale = torch.from_numpy(alpha_cum**0.5).float().unsqueeze(1).to(device)
for n in tqdm(range(len(alpha) - 1, -1, -1)):
c1 = 1 / alpha[n]**0.5
c2 = (1 - alpha[n]) / (1 - alpha_cum[n])**0.5
audio = c1 * (audio - c2 * model(audio, spectrogram, noise_scale[n]).squeeze(1))
if n > 0:
noise = torch.randn_like(audio)
sigma = ((1.0 - alpha_cum[n-1]) / (1.0 - alpha_cum[n]) * beta[n])**0.5
audio += sigma * noise
audio = torch.clamp(audio, -1.0, 1.0)
print('Wavegrad Time', time.time()-start)
return audio.cpu().numpy(), model.params.sample_rate
```
Edresson
on 21 Sep 2020
Thanks, I will try it! But do you only extract the spectrogram from TTS?
torch.FloatTensor(mel_postnet_spec.T)
np.save("spectrogram.npy", spectrogram)
george-roussos
on 21 Sep 2020
That's right.
The model is compatible with the following configuration:
// AUDIO PARAMETERS
"audio":{
// Audio processing parameters
"num_mels": 80, // size of the mel spec frame.
"fft_size": 1024, // number of stft frequency levels. Size of the linear spectogram frame.
"sample_rate": 22050, // DATASET-RELATED: wav sample-rate. If different than the original data, it is resampled.
"win_length": 1024, // stft window length in ms.
"hop_length": 256, // stft window hop-lengh in ms.
"frame_length_ms": null, // stft window length in ms.If null, 'win_length' is used.
"frame_shift_ms": null, // stft window hop-lengh in ms. If null, 'hop_length' is used.
"preemphasis": 0.98, // pre-emphasis to reduce spec noise and make it more structured. If 0.0, no -pre-emphasis.
"min_level_db": -100, // normalization range
"ref_level_db": 20, // reference level db, theoretically 20db is the sound of air.
"power": 1.5, // value to sharpen wav signals after GL algorithm.
"griffin_lim_iters": 60,// #griffin-lim iterations. 30-60 is a good range. Larger the value, slower the generation.
"stft_pad_mode": "reflect",
// Normalization parameters
"signal_norm": true, // normalize the spec values in range [0, 1]
"symmetric_norm": true, // move normalization to range [-1, 1]
"max_norm": 4.0, // scale normalization to range [-max_norm, max_norm] or [0, max_norm]
"clip_norm": true, // clip normalized values into the range.
"mel_fmin": 0.0, // minimum freq level for mel-spec. ~50 for male and ~95 for female voices. Tune for dataset!!
"mel_fmax": 8000.0, // maximum freq level for mel-spec. Tune for dataset!!
"spec_gain": 20.0,
"do_trim_silence": false, // enable trimming of slience of audio as you load it. LJspeech (false), TWEB (false), Nancy (true)
"trim_db": 60 // threshold for timming silence. Set this according to your dataset.
},
Coolio, it worked yay! Although it gives me strong vocal fry in low frequencies, so I wonder if it is my TTS or WaveGrad. Oh well!
george-roussos
on 21 Sep 2020
thanks @Edresson
erogol
on 22 Sep 2020
Coolio, it worked yay! Although it gives me strong vocal fry in low frequencies, so I wonder if it is my TTS or WaveGrad. Oh well!
You can see if the noise comes from your TTS or the vocoder by computing the mel spectogram with the audio processor and the passing it trough the vocoder.
WeberJulian
on 22 Sep 2020
Coolio, it worked yay! Although it gives me strong vocal fry in low frequencies, so I wonder if it is my TTS or WaveGrad. Oh well!
You can see if the noise comes from your TTS or the vocoder by computing the mel spectogram with the audio processor and the passing it trough the vocoder.
This is what I am doing right now. I extract the spectrogram from TTS with synthesize.py and then I pass it through WaveGrad. Are you on about something different?
george-roussos
on 22 Sep 2020
No compute the mel directly from the wav sample without using the TTS model. That way you're only testing the vocoder independently from the TTS model.
WeberJulian
on 22 Sep 2020
Oh you mean a ground truth sample? Good idea! I will try it. Thanks :)
george-roussos
on 22 Sep 2020
WaveGrad is still training, I will keep it for some more time. After I update the checkpoint here.
Edresson
on 22 Sep 2020
Thanks Edresson. What are your impressions? I think it is very promising quality-wise, but it was kind of slow when I tried to synthesize a 20-sec spectrogram on a V100. I also only trained for one day on one speaker and I used 1000 iterations.
I think I will try PWGAN with LibriTTS next week using the configuration you used for WaveGrad and maybe Eren's resnet changes.
george-roussos
on 22 Sep 2020
Did you see the Full-Band MelGAN (Universal) that Eren release recently?
Its's really fast.
I'm only training WaveGrad because of the quality :).
Edresson
on 22 Sep 2020
I did! It's very good. But there are many differences to my TTS and I think ParallelWaveGAN sounds a bit more natural.
george-roussos
on 22 Sep 2020
I ran some tests with FullBand MelGan universal, WaveGrad universal (thanks @erogol and @Edresson) and a ParallelWaveGAN (single speaker) I trained. Setup is CPU and same sentence:
FullBand Melgan:
Run-time: 13.553260803222656
Real-time factor: 0.5267899864418801
Time per step: 2.3890714941122688e-05
ParallelWaveGAN:
Run-time: 58.56965398788452
Real-time factor: 2.7854093622135445
Time per step: 0.00012632243959543506
WaveGrad:
Run-time: 131.25670504570007
Real-time factor: 2894210.104429722
Time per step: 131.25670289993286
WaveGrad ran with Edresson's final model and a custom noise schedule for my dataset, 6 iterations. With strength of preference in terms of quality, I choose
WaveGrad > ParallelWaveGAN > FullBand Melgan. However, I think pwGAN really is the sweet spot with quality and speed.
I ran this on a Mac and torch audio kept throwing an error about a library. If you run into the same problem you can run
sudo install_name_tool -change @rpath/libc++.1.dylib /usr/lib/libc++.1.dylib /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torchaudio/_torchaudio.so
george-roussos
on 24 Sep 2020
This model is universal, that is, it works for speakers not seen in training. This model is a modified version of the WaveGrad vocoder. As our TTS model was trained using a length of 256 hops, instead of 300 as reported in the original vocoder paper, we had to change the upsampling factors to WaveGrad five blocks of upsampling, changing factors 5, 5, 3, 2, 2 to 4, 4, 4, 2, 2. In addition, we trained WaveGrad with a sample rate of 22 kHz instead of 24 kHz. The model was trained in 832k steps, with a batch size of 200 and a learning rate of 0.0002, using the train-clean-100 and train-clean-360 subsets of the LibriTTS dataset.
The speed of waveform generation depends on the number of iterations of the noise schedule. In my initial tests the use of 6 iterations brings a good compromise between quality and speed. I adjusted the noise schedule in the VCTK dataset (I did this because our multi-speaker model is currently trained in this dataset). Depending on the differences in the dataset you can adjust the noise schedule for your dataset to obtain better quality and less noise in the synthesis.
Below the table with the vectors for the noise schedule are:
| Number iters | Numpy Array |
|:------------:|:----------------------------------------------------------------------------------:|
| 4 | https://drive.google.com/file/d/1o6_2LFPObxTdCE9RlDfqJZ0XfYDwUWnd/view?usp=sharing |
| 6 | https://drive.google.com/file/d/1ni_yWbNypPkl6a-rOwIADrqMmhBp6X9K/view?usp=sharing |
| 10 | https://drive.google.com/file/d/1R6JlgoQ3SgyEXN-7DoKzN4NS7lnsmlTK/view?usp=sharing |
| 20 | https://drive.google.com/file/d/1TCAdo-BDUmm9Gn6Gcy5sl-awvWoUrMKx/view?usp=sharing |
| 50 | https://drive.google.com/file/d/1tFIUbpdK45VPDNAOEIhwpfrTP_xjYwnO/view?usp=sharing |
Colab example notebook for use with Mozilla TTS: https://colab.research.google.com/drive/1oBFvgVFbR0g8f_dJ4eDW5dbT-Fk7H0kc?usp=sharing
Github repository: https://github.com/freds0/wavegrad
WavGrad's checkpoint and tensorboard logs: https://drive.google.com/drive/folders/1ZfJ_Bb2y_VHZYBZof6HVbnPw4mlDOt6v?usp=sharing
Edresson
on 14 Oct 2020
Hello.
I’m experimenting with WaveGrad implementation of https://github.com/ivanvovk/WaveGrad/. Thanks to nice support by @ivanvovk training is currently running on my public „thorsten“ german dataset.
Parameters has been adjusted to match our taco2 training params. Is the model (once it's ready) compatible with Mozilla TTS or do i need to use repo from @freds0 for Mozilla TTS usage?
Here adjusted params of WaveGrad:
"factors": [4, 4, 4, 2, 2],
"data_config": {
"sample_rate": 22050,
"n_fft": 1024,
"win_length": 1024,
"hop_length": 256,
"f_min": 0.0,
"f_max": 8000,
"n_mels": 80
},
"batch_size": 48,
"segment_length": 7168,
"lr": 1e-3,
"use_fp16": false,
Training is currently at 28 epochs and sounds not too bad.
thorstenMueller
on 3 Nov 2020
@thorstenMueller That depends, because we use some extra normalizations in the Mel Spectrograms so without those normalizations, it might not work well. I made a universal model available above, do you have any special reason to train your own?
Edresson
on 3 Nov 2020
Edresson's model works nicely for unseen speakers. However, if you do have a spare GPU, I may also suggest checking out HiFiGAN. I have been getting the best results using a GAN. There's an initial implementation on my git and one on Edresson's too and they both support Mozilla TTS spectrograms.
george-roussos
on 3 Nov 2020
Thanks @Edresson and @george-roussos for your quick feedback.
I started my own training because i thought all audio params must exactly match our taco2 model training. Maybe this was a misunderstanding. Is there an easy way to use your above checkpoints with Mozilla TTS and our dataset (i haven't found anything WaveGrad related here: https://github.com/mozilla/TTS/tree/dev/TTS/vocoder/configs). Otherwise i'll be playing around with your notebook.
When i got @domcross right he's planning to play around with HifiGAN on our german dataset.
thorstenMueller
on 4 Nov 2020
Hi @Edresson .
I copied your notebook and adjusted it (_as far as i understood your logic_) for usage with my taco2 single speaker model. But the results sounds very scratchy. Would you mind taking a look at my adjusted notebook. Maybe you've an idea what the problem might be.
https://colab.research.google.com/drive/1uZHCcmLoNckU1dDgkKspg3qXct9oRW4W?usp=sharing
thorstenMueller
on 9 Nov 2020
@thorstenMueller the WaveGrad model was trained with preemphasis=0.98, your tts model has to match those values
SanjaESC
on 9 Nov 2020
Thanks @SanjaESC .
That's the point 👍 . Our taco2 model has set preemphasis to 0.0 in config.
What would your recommendation be:
a) Train new taco2 model with preemphasis=0.98
b) Train WaveGrad vocoder with preemphasis=0.0
Hop length (256) and sr is matching (22khz).
If this i going to be offtopic i'd open a new issue or ask in mozilla discourse for help. I do not want to hijack this issue.
thorstenMueller
on 9 Nov 2020
Try finetuning the TTS with 0.98 for a few steps. I did it with mine when it was 1100/275 and wanted 1024/256 and it worked fine. :)
george-roussos
on 9 Nov 2020
@thorstenMueller Try finetuning first as suggested by @george-roussos.
I think it's best to adjust the vocoder to match the tts settings as it is the "important" part of speech generation.
SanjaESC
on 9 Nov 2020
I've been in discussion with @olafthiele and @domcross from our TTS group and we would continue our taco2 model training with adjusted params. I checked params from our model with params from @Edresson (https://github.com/mozilla/TTS/issues/518#issuecomment-696122892) and found following values to be different.
Can we/should we adjust all of these before continuing training?
| Param | Value thorsten model | Value Edresson model |
| ------------- |:-------------:| -----:|
| stft_pad_mode | not existing | reflect |
| max_norm | 1.0 | 4.0 |
| do_trim_silence | true | false |
| preemphasis | 0.0 | 0.98 |
How many steps would you recommend @george-roussos or @SanjaESC? Model has been trained with 460k steps.
thorstenMueller
on 10 Nov 2020
I've been in discussion with @olafthiele and @domcross from our TTS group and we would continue our taco2 model training with adjusted params. I checked params from our model with params from @Edresson (#518 (comment)) and found following values to be different.
Can we/should we adjust all of these before continuing training?
Param Value thorsten model Value Edresson model
stft_pad_mode not existing reflect
max_norm 1.0 4.0
do_trim_silence true false
preemphasis 0.0 0.98
How many steps would you recommend @george-roussos or @SanjaESC? Model has been trained with 460k steps.
It should definitely work within 50K steps, if not less. When I did the finetuning from 1100/275 to 1024/256 it took 50K steps to be fully okay -- before that the sound was too fast because of the smaller sizes. The trim_silence matters if you want to train with GTA features.
Good luck! 😀
george-roussos
on 10 Nov 2020
Currently, we have wavegrad implementation in Mozilla TTS dev branch for anyone who likes to give it a shot. I should say it is the best quality model we trained so far.
erogol
on 11 Nov 2020
Thanks @erogol
Having a go training with wavegrad right now. I'll monitor progress and quality, but any general suggestions on how many steps I should expect to run it for?
nmstoker
on 13 Nov 2020
Currently, we have wavegrad implementation in Mozilla TTS dev branch for anyone who likes to give it a shot. I should say it is the best quality model we trained so far.
Thanks @erogol ! I ran some quick experiments and quality is indeed much better than previous recipes I've tried here for multi-speaker TTS. It's definitely slower, ~x5 slower on GPU as compared to the fastest recipes I checked. I'm seeing rtf ranges 1.8-2.9 when running two synthesis instances on a single Titan X.
oytunturk
on 13 Nov 2020
Hi guys.
@olafthiele continued our taco2 training with adjusted params for further 50k steps (now 510k total taco2 steps). Just following two params differ:
- max_norm
- stft_pad_mode (we don't have these param in our config)
Sadly the quality didn't improve even we use new model, config and scale_stats. I try running an adjusted notebook from @Edresson . Seee my test notebook here https://colab.research.google.com/drive/1uZHCcmLoNckU1dDgkKspg3qXct9oRW4W?usp=sharing .
I'm not sure on what to do. Is the pretrained vocoder model incompatible with our taco2 traininig or did i make a mistake on adjusting the existing notebook from @Edresson .
> Setting up Audio Processor...
| > sample_rate:22050
| > num_mels:80
| > min_level_db:-100
| > frame_shift_ms:None
| > frame_length_ms:None
| > ref_level_db:20
| > fft_size:1024
| > power:1.5
| > preemphasis:0.98
| > griffin_lim_iters:60
| > signal_norm:True
| > symmetric_norm:True
| > mel_fmin:0
| > mel_fmax:8000.0
| > spec_gain:20.0
| > stft_pad_mode:reflect
| > max_norm:1.0
| > clip_norm:True
| > do_trim_silence:True
| > trim_db:60
| > do_sound_norm:True
| > stats_path:./scale_stats.npy
| > hop_length:256
| > win_length:1024
> Using model: Tacotron2
use_external_speaker_embedding_file set in config file
I've uploaded three audio samples and 1_spec.pt files. Maybe anyone has an idea what the problem might be.
thorstenMueller
on 14 Nov 2020
If you want to train from scratch, it doesn't taker longer than 2 days to reach full convergence 🙂
george-roussos
on 14 Nov 2020
I ran into a minor issue when trying to use --continue_path in training as it didn't seem to set the optimiser correctly. It was late at night so I commented out the resetting of the optimiser, which let it carry on. (similar issue seems to have come up with the other vocoders but I couldn't quite infer what the exact problem was).
Now I've gone to test the vocoder with server.py and it seems to have an issue with the noise_level not being set. I will update this with better details shortly (eg Sunday morning) but thought I'd mention it in case others had similar issues. It's posible the second problem a side-effect of my optimiser fix so I'll try digging into that tomorrow.
File "/home/neil/main/Projects/TTSNov2020/TTS/TTS/vocoder/models/wavegrad.py", line 92, in inference
sqrt_alpha_hat = self.noise_level.to(x)
AttributeError: 'NoneType' object has no attribute 'to'
I haven't quite figured out what I need to do to get the noise_level set (so the .to(x) part can work), but here is the additional info on my setup that I didn't have time to add last night.
In the end, to rule out the risk that me messing with cutting out the optimiser was behind the noise_level issue, I simply restarted my training from scratch and waited till it got to 75k steps to try again. Before restarting I updated from the repo so I got the most recent changes applied early yesterday (which had been added right after my first training run above).
The vocoder settings are below. Mostly as it is in the standard config file, except minor adjustments simply to make it match the TTS settings for the Taco2 TTS model I'm hoping to use with it.
Command line for server.py is included here but it and the output including the error are in the log file details below (if you expand the section)
python TTS/server/server.py --tts_config ../../TTSOct2020/models/neil18/neil18_2-September-27-2020_01+10AM-665f7ca/config.json --tts_checkpoint ../../TTSOct2020/models/neil18/neil18_2-September-27-2020_01+10AM-665f7ca/checkpoint_408000.pth.tar --vocoder_config /home/neil/main/Projects/TTSNov2020/models/wavegrad-neil18-November-14-2020_11+28PM-a2a142d/config.json --vocoder_checkpoint /home/neil/main/Projects/TTSNov2020/models/wavegrad-neil18-November-14-2020_11+28PM-a2a142d/best_model.pth.tar --debug False --use_cuda True
Details
Platform OS
- Linux
Python Environment
Python 3.8.6
Virtual env: Conda
Package Installation
TTS installed from source on GitHub
Click to see package list (package count: 112)
:package: Package list from Pip
| Package | Version |
| ------------- | ------------- |
|absl-py|0.9.0|
|astroid|2.4.2|
|astunparse|1.6.3|
|attrdict|2.0.1|
|attrs|19.3.0|
|audioread|2.1.8|
|bokeh|1.4.0|
|cachetools|4.1.1|
|cardboardlint|1.3.0|
|certifi|2020.11.8|
|cffi|1.14.1|
|chardet|3.0.4|
|click|7.1.2|
|clldutils|3.5.2|
|colorful|0.5.4|
|colorlog|4.2.1|
|csvw|1.7.0|
|cycler|0.10.0|
|Cython|0.29.21|
|dataclasses|0.8|
|decorator|4.4.2|
|filelock|3.0.12|
|Flask|1.1.2|
|future|0.18.2|
|fuzzywuzzy|0.18.0|
|gast|0.3.3|
|gdown|3.12.0|
|google-auth|1.20.1|
|google-auth-oauthlib|0.4.1|
|google-pasta|0.2.0|
|grpcio|1.31.0|
|h5py|2.10.0|
|idna|2.10|
|importlib-metadata|1.7.0|
|inflect|4.1.0|
|isodate|0.6.0|
|isort|4.3.21|
|itsdangerous|2.0.0a1|
|Jinja2|3.0.0a1|
|joblib|0.16.0|
|kenlm|0.0.0|
|Keras-Preprocessing|1.1.2|
|kiwisolver|1.2.0|
|lazy-object-proxy|1.4.3|
|librosa|0.7.2|
|llvmlite|0.31.0|
|Markdown|3.2.2|
|MarkupSafe|2.0.0a1|
|matplotlib|3.3.1|
|mccabe|0.6.1|
|mkl-fft|1.1.0|
|mkl-random|1.1.1|
|mkl-service|2.3.0|
|nose|1.3.7|
|numba|0.48.0|
|numpy|1.18.5|
|oauthlib|3.1.0|
|olefile|0.46|
|opt-einsum|3.3.0|
|packaging|20.4|
|phonemizer|2.2.1|
|Pillow|7.2.0|
|pip|20.2.2|
|protobuf|3.13.0|
|pyasn1|0.4.8|
|pyasn1-modules|0.2.8|
|pycparser|2.20|
|pylint|2.5.3|
|pyparsing|3.0.0a2|
|pysbd|0.3.1|
|PySocks|1.7.1|
|python-dateutil|2.8.1|
|pyworld|0.2.11.post0|
|PyYAML|5.3.1|
|regex|2020.7.14|
|requests|2.24.0|
|requests-oauthlib|1.3.0|
|resampy|0.2.2|
|rfc3986|1.4.0|
|rsa|4.6|
|scikit-learn|0.23.2|
|scipy|1.4.1|
|segments|2.1.3|
|setuptools|49.6.0.post20200814|
|six|1.15.0|
|SoundFile|0.10.3.post1|
|tabulate|0.8.7|
|tensorboard|2.3.0|
|tensorboard-plugin-wit|1.7.0|
|tensorboardX|2.1|
|tensorflow|2.3.1|
|tensorflow-estimator|2.3.0|
|termcolor|1.1.0|
|threadpoolctl|2.1.0|
|toml|0.10.1|
|torch|1.7.0|
|torchaudio|0.7.0|
|torchvision|0.8.1|
|tornado|6.0.4|
|tqdm|4.48.2|
|TTS|0.0.6+a2a142d|
|typed-ast|1.4.1|
|typing-extensions|3.7.4.3|
|umap-learn|0.4.6|
|Unidecode|0.4.20|
|uritemplate|3.0.1|
|urllib3|1.25.10|
|wavegrad|0.1.2|
|Werkzeug|1.0.1|
|wheel|0.35.1|
|wrapt|1.12.1|
|zipp|3.1.0|
Configuration
Click to see config file (lines: 117)
:page_facing_up: Contents from: /home/neil/main/Projects/TTSNov2020/models/wavegrad-neil18-November-14-2020_11+28PM-a2a142d/config.json
````javascript
{
"github_branch":"* dev",
"run_name": "wavegrad-neil18",
"run_description": "wavegrad neil18",
"audio":{
"fft_size": 1024, // number of stft frequency levels. Size of the linear spectogram frame.
"win_length": 1024, // stft window length in ms.
"hop_length": 256, // stft window hop-lengh in ms.
"frame_length_ms": null, // stft window length in ms.If null, 'win_length' is used.
"frame_shift_ms": null, // stft window hop-lengh in ms. If null, 'hop_length' is used.
// Audio processing parameters
"sample_rate": 22050, // DATASET-RELATED: wav sample-rate. If different than the original data, it is resampled.
"preemphasis": 0.99, // pre-emphasis to reduce spec noise and make it more structured. If 0.0, no -pre-emphasis.
"ref_level_db": 0, // reference level db, theoretically 20db is the sound of air.
// Silence trimming
"do_trim_silence": true,// enable trimming of slience of audio as you load it. LJspeech (false), TWEB (false), Nancy (true)
"trim_db": 60, // threshold for timming silence. Set this according to your dataset.
// MelSpectrogram parameters
"num_mels": 80, // size of the mel spec frame.
"mel_fmin": 0.0, // minimum freq level for mel-spec. ~50 for male and ~95 for female voices. Tune for dataset!!
"mel_fmax": 8000.0, // maximum freq level for mel-spec. Tune for dataset!!
"spec_gain": 1.0, // scaler value appplied after log transform of spectrogram.
// Normalization parameters
"signal_norm": true, // normalize spec values. Mean-Var normalization if 'stats_path' is defined otherwise range normalization defined by the other params.
"min_level_db": -10, // lower bound for normalization
"symmetric_norm": true, // move normalization to range [-1, 1]
"max_norm": 4.0, // scale normalization to range [-max_norm, max_norm] or [0, max_norm]
"clip_norm": true, // clip normalized values into the range.
"stats_path": null
//"stats_path": "/home/erogol/Data/libritts/LibriTTS/scale_stats_wavegrad.npy" // DO NOT USE WITH MULTI_SPEAKER MODEL. scaler stats file computed by 'compute_statistics.py'. If it is defined, mean-std based notmalization is used and other normalization params are ignored
},
// DISTRIBUTED TRAINING
"mixed_precision": true, // enable torch mixed precision training (true, false)
"distributed":{
"backend": "nccl",
"url": "tcp:\/\/localhost:54322"
},
"target_loss": "avg_wavegrad_loss", // loss value to pick the best model to save after each epoch
// MODEL PARAMETERS
"generator_model": "wavegrad",
"model_params":{
"use_weight_norm": true,
"y_conv_channels":32,
"x_conv_channels":768,
"ublock_out_channels": [512, 512, 256, 128, 128],
"dblock_out_channels": [128, 128, 256, 512],
"upsample_factors": [4, 4, 4, 2, 2],
"upsample_dilations": [
[1, 2, 1, 2],
[1, 2, 1, 2],
[1, 2, 4, 8],
[1, 2, 4, 8],
[1, 2, 4, 8]]
},
// DATASET
"data_path": "/home/neil/data/Projects/NeilTTS/neil18/wavs_wavegrad/", // root data path. It finds all wav files recursively from there.
"feature_path": null, // if you use precomputed features
"seq_len": 6144, // 24 * hop_length
"pad_short": 0, // additional padding for short wavs
"conv_pad": 0, // additional padding against convolutions applied to spectrograms
"use_noise_augment": false, // add noise to the audio signal for augmentation
"use_cache": false, // use in memory cache to keep the computed features. This might cause OOM.
"reinit_layers": [], // give a list of layer names to restore from the given checkpoint. If not defined, it reloads all heuristically matching layers.
// TRAINING
"batch_size": 96, // Batch size for training.
"train_noise_schedule":{
"min_val": 1e-6,
"max_val": 1e-2,
"num_steps": 1000
},
"test_noise_schedule":{
"min_val": 1e-6,
"max_val": 1e-2,
"num_steps": 50
},
// VALIDATION
"run_eval": true, // enable/disable evaluation run
// OPTIMIZER
"epochs": 10000, // total number of epochs to train.
"clip_grad": 1.0, // Generator gradient clipping threshold. Apply gradient clipping if > 0
"lr_scheduler": "MultiStepLR", // one of the schedulers from https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
"lr_scheduler_params": {
"gamma": 0.5,
"milestones": [100000, 200000, 300000, 400000, 500000, 600000]
},
"lr": 1e-4, // Initial learning rate. If Noam decay is active, maximum learning rate.
// TENSORBOARD and LOGGING
"print_step": 50, // Number of steps to log traning on console.
"print_eval": false, // If True, it prints loss values for each step in eval run.
"save_step": 5000, // Number of training steps expected to plot training stats on TB and save model checkpoints.
"checkpoint": true, // If true, it saves checkpoints per "save_step"
"tb_model_param_stats": true, // true, plots param stats per layer on tensorboard. Might be memory consuming, but good for debugging.
// DATA LOADING
"num_loader_workers": 4, // number of training data loader processes. Don't set it too big. 4-8 are good values.
"num_val_loader_workers": 4, // number of evaluation data loader processes.
"eval_split_size": 256,
// PATHS
"output_path": "/home/neil/main/Projects/TTSNov2020/models/"
}
````
Logfile
Click to see log file (lines: 70)
:page_facing_up: Logfile: /home/neil/main/Projects/TTSNov2020/models/server_problem.log
(tts_nov2020) [neil@ramandu TTS]$ python TTS/server/server.py --tts_config ../../TTSOct2020/models/neil18/neil18_2-September-27-2020_01+10AM-665f7ca/config.json --tts_checkpoint ../../TTSOct2020/models/neil18/neil18_2-September-27-2020_01+10AM-665f7ca/checkpoint_408000.pth.tar --vocoder_config /home/neil/main/Projects/TTSNov2020/models/wavegrad-neil18-November-14-2020_11+28PM-a2a142d/config.json --vocoder_checkpoint /home/neil/main/Projects/TTSNov2020/models/wavegrad-neil18-November-14-2020_11+28PM-a2a142d/best_model.pth.tar --debug False --use_cuda True
2020-11-15 21:20:16.134403: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/opt/kaldi/tools/openfst/lib
2020-11-15 21:20:16.134427: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Namespace(debug=False, is_wavernn_batched=False, port=5002, tts_checkpoint='../../TTSOct2020/models/neil18/neil18_2-September-27-2020_01+10AM-665f7ca/checkpoint_408000.pth.tar', tts_config='../../TTSOct2020/models/neil18/neil18_2-September-27-2020_01+10AM-665f7ca/config.json', tts_speakers=None, use_cuda=True, vocoder_checkpoint='/home/neil/main/Projects/TTSNov2020/models/wavegrad-neil18-November-14-2020_11+28PM-a2a142d/best_model.pth.tar', vocoder_config='/home/neil/main/Projects/TTSNov2020/models/wavegrad-neil18-November-14-2020_11+28PM-a2a142d/config.json', wavernn_checkpoint=None, wavernn_config=None, wavernn_lib_path=None)
> Loading TTS model ...
| > model config: ../../TTSOct2020/models/neil18/neil18_2-September-27-2020_01+10AM-665f7ca/config.json
| > checkpoint file: ../../TTSOct2020/models/neil18/neil18_2-September-27-2020_01+10AM-665f7ca/checkpoint_408000.pth.tar
> Setting up Audio Processor...
| > sample_rate:22050
| > num_mels:80
| > min_level_db:-10
| > frame_shift_ms:None
| > frame_length_ms:None
| > ref_level_db:0
| > fft_size:1024
| > power:1.8
| > preemphasis:0.99
| > griffin_lim_iters:60
| > signal_norm:True
| > symmetric_norm:True
| > mel_fmin:0
| > mel_fmax:8000.0
| > spec_gain:1.0
| > stft_pad_mode:reflect
| > max_norm:4.0
| > clip_norm:True
| > do_trim_silence:True
| > trim_db:60
| > do_sound_norm:False
| > stats_path:None
| > hop_length:256
| > win_length:1024
> Using model: Tacotron2
> model reduction factor: 1
> Generator Model: wavegrad
* Serving Flask app "server" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
[INFO] * Running on http://0.0.0.0:5002/ (Press CTRL+C to quit)
[INFO] 192.168.1.66 - - [15/Nov/2020 21:20:33] "GET / HTTP/1.1" 200 -
[INFO] 192.168.1.66 - - [15/Nov/2020 21:20:34] "GET /favicon.ico HTTP/1.1" 404 -
> Model input: How are you today?
['How are you today?']
[ERROR] Exception on /api/tts [GET]
Traceback (most recent call last):
File "/home/neil/.conda/envs/tts_nov2020/lib/python3.7/site-packages/Flask-1.1.2-py3.7.egg/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/home/neil/.conda/envs/tts_nov2020/lib/python3.7/site-packages/Flask-1.1.2-py3.7.egg/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/neil/.conda/envs/tts_nov2020/lib/python3.7/site-packages/Flask-1.1.2-py3.7.egg/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/neil/.conda/envs/tts_nov2020/lib/python3.7/site-packages/Flask-1.1.2-py3.7.egg/flask/_compat.py", line 39, in reraise
raise value
File "/home/neil/.conda/envs/tts_nov2020/lib/python3.7/site-packages/Flask-1.1.2-py3.7.egg/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/home/neil/.conda/envs/tts_nov2020/lib/python3.7/site-packages/Flask-1.1.2-py3.7.egg/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "TTS/server/server.py", line 77, in tts
data = synthesizer.tts(text)
File "/home/neil/main/Projects/TTSNov2020/TTS/TTS/server/synthesizer.py", line 154, in tts
wav = self.vocoder_model.inference(vocoder_input)
File "/home/neil/.conda/envs/tts_nov2020/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
return func(*args, **kwargs)
File "/home/neil/main/Projects/TTSNov2020/TTS/TTS/vocoder/models/wavegrad.py", line 92, in inference
sqrt_alpha_hat = self.noise_level.to(x)
AttributeError: 'NoneType' object has no attribute 'to'
[INFO] 192.168.1.66 - - [15/Nov/2020 21:20:39] "GET /api/tts?text=How%20are%20you%20today%3F HTTP/1.1" 500 -
_- generated at 21:31 on Nov 15 2020 using Gather Up tool_ :gift:
nmstoker
on 15 Nov 2020
I don't think WaveGrad functionality was added to the server yet.
To compute the noise schedule you could try adding self.vocoder_model.compute_noise_level(50, 1e-6, 1e-2) at line 92 in synthesizer.py.
def load_vocoder(self, model_file, model_config, use_cuda):
self.vocoder_config = load_config(model_config)
self.vocoder_model = setup_generator(self.vocoder_config)
self.vocoder_model.load_state_dict(torch.load(model_file, map_location="cpu")["model"])
self.vocoder_model.remove_weight_norm()
self.vocoder_model.inference_padding = 0
self.vocoder_config = load_config(model_config)
if use_cuda:
self.vocoder_model.cuda()
self.vocoder_model.eval()
self.vocoder_model.compute_noise_level(50, 1e-6, 1e-2)
There is also https://github.com/mozilla/TTS/blob/dev/TTS/bin/tune_wavegrad.py which will compute a noise schedule adapted to the trained model and your dataset. But the above should work for a start.
SanjaESC
on 15 Nov 2020
Thanks @SanjaESC - that's very kind of you, it has done the trick!
I reverted to c80225544e2fb43abbccd94148cc2045d95f8f63 as the most recent commit to allow flexibility with the beta details in compute_noise_level had changed the parameters it expected. (Edit: _I realise now I could've gone to a newer commit, so long as it was before the ones in Saturday morning_)
The quality seems good - I experimented with the number of steps; 50 to 100 seems decent, 10 has a lot more noise and above 100 wasn't noticeably doing anything for the quality. I suspect my vocoder needs to train for longer still, as there's a subtle hissing between words. I'd broken off training a little early (to test it at 75k steps) which is around 20ish hours on my 1080Ti.
Here's a sample: https://soundcloud.com/user-726556259/sherlock-wavegrad-sample
nmstoker
on 16 Nov 2020
I've started a wavegrad training on a taco2 model (taco2 "thorsten" training for 510k steps). After 27 hours (epoch 64) it stopped with following error:
--> STEP: 173/233 -- GLOBAL_STEP: 525150
| > wavegrad_loss: 0.04379 (0.04557)
| > step_time: 2.93
| > loader_time: 1.4346
| > current_lr: 0.0001
| > grad_norm: 1.336453914642334
--> STEP: 223/233 -- GLOBAL_STEP: 525200
| > wavegrad_loss: 0.03929 (0.04581)
| > step_time: 2.94
| > loader_time: 0.0206
| > current_lr: 0.0001
| > grad_norm: 2.0447957515716553
[WARNING] NaN or Inf found in input tensor.
--> TRAIN PERFORMACE -- EPOCH TIME: 688.11 sec -- GLOBAL_STEP: 525210
| > avg_wavegrad_loss: 0.04597
| > avg_loader_time: 0.47541
| > avg_step_time: 2.94022
[WARNING] NaN or Inf found in input tensor.
! Run is kept in /home/thorsten/___prj/tts/models/thorsten-wavegrad/
Traceback (most recent call last):
File "./TTS/bin/train_vocoder_wavegrad.py", line 502, in <module>
main(args)
File "./TTS/bin/train_vocoder_wavegrad.py", line 401, in main
epoch)
File "./TTS/bin/train_vocoder_wavegrad.py", line 223, in train
tb_logger.tb_model_weights(model, global_step)
File "/home/thorsten/___prj/tts/mozilla/TTS/TTS/utils/tensorboard_logger.py", line 35, in tb_model_weights
"layer{}-{}/grad".format(layer_num, name), param.grad, step)
File "/home/thorsten/___prj/tts/mozilla/lib/python3.6/site-packages/tensorboardX/writer.py", line 503, in add_histogram
histogram(tag, values, bins, max_bins=max_bins), global_step, walltime)
File "/home/thorsten/___prj/tts/mozilla/lib/python3.6/site-packages/tensorboardX/summary.py", line 210, in histogram
hist = make_histogram(values.astype(float), bins, max_bins)
File "/home/thorsten/___prj/tts/mozilla/lib/python3.6/site-packages/tensorboardX/summary.py", line 248, in make_histogram
raise ValueError('The histogram is empty, please file a bug report.')
ValueError: The histogram is empty, please file a bug report.
I've changed following two config values and started a new run. Hopefully this will help.
lr: (old): 1e-4 ---- (new): 5e-4
scheduler gamma: (old) 0.5 --- (new): 0.9
Do you have any suggestion on that?
thorstenMueller
on 22 Nov 2020
Training failed again (this time already in epoch 9).
> EPOCH: 9/10000
> TRAINING (2020-11-22 12:14:53)
--> STEP: 43/233 -- GLOBAL_STEP: 512150
| > wavegrad_loss: 0.05503 (0.05862)
| > step_time: 2.97
| > loader_time: 0.0173
| > current_lr: 0.0005
| > grad_norm: 1.5453391075134277
--> STEP: 93/233 -- GLOBAL_STEP: 512200
| > wavegrad_loss: 0.07329 (0.06146)
| > step_time: 2.99
| > loader_time: 0.0398
| > current_lr: 0.0005
| > grad_norm: 3.6150529384613037
[WARNING] NaN or Inf found in input tensor.
! Run is kept in /home/thorsten/___prj/tts/models/thorsten-wavegrad/
Traceback (most recent call last):
File "./TTS/bin/train_vocoder_wavegrad.py", line 502, in <module>
main(args)
File "./TTS/bin/train_vocoder_wavegrad.py", line 401, in main
epoch)
File "./TTS/bin/train_vocoder_wavegrad.py", line 128, in train
raise RuntimeError(f'Detected NaN loss at step {global_step}.')
RuntimeError: Detected NaN loss at step 512217.
I encountered these "NaN" issues while trying another WaveGrad implementation (see this issue: https://github.com/ivanvovk/WaveGrad/issues/8#issuecomment-706767562).
Should i upload my config.json for vocoder training or which parameters would be helpful to share for further analysis (batchsize is set to 96 right now).
@george-roussos or @erogol : Do you have any ideas? How do i know which values for lr, etc. are working?
thorstenMueller
on 22 Nov 2020
I never received this, but I have only trained using the fork here https://github.com/freds0/wavegrad
Then I took the spectrogram from TTS and passed it there
Can you check if you are using the original LR etc. values?
george-roussos
on 22 Nov 2020
Compared with original config file (https://github.com/mozilla/TTS/blob/dev/TTS/vocoder/configs/wavegrad_libritts.json) i've changed following config values so it matches our taco2 training config:
"restore_path":"/home/thorsten/___prj/tts/models/thorsten-wavegrad/wavegrad510.pth.tar",
"github_branch":"* dev",
"sample_rate": 22050, (ORIG: 24000)
"preemphasis": 0.98, (ORIG: 0.0)
"ref_level_db": 20, (ORIG: 0)
"do_sound_norm": true, (ORIG: Key in config not defined)
"mel_fmin": 0.0, (ORIG: 50.0)
"mel_fmax": 8000.0, (ORIG: 7600.0)
"spec_gain": 20.0, (ORIG: 1.0)
"max_norm": 1.0, (ORIG: 4.0)
"stats_path": "/home/thorsten/___prj/tts/models/thorsten-wavegrad/wg_scale_stats.npy",
"data_path": "/home/thorsten/___prj/tts/datasets/thorsten-de_v02/",
"gamma": 0.9, (ORIG: 0.5)
"lr": 5e-4, (ORIG: 1e-4)
"output_path": "/home/thorsten/___prj/tts/models/thorsten-wavegrad/output"
@george-roussos As far i know vocoder training should match values from taco2 training, or? So resetting all values to original values as defined in Mozilla tts repo might not be a good idea.
thorstenMueller
on 22 Nov 2020
These definitely have to match. But a NaN error means a gradient problem (please someone else correct me if wrong). Can you check if it happens with the original LR and gamma values? If everything fails, you can always use the fork I linked (it uses Mozilla standardization), then grab the spectrogram from synthesize.py and feed it. It will work the same.
george-roussos
on 22 Nov 2020
Thanks @george-roussos
Maybe i'll try with default mozilla config values (except data path) to just ensure that my setup and dependencies are working correct - not for creating a useable model.
thorstenMueller
on 22 Nov 2020
I never received this, but I have only trained using the fork here https://github.com/freds0/wavegrad
Then I took the spectrogram from TTS and passed it thereCan you check if you are using the original LR etc. values?
Hey @george-roussos .
I've decided to give repo from freds0 a try. Our taco2 model uses a hop length from 256, but in wavegrad params.py is hop_samples set to 300 with hint not to change this value. Is this a problem?
thorstenMueller
on 22 Nov 2020
Hey @george-roussos .
I've decided to give repo from freds0 a try. Our taco2 model uses a hop length from 256, but in wavegrad params.py is hop_samples set to 300 with hint not to change this value. Is this a problem?
You linked to the original repository.
freds0's fork has the needed adaptation to work with mozilla's tts.
https://github.com/freds0/wavegrad/blob/master/src/wavegrad/params.py (only the stats_path is missing i guess)
SanjaESC
on 23 Nov 2020
Sorry @SanjaESC for the confusion. My fault - too late and too many open browser tabs ;-).
Training is using the right repo. What confuses me is that Tensorboard doesn't producing audio samples (the only one shown has 0 seconds).
Will audio samples be produced on further training steps?
thorstenMueller
on 23 Nov 2020
I think that is normal. When I used it it would only produce those 1 sec audio samples
george-roussos
on 23 Nov 2020
Okay thanks. I'll keep training running.
Just one question. Should i continue my questions/discussions on WaveGrad training progress on "thorsten" dataset in Mozilla discourse to not overtax this issue thread?
thorstenMueller
on 23 Nov 2020
I'll go crazy .
I tried several runs with freds0 training, double checked all config params (_and adjusted them in freds0 if needed_) and still have no success.
I tried these steps:
- Forked freds0 repo
- Adjusted following values in code (to match our taco2 model config)
mozilla_tts_audio.py: do_trim_silence=True (was false)
mozilla_tts_audio.py: do_sound_norm=True (was false)
params.py: preemphasis=0.0 (was 0.98)
params.py: max_norm=1.0 (was 4.0)
params.py: do_trim_silence=True (was false)
- Build WaveGrad with "pip install ."
"pip list" shows wavegrad installed (venv environment):
wavegrad 0.1.2
Running preprocessing "python -m wavegrad.preprocess /path/to/dir/containing/wavs"
Created a spectogram (*.npy) file for every wav file in my wav dataset directory
python -m wavegrad /path/to/model/dir_for_this_training /path/to/dir/containing/wavs
Training is running (slow but constant) and checkpoints are written
Now i want to try if our taco2 model is compatible with the WaveGrad training:
- python -m wavegrad.inference /path/to/model/dir_for_this_training/ /tmp/spec-taco2-Das_klappt_bestimmt.npy -o /tmp/output.wav
And this gives following error:
RuntimeError: Given groups=1, weight of size [768, 80, 3], expected input[1, 112, 80] to have 80 channels, but got 112 channels instead
Is this just a format definition error in array order or a configurational mismatch?
I've no more ideas and i'm about to give up on WaveGrad.
thorstenMueller
on 24 Nov 2020
Hi, sorry, this sounds frustrating. How do you extract the spectrogram from synthesize.py? I do it like this:
It always worked okay for me. My params.py looked like this:
params = AttrDict(
# Training params
batch_size=48,
learning_rate=2e-4,
max_grad_norm=1.0,
# upsample factors
#factors=[5, 5, 3, 2, 2], # 5*5*3*2*2=300 (hop_lenght=300)
factors=[4, 4, 4, 2, 2], # 4*4*4*2*2=256 (this is necessary to be equal to hop_lenght)
# Audio params
num_mels=80,
fft_size=1024,
sample_rate=22050,
win_length=1024,
hop_length=256, # if you change that change factors
frame_length_ms=None,
frame_shift_ms=None,
preemphasis=0.98,
min_level_db=-100,
ref_level_db=20,
power=1.5,
griffin_lim_iters=60,
stft_pad_mode="reflect",
signal_norm=True,
symmetric_norm=True,
max_norm=4.0,
clip_norm=True,
mel_fmin=0.0,
mel_fmax=8000.0,
spec_gain=20.0,
do_trim_silence=False,
trim_db=60,
# Data params
crop_mel_frames=24,
# Model params
noise_schedule=np.linspace(1e-6, 0.01, 1000).tolist(),
)
Some observations: I did not perform mean-var. So I am not sure where do_sound_norm should be set to True or not, even if you are doing mean-var training. Just try adding your stats_path and let do_sound_norm set to False if it ends up not working. But this all above is assuming WaveGrad is not working correctly. If it is, then it is not relevant and your configuration is correct.
george-roussos
on 24 Nov 2020
Is this just a format definition error in array order or a configurational mismatch?
I guess you are trying to load the .npy file created during preprocess? I think it wasn't intended for inference just for training that's why you get the mismatch. You could use transpose to match the expected order.
But the easiest way is to test it on synthesized spectrograms directly from TTS. @george-roussos linked an examples above, how you can save it to an npy file. Further above is also example code to run it directly. https://github.com/mozilla/TTS/issues/518#issuecomment-696117318
SanjaESC
on 24 Nov 2020
@george-roussos :
Hi, sorry, this sounds frustrating.
Yes, a little but I knew what I was getting into, so I'll keep trying.
My code modifications looks similar as i received a comparable code snipplet from @domcross too. I started working on a pr yesterday for saving spectograms during synthesize.py.
I'll check your ideas as soon as possible, thanks.
@SanjaESC :
I guess you are trying to load the .npy file created during preprocess?
No, i saved the spectograms in synthesize.py called from tts server speaking a random phrase (no ground truth). So hopefully this is the right spectogram for further processing through WaveGrad.
Even if it is complex I don't plan to give up, so thanks for your great support guys :-).
thorstenMueller
on 24 Nov 2020
Could you share the code you are using to save it? The problem lays in the format you are exporting the spectrogram in.
SanjaESC
on 24 Nov 2020
around line 180 in synthesizer.py (dev branch):
else:
# use GL
if self.use_cuda:
postnet_output = postnet_output[0].cpu()
# Save spectogram for further processing in wavegrad
# =========================
spec_filename = "spec-" + sen.replace(" ", "_")
np.save("/tmp/" + spec_filename + ".npy", postnet_output)
# =========================
else:
postnet_output = postnet_output[0]
postnet_output = postnet_output.numpy()
wav = inv_spectrogram(postnet_output, self.ap, self.tts_config)
btw. the tts-function definition is the following:
def tts(self, text, speaker_id=None):
try this
# use GL
if self.use_cuda:
# Save spectogram for further processing in wavegrad
# =========================
spec_filename = "spec-" + sen.replace(" ", "_")
np.save("/tmp/" + spec_filename + ".npy", postnet_output[0].T)
# =========================
postnet_output = postnet_output[0].cpu()
else:
postnet_output = postnet_output[0]
postnet_output = postnet_output.numpy()
wav = inv_spectrogram(postnet_output, self.ap, self.tts_config)
Thanks @SanjaESC .
This leads to following error:
can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
I played around with adding .cpu, etc. (found some tips on googling that error) without success. Would it help to check another vocoder than GL?
thorstenMueller
on 24 Nov 2020
np.save("/tmp/" + spec_filename + ".npy", postnet_output[0].T.cpu()) should also work i guess.
Should i continue my questions/discussions on WaveGrad training progress on "thorsten" dataset in Mozilla discourse to not overtax this issue thread?
I guess at this point it would be a good idea. ^^
You are welcome to write me a PM at discourse.
SanjaESC
on 24 Nov 2020
One simple trick to run WaveGrad even faster.
- Generate a rough waveform with Griffin-Lim
- Give it to WaveGrad and use it instead of random noise as input.
It is able to generate way higher quality in 6 iterations compared to the normal run. And you can even generate good quality in 3 iterations.
erogol
on 25 Nov 2020
Might the same be true if the output of say Multi-Band MelGAN were fed to WaveGrad?
Vproject
on 26 Nov 2020
I took code snipplets from @SanjaESC and @domcross and put together this pr (https://github.com/mozilla/TTS/pull/580). Maybe it's more easy for future beginners (_like me_) to work with models raw spectrograms.
Thank you guys for your great support.
thorstenMueller
on 27 Nov 2020
One simple trick to run WaveGrad even faster.
- Generate a rough waveform with Griffin-Lim
- Give it to WaveGrad and use it instead of random noise as input.
It is able to generate way higher quality in 6 iterations compared to the normal run. And you can even generate good quality in 3 iterations.
Does the model need to be retrained ? Since the input is no longer random noise, but waveform.
Shuxinz
on 3 Dec 2020
retrain is better but it should also work otherwise.
erogol
on 3 Dec 2020
With kind permission from @SanjaESC i've send a pr for improve robustness for synthesize with wavegrad.
https://github.com/mozilla/TTS/pull/602
@nmstoker you've had same issue i ran into too (https://github.com/mozilla/TTS/issues/518#issuecomment-727254570)
AttributeError: 'NoneType' object has no attribute 'to'
I've no idea if you still encounter this problem but it should be addressed with this pr.
Thanks @SanjaESC for your 🥇 first class support and permission to pack you knowhow into a pr.
thorstenMueller
on 19 Dec 2020
Related issues
erogol
·
10Comments
haqkiemdaim
·
5Comments
domcross
·
6Comments
joshuadeisenberg
·
3Comments
xoxoxo13102020
·
5Comments
Most helpful comment
This model is universal, that is, it works for speakers not seen in training. This model is a modified version of the WaveGrad vocoder. As our TTS model was trained using a length of 256 hops, instead of 300 as reported in the original vocoder paper, we had to change the upsampling factors to WaveGrad five blocks of upsampling, changing factors 5, 5, 3, 2, 2 to 4, 4, 4, 2, 2. In addition, we trained WaveGrad with a sample rate of 22 kHz instead of 24 kHz. The model was trained in 832k steps, with a batch size of 200 and a learning rate of 0.0002, using the train-clean-100 and train-clean-360 subsets of the LibriTTS dataset.
The speed of waveform generation depends on the number of iterations of the noise schedule. In my initial tests the use of 6 iterations brings a good compromise between quality and speed. I adjusted the noise schedule in the VCTK dataset (I did this because our multi-speaker model is currently trained in this dataset). Depending on the differences in the dataset you can adjust the noise schedule for your dataset to obtain better quality and less noise in the synthesis.
Below the table with the vectors for the noise schedule are:
| Number iters | Numpy Array |
|:------------:|:----------------------------------------------------------------------------------:|
| 4 | https://drive.google.com/file/d/1o6_2LFPObxTdCE9RlDfqJZ0XfYDwUWnd/view?usp=sharing |
| 6 | https://drive.google.com/file/d/1ni_yWbNypPkl6a-rOwIADrqMmhBp6X9K/view?usp=sharing |
| 10 | https://drive.google.com/file/d/1R6JlgoQ3SgyEXN-7DoKzN4NS7lnsmlTK/view?usp=sharing |
| 20 | https://drive.google.com/file/d/1TCAdo-BDUmm9Gn6Gcy5sl-awvWoUrMKx/view?usp=sharing |
| 50 | https://drive.google.com/file/d/1tFIUbpdK45VPDNAOEIhwpfrTP_xjYwnO/view?usp=sharing |
Colab example notebook for use with Mozilla TTS: https://colab.research.google.com/drive/1oBFvgVFbR0g8f_dJ4eDW5dbT-Fk7H0kc?usp=sharing
Github repository: https://github.com/freds0/wavegrad
WavGrad's checkpoint and tensorboard logs: https://drive.google.com/drive/folders/1ZfJ_Bb2y_VHZYBZof6HVbnPw4mlDOt6v?usp=sharing