Tts: Universal ParallelWaveGAN

[george-roussos]

Hi, as Eren requested, this is an issue to follow progress of the training a larger PWGAN model for multiple speakers.

[erogol]

There it is the config I used for anyone to replica the progress here: https://discourse.mozilla.org/t/training-a-universal-vocoder/65388/14?u=erogol

{
"github_branch":"* generic_vocoder",
"restore_path":"/data2/rw/home/Trainings/LJSpeech/pwgan-generic-August-06-2020_01+07PM-29aec7c/checkpoint_275000.pth.tar",
"github_branch":"* generic_vocoder",
    "run_name": "pwgan-generic",
    "run_description": "parallel-wavegan generic vocoder",

    // AUDIO PARAMETERS
    "audio":{
        "fft_size": 1024,         // number of stft frequency levels. Size of the linear spectogram frame.
        "win_length": 1024,      // stft window length in ms.
        "hop_length": 256,       // stft window hop-lengh in ms.
        "frame_length_ms": null, // stft window length in ms.If null, 'win_length' is used.
        "frame_shift_ms": null,  // stft window hop-lengh in ms. If null, 'hop_length' is used.

        // Audio processing parameters
        "sample_rate": 24000,   // DATASET-RELATED: wav sample-rate. If different than the original data, it is resampled.
        "preemphasis": 0.0,     // pre-emphasis to reduce spec noise and make it more structured. If 0.0, no -pre-emphasis.
        "ref_level_db": 0,     // reference level db, theoretically 20db is the sound of air.

        // Silence trimming
        "do_trim_silence": true,// enable trimming of slience of audio as you load it. LJspeech (false), TWEB (false), Nancy (true)
        "trim_db": 60,          // threshold for timming silence. Set this according to your dataset.

        // MelSpectrogram parameters
        "num_mels": 80,         // size of the mel spec frame.
        "mel_fmin": 50.0,        // minimum freq level for mel-spec. ~50 for male and ~95 for female voices. Tune for dataset!!
        "mel_fmax": 7600.0,     // maximum freq level for mel-spec. Tune for dataset!!
        "spec_gain": 1.0,         // scaler value appplied after log transform of spectrogram.

        // Normalization parameters
        "signal_norm": true,    // normalize spec values. Mean-Var normalization if 'stats_path' is defined otherwise range normalization defined by the other params.
        "min_level_db": -100,   // lower bound for normalization
        "symmetric_norm": true, // move normalization to range [-1, 1]
        "max_norm": 4.0,        // scale normalization to range [-max_norm, max_norm] or [0, max_norm]
        "clip_norm": true,      // clip normalized values into the range.
        "stats_path": "/data/rw/home/Data/LibriTTS/scale_stats.npy"    // DO NOT USE WITH MULTI_SPEAKER MODEL. scaler stats file computed by 'compute_statistics.py'. If it is defined, mean-std based notmalization is used and other normalization params are ignored
    },

    // DISTRIBUTED TRAINING
    // "distributed":{
    //     "backend": "nccl",
    //     "url": "tcp:\/\/localhost:54321"
    // },

    // MODEL PARAMETERS
    "use_pqmf": true,

    // LOSS PARAMETERS
    "use_stft_loss": true,
    "use_subband_stft_loss": false,  // USE ONLY WITH MULTIBAND MODELS
    "use_mse_gan_loss": true,
    "use_hinge_gan_loss": false,
    "use_feat_match_loss": false,  // use only with melgan discriminators

    // loss weights
    "stft_loss_weight": 0.5,
    "subband_stft_loss_weight": 0.5,
    "mse_G_loss_weight": 2.5,
    "hinge_G_loss_weight": 2.5,
    "feat_match_loss_weight": 25,

    // multiscale stft loss parameters
    "stft_loss_params": {
        "n_ffts": [1024, 2048, 512],
        "hop_lengths": [120, 240, 50],
        "win_lengths": [600, 1200, 240]
    },

    // subband multiscale stft loss parameters
    "subband_stft_loss_params":{
        "n_ffts": [384, 683, 171],
        "hop_lengths": [30, 60, 10],
        "win_lengths": [150, 300, 60]
    },

    "target_loss": "avg_G_loss",  // loss value to pick the best model to save after each epoch

    // DISCRIMINATOR
    "discriminator_model": "parallel_wavegan_discriminator",
    "discriminator_model_params":{
        "num_layers": 10
    },
    "steps_to_start_discriminator": 200000,      // steps required to start GAN trainining.1

    // GENERATOR
    "generator_model": "parallel_wavegan_generator",
    "generator_model_params": {
        "upsample_factors":[4, 4, 4, 4],
        "stacks": 3,
        "num_res_blocks": 30
    },

    // DATASET
    "data_path": "/data/rw/home/Data/LibriTTS/LibriTTS/train-clean-360/",
    "feature_path": null,
    "seq_len": 25600,
    "pad_short": 2000,
    "conv_pad": 0,
    "use_noise_augment": false,
    "use_cache": true,

    "reinit_layers": [],    // give a list of layer names to restore from the given checkpoint. If not defined, it reloads all heuristically matching layers.

    // TRAINING
    "batch_size": 12,       // Batch size for training. Lower values than 32 might cause hard to learn attention. It is overwritten by 'gradual_training'.
    "sampling_rates": [16000, 22050, 24000],  // pick different sampling rate for each iteration to train generic PWGAN model.

    // VALIDATION
    "run_eval": true,
    "test_delay_epochs": 10,  //Until attention is aligned, testing only wastes computation time.
    "test_sentences_file": null,  // set a file to load sentences to be used for testing. If it is null then we use default english sentences.

    // OPTIMIZER
    "epochs": 10000,                // total number of epochs to train.
    "wd": 0.0,                // Weight decay weight.
    "gen_clip_grad": -1,      // Generator gradient clipping threshold. Apply gradient clipping if > 0
    "disc_clip_grad": -1,     // Discriminator gradient clipping threshold.
    "lr_scheduler_gen": "MultiStepLR",   // one of the schedulers from https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
    "lr_scheduler_gen_params": {
        "gamma": 0.5,
        "milestones": [100000, 200000, 300000, 400000, 500000, 600000]
    },
    "lr_scheduler_disc": "MultiStepLR",   // one of the schedulers from https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
    "lr_scheduler_disc_params": {
        "gamma": 0.5,
        "milestones": [100000, 200000, 300000, 400000, 500000, 600000]
    },
    "lr_gen": 1e-4,                  // Initial learning rate. If Noam decay is active, maximum learning rate.
    "lr_disc": 1e-4,

    // TENSORBOARD and LOGGING
    "print_step": 25,       // Number of steps to log traning on console.
    "print_eval": false,     // If True, it prints loss values for each step in eval run.
    "save_step": 25000,      // Number of training steps expected to plot training stats on TB and save model checkpoints.
    "checkpoint": true,     // If true, it saves checkpoints per "save_step"
    "tb_model_param_stats": false,     // true, plots param stats per layer on tensorboard. Might be memory consuming, but good for debugging.

    // DATA LOADING
    "num_loader_workers": 8,        // number of training data loader processes. Don't set it too big. 4-8 are good values.
    "num_val_loader_workers": 4,    // number of evaluation data loader processes.
    "eval_split_size": 10,

    // PATHS
    "output_path": "/data2/rw/home/Trainings/LJSpeech/"
}

[george-roussos]

Thanks Eren! Is this the one for the ordinary, or the larger model?

I will give it a go later this week or next week. :)

[erogol]

No problem. This is the normal model. You need to change layer values in the code or add them to the config file to run a larger model.

[SanjaESC]

How come you are using a scale_stats file? Is it okay to compute one for the vocoder? Also I've never maganed to train a multi-speaker vocoder with spec_gain: 1 but 20 worked. Is it correlated with the scale_stats file?
PS: will post my mutli band melgan results later this week.

[erogol]

scale stats is the mean spectrogram frame therefore anythong about spec. computation pertains it.

You can compute it yourself using `bin/compute_statistics.py for your dataset.

I'll share the model I've trained which will include the file for LibriTTS

[erogol]

I've started a larger model training using https://github.com/erogol/TTS_experiments/tree/generic_vocoder

[george-roussos]

I've started a larger model training using https://github.com/erogol/TTS_experiments/tree/generic_vocoder

I will start training the larger model too, because you mentioned that at 500K steps, the noise was still there and I guess it will be the same, especially seeing as so far with the small model architecture I have never been able to get rid of it. Is that okay? I will train on LibriTTS, do you want me to resample it to 22050 or keep 24000? Also, is it okay if I train with fft_size = 1100, win_length = 1100 and hop_length = 275 instead? All the TTS's I have been training use these processing values. I tried to finetune using 1024 and 256, but the attention was worse for these models.

[erogol]

"the same" to exactly which model? Do you mean a background noise or worse than that?

How large is your model?

Do you train with a single sampling rate or multiple sampling-rates?

It is better if you use the default values to be compatible with common models. But it is your call.

[george-roussos]

"the same" to exactly which model? Do you mean a background noise or worse than that?

When I trained I had background noise and then I tried with more speakers and the sound was muffled. I trained using your old ParallelWaveGAN fork.

How large is your model?

I guess it was the default model

Do you train with a single sampling rate or multiple sampling-rates?

It was single, 22050. I took LibriTTS and downsampled it.

It is better if you use the default values to be compatible with common models. But it is your call.

I will train using the default values 🙂 I will train my TTS again.

I can train the smaller model if you think it is better/smarter/better use of our time. And you can train the larger one

[SanjaESC]

I've also started a training session with the current generic_vocoder branch.
With only changes made to:

        "preemphasis": 0.98,   
        "ref_level_db": 20,    
        "mel_fmin": 40.0,       
        "mel_fmax": 7800.0,    

---

And here are some of my results from the current dev branch: GDrive

[george-roussos]

These sound nice. How long did it take to reach 675K on PWGan and what GPU? Are you training using LibriTTS?

[SanjaESC]

These sound nice. How long did it take to reach 675K on PWGan and what GPU? Are you training using LibriTTS?

~3 days on a V100 but the HDD and CPU are a bit of a bottleneck on the machine I'm using.
The datasets consists of audio mined from the games Gothic 1 - 3. Therefore the audio quality is pretty good.

[george-roussos]

These sound nice. How long did it take to reach 675K on PWGan and what GPU? Are you training using LibriTTS?

~3 days on a V100 but the HDD and CPU are a bit of a bottleneck on the machine I'm using.
The datasets consists of audio mined from the games Gothic 1 - 3. Therefore the audio quality is pretty good.

How many speakers does it have and how many minutes of speech for each?

[SanjaESC]

How many speakers does it have and how many minutes of speech for each?

Not really sure about the number of speakers, would say around ~50. Length of speech varies from few hours to just some minutes.

[george-roussos]

How many speakers does it have and how many minutes of speech for each?

Not really sure about the number of speakers, would say around ~50. Length of speech varies from few hours to just some minutes.

How many hours is the whole dataset and how many workers are you using for the data loading? I am using a V100 and 8 workers on a 30 hour dataset and I am lucky if I get 300k steps in 3 days after the discriminator kicks in, with a batch size of 10 and a length 27500.

[SanjaESC]

How many hours is the whole dataset and how many workers are you using for the data loading? I am using a V100 and 8 workers on a 30 hour dataset and I am lucky if I get 300k steps in 3 days after the discriminator kicks in, with a batch size of 10 and a length 27500.

~40 hours of data. I'm using 4 workes each with OMP_NUM_THREADS=1 as prefix when training. Using the default config values, batch_size: 6 and sq_len: 25600. But for now I'm giving up on training a pwgan vocoder. Can't get rid of the cracking and hiss sounds.

The MB MelGan vocoder yielded better results for me even though the sound produced sounds a bit more tinny/hollow.

[erogol]

I am training a larger PWGAN model. After discriminator, it raised an error when I was PTO and now I restarted it. However, from the first sight, I can tell the results look better even without discriminator training.

@SanjaESC can you share some exampled with MBMelGAN? BTW in the same paper, they use a larger MelGAN model with better results. So maybe you can try that one. They only increase the model's receptive field compared to the original MelGAN.

[george-roussos]

I am very sorry for not having time to train universal (and for the off-topic to follow). I have been testing PWGAN with single speaker, but I have had trouble with getting it to converge. Before the discriminator, it produces waveforms that sound acceptable, but when the discriminator kicks in, it sounds more metallic, especially when the speaker glottalizes. I've tried different batch sizes but it did not really help. I also tried to start training the discriminator at 100K steps instead but I got no noticeable improvements, either. Is it worth to pretrain the generator for 400K steps instead? I wonder why it does not want to converge, the PWGAN on LJSpeech sounds very good on 675K steps and my dataset has considerably better audio than LJSpeech. I did not have a lot of time to try the generic_vocoder branch.

This is the config I use and I will use the same when I train the LibriTTS PWGAN, unless something is wrong with it. I decided I will go for 1100 and 275 instead, because in my tests, 1024 and 256 give me pronunciation issues with the TTS (all other settings are the same). This config here does not have normalization stats, though (because I have not used them with my TTS). Do they help a lot? I did not want to compute them, because I wanted more variety in the speech.

I will switch to universal training this week. 🙂

{
"github_branch":"* dev",
    "run_name": "pwgan",
    "run_description": "parallel-wavegan training",

    // AUDIO PARAMETERS
    "audio":{
        "fft_size": 1100,         // number of stft frequency levels. Size of the linear spectogram frame.
        "win_length": 1100,      // stft window length in ms.
        "hop_length": 275,       // stft window hop-lengh in ms.
        "frame_length_ms": null, // stft window length in ms.If null, 'win_length' is used.
        "frame_shift_ms": null,  // stft window hop-lengh in ms. If null, 'hop_length' is used.

        // Audio processing parameters
        "sample_rate": 22050,   // DATASET-RELATED: wav sample-rate. If different than the original data, it is resampled.
        "preemphasis": 0.98,     // pre-emphasis to reduce spec noise and make it more structured. If 0.0, no -pre-emphasis.
        "ref_level_db": 0,     // reference level db, theoretically 20db is the sound of air.

        // Silence trimming
        "do_trim_silence": false,// enable trimming of slience of audio as you load it. LJspeech (false), TWEB (false), Nancy (true)
        "trim_db": 60,          // threshold for timming silence. Set this according to your dataset.

        // MelSpectrogram parameters
        "num_mels": 80,         // size of the mel spec frame.
        "mel_fmin": 0.0,        // minimum freq level for mel-spec. ~50 for male and ~95 for female voices. Tune for dataset!!
        "mel_fmax": 8000.0,     // maximum freq level for mel-spec. Tune for dataset!!
        "spec_gain": 20.0,         // scaler value appplied after log transform of spectrogram.

        // Normalization parameters
        "signal_norm": true,    // normalize spec values. Mean-Var normalization if 'stats_path' is defined otherwise range normalization defined by the other params.
        "min_level_db": -100,   // lower bound for normalization
        "symmetric_norm": true, // move normalization to range [-1, 1]
        "max_norm": 4.0,        // scale normalization to range [-max_norm, max_norm] or [0, max_norm]
        "clip_norm": true,      // clip normalized values into the range.
        "stats_path": null    // DO NOT USE WITH MULTI_SPEAKER MODEL. scaler stats file computed by 'compute_statistics.py'. If it is defined, mean-std based notmalization is used and other normalization params are ignored
    },

    // DISTRIBUTED TRAINING
    // "distributed":{
    //     "backend": "nccl",
    //     "url": "tcp:\/\/localhost:54321"
    // },

    // MODEL PARAMETERS
    "use_pqmf": true,

    // LOSS PARAMETERS
    "use_stft_loss": true,
    "use_subband_stft_loss": false,  // USE ONLY WITH MULTIBAND MODELS
    "use_mse_gan_loss": true,
    "use_hinge_gan_loss": false,
    "use_feat_match_loss": false,  // use only with melgan discriminators

    // loss weights
    "stft_loss_weight": 0.5,
    "subband_stft_loss_weight": 0.5,
    "mse_G_loss_weight": 2.5,
    "hinge_G_loss_weight": 2.5,
    "feat_match_loss_weight": 25,

    // multiscale stft loss parameters
    "stft_loss_params": {
        "n_ffts": [1024, 2048, 512],
        "hop_lengths": [120, 240, 50],
        "win_lengths": [600, 1200, 240]
    },

    // subband multiscale stft loss parameters
    "subband_stft_loss_params":{
        "n_ffts": [384, 683, 171],
        "hop_lengths": [30, 60, 10],
        "win_lengths": [150, 300, 60]
    },

    "target_loss": "avg_G_loss",  // loss value to pick the best model to save after each epoch

    // DISCRIMINATOR
    "discriminator_model": "parallel_wavegan_discriminator",
    "discriminator_model_params":{
        "num_layers": 10
    },
    "steps_to_start_discriminator": 200000,      // steps required to start GAN trainining.1

    // GENERATOR
    "generator_model": "parallel_wavegan_generator",
    "generator_model_params": {
        "upsample_factors":[5, 5, 11],
        "stacks": 3,
        "num_res_blocks": 30
    },

    // DATASET
    "data_path": "../../../dataset/wavs/",
    "feature_path": null,
    "seq_len": 27500,
    "pad_short": 2000,
    "conv_pad": 0,
    "use_noise_augment": false,
    "use_cache": true,

    "reinit_layers": [],    // give a list of layer names to restore from the given checkpoint. If not defined, it reloads all heuristically matching layers.

    // TRAINING
    "batch_size": 6,       // Batch size for training. Lower values than 32 might cause hard to learn attention. It is overwritten by 'gradual_training'.

    // VALIDATION
    "run_eval": true,
    "test_delay_epochs": 10,  //Until attention is aligned, testing only wastes computation time.
    "test_sentences_file": null,  // set a file to load sentences to be used for testing. If it is null then we use default english sentences.

    // OPTIMIZER
    "epochs": 10000,                // total number of epochs to train.
    "wd": 0.0,                // Weight decay weight.
    "gen_clip_grad": -1,      // Generator gradient clipping threshold. Apply gradient clipping if > 0
    "disc_clip_grad": -1,     // Discriminator gradient clipping threshold.
    "lr_scheduler_gen": "MultiStepLR",   // one of the schedulers from https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
    "lr_scheduler_gen_params": {
        "gamma": 0.5,
        "milestones": [100000, 200000, 300000, 400000, 500000, 600000]
    },
    "lr_scheduler_disc": "MultiStepLR",   // one of the schedulers from https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
    "lr_scheduler_disc_params": {
        "gamma": 0.5,
        "milestones": [100000, 200000, 300000, 400000, 500000, 600000]
    },
    "lr_gen": 1e-4,                  // Initial learning rate. If Noam decay is active, maximum learning rate.
    "lr_disc": 1e-4,

    // TENSORBOARD and LOGGING
    "print_step": 25,       // Number of steps to log traning on console.
    "print_eval": false,     // If True, it prints loss values for each step in eval run.
    "save_step": 10000,      // Number of training steps expected to plot training stats on TB and save model checkpoints.
    "checkpoint": true,     // If true, it saves checkpoints per "save_step"
    "tb_model_param_stats": false,     // true, plots param stats per layer on tensorboard. Might be memory consuming, but good for debugging.

    // DATA LOADING
    "num_loader_workers": 8,        // number of training data loader processes. Don't set it too big. 4-8 are good values.
    "num_val_loader_workers": 8,    // number of evaluation data loader processes.
    "eval_split_size": 10,

    // PATHS
    "output_path": "/home/"
}

[george-roussos]

@erogol are you training LibriTTS using the config in the generic_vocoder branch, or the one above and the extra layers? What batch size do you use? I think I will switch to universal now too.

[SanjaESC]

@erogol Here are some samples from the current run.

[george-roussos]

I am training the larger model now too. I am training for 275 and 1100 and for 22050 only, in hopes for saving some time, but it is taking so long. I could only fit a batch size of 8. This is the config I use.

{
"github_branch":"* generic_vocoder",
"run_name": "pwgan",
"run_description": "parallel-wavegan training",

// AUDIO PARAMETERS
"audio":{
    "fft_size": 1100,         // number of stft frequency levels. Size of the linear spectogram frame.
    "win_length": 1100,      // stft window length in ms.
    "hop_length": 275,       // stft window hop-lengh in ms.
    "frame_length_ms": null, // stft window length in ms.If null, 'win_length' is used.
    "frame_shift_ms": null,  // stft window hop-lengh in ms. If null, 'hop_length' is used.

    // Audio processing parameters
    "sample_rate": 22050,   // DATASET-RELATED: wav sample-rate. If different than the original data, it is resampled.
    "preemphasis": 0.0,     // pre-emphasis to reduce spec noise and make it more structured. If 0.0, no -pre-emphasis.
    "ref_level_db": 0,     // reference level db, theoretically 20db is the sound of air.

    // Silence trimming
    "do_trim_silence": true,// enable trimming of slience of audio as you load it. LJspeech (false), TWEB (false), Nancy (true)
    "trim_db": 48,          // threshold for timming silence. Set this according to your dataset.

    // MelSpectrogram parameters
    "num_mels": 80,         // size of the mel spec frame.
    "mel_fmin": 0.0,        // minimum freq level for mel-spec. ~50 for male and ~95 for female voices. Tune for dataset!!
    "mel_fmax": 8000.0,     // maximum freq level for mel-spec. Tune for dataset!!
    "spec_gain": 20.0,         // scaler value appplied after log transform of spectrogram.

    // Normalization parameters
    "signal_norm": true,    // normalize spec values. Mean-Var normalization if 'stats_path' is defined otherwise range normalization defined by the other params.
    "min_level_db": -100,   // lower bound for normalization
    "symmetric_norm": true, // move normalization to range [-1, 1]
    "max_norm": 4.0,        // scale normalization to range [-max_norm, max_norm] or [0, max_norm]
    "clip_norm": true,      // clip normalized values into the range.
    "stats_path": null    // DO NOT USE WITH MULTI_SPEAKER MODEL. scaler stats file computed by 'compute_statistics.py'. If it is defined, mean-std based notmalization is used and other normalization params are ignored
},

// DISTRIBUTED TRAINING
//    "distributed":{
//        "backend": "nccl",
//        "url": "tcp:\/\/localhost:54321"
//    },

// MODEL PARAMETERS
"use_pqmf": true,

// LOSS PARAMETERS
"use_stft_loss": true,
"use_subband_stft_loss": false,  // USE ONLY WITH MULTIBAND MODELS
"use_mse_gan_loss": true,
"use_hinge_gan_loss": false,
"use_feat_match_loss": false,  // use only with melgan discriminators

// loss weights
"stft_loss_weight": 0.5,
"subband_stft_loss_weight": 0.5,
"mse_G_loss_weight": 2.5,
"hinge_G_loss_weight": 2.5,
"feat_match_loss_weight": 25,

// multiscale stft loss parameters
"stft_loss_params": {
    "n_ffts": [1024, 2048, 512],
    "hop_lengths": [120, 240, 50],
    "win_lengths": [600, 1200, 240]
},

// subband multiscale stft loss parameters
"subband_stft_loss_params":{
    "n_ffts": [384, 683, 171],
    "hop_lengths": [30, 60, 10],
    "win_lengths": [150, 300, 60]
},

"target_loss": "avg_G_loss",  // loss value to pick the best model to save after each epoch

// DISCRIMINATOR
"discriminator_model": "parallel_wavegan_discriminator",
"discriminator_model_params":{
    "num_layers": 10
},
"steps_to_start_discriminator": 200000,      // steps required to start GAN trainining.1

// GENERATOR
"generator_model": "parallel_wavegan_generator",
"generator_model_params": {
    "upsample_factors":[5, 5, 11],
    "stacks": 3,
    "num_res_blocks": 30,
    "res_channels": 96,
    "gate_channels": 192,
    "skip_channels": 96
},

// DATASET
"data_path": "/home/george/libri/all",
"feature_path": null,
"seq_len": 27500,
"pad_short": 2000,
"conv_pad": 0,
"use_noise_augment": false,
"use_cache": true,

"reinit_layers": [],    // give a list of layer names to restore from the given checkpoint. If not defined, it reloads all heuristically matching layers.

// TRAINING
"batch_size": 8,       // Batch size for training. Lower values than 32 might cause hard to learn attention. It is overwritten by 'gradual_training'.
"sampling_rates": [22050],  // pick different sampling rate for each iteration to train generic PWGAN model.

// VALIDATION
"run_eval": true,
"test_delay_epochs": 10,  //Until attention is aligned, testing only wastes computation time.
"test_sentences_file": null,  // set a file to load sentences to be used for testing. If it is null then we use default english sentences.

// OPTIMIZER
"epochs": 10000,                // total number of epochs to train.
"wd": 0.0,                // Weight decay weight.
"gen_clip_grad": -1,      // Generator gradient clipping threshold. Apply gradient clipping if > 0
"disc_clip_grad": -1,     // Discriminator gradient clipping threshold.
"lr_scheduler_gen": "MultiStepLR",   // one of the schedulers from https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
"lr_scheduler_gen_params": {
    "gamma": 0.5,
    "milestones": [100000, 200000, 300000, 400000, 500000, 600000]
},
"lr_scheduler_disc": "MultiStepLR",   // one of the schedulers from https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
"lr_scheduler_disc_params": {
    "gamma": 0.5,
    "milestones": [100000, 200000, 300000, 400000, 500000, 600000]
},
"lr_gen": 1e-4,                  // Initial learning rate. If Noam decay is active, maximum learning rate.
"lr_disc": 1e-4,

// TENSORBOARD and LOGGING
"print_step": 25,       // Number of steps to log traning on console.
"print_eval": false,     // If True, it prints loss values for each step in eval run.
"save_step": 25000,      // Number of training steps expected to plot training stats on TB and save model checkpoints.
"checkpoint": true,     // If true, it saves checkpoints per "save_step"
"tb_model_param_stats": false,     // true, plots param stats per layer on tensorboard. Might be memory consuming, but good for debugging.

// DATA LOADING
"num_loader_workers": 8,        // number of training data loader processes. Don't set it too big. 4-8 are good values.
"num_val_loader_workers": 8,    // number of evaluation data loader processes.
"eval_split_size": 10,

// PATHS
"output_path": "/home/george/"
}

[george-roussos]

@erogol can you give some info on the setup you're training on? Number of GPUs and batch size?

[erogol]

batch 6 gpu 3x1080ti

but the model did not work well even though it converged better

now i started a new training with a constant sampling rate and a 6 block resnet on provided spectrograms before the vocoder as in the wavernn we use.

[george-roussos]

Thanks a lot, it helps to know. Mine was also not good. But I also trained on one GPU and it was extremely slow.

[erogol]

Maybe we need to add WaveRNN into Mozilla TTS the list of vocoders until we have something comparable. Is there anyone willing to do that?

[george-roussos]

@erogol Do you think increasing the resolutions in STFT loss may help? I think (I am not sure) I read a paper where they increase them.

[erogol]

It sounds like a good idea but what do you mean exatly by increased resolution?

[george-roussos]

Sorry, I should have been clearer. I tried to find the paper but I could not find it (I can look again). I think what they did was add more parameters in the the STFT loss parameters, so I believe they added an extra loss (so that means an extra FFT size computation, along with window size and frame shift). I don't know if it would help, but I was thinking yesterday of how I read something similar somewhere and the original PWGAN was improved when the multi-resolution loss was added. So maybe this would help, but personally I wouldn't really know what values to try.

[erogol]

Progress update... I am still continuing the last PWGAN model training and in the meantime I started FullBand-MelGAN training on LibriTTS.

[erogol]

Sorry, I should have been clearer. I tried to find the paper but I could not find it (I can look again). I think what they did was add more parameters in the the STFT loss parameters, so I believe they added an extra loss (so that means an extra FFT size computation, along with window size and frame shift). I don't know if it would help, but I was thinking yesterday of how I read something similar somewhere and the original PWGAN was improved when the multi-resolution loss was added. So maybe this would help, but personally I wouldn't really know what values to try.

makes sense. I can try this after a bit more training in the current run.

I was more thinking to replace the random noise input conditioned on the provided mel-spectrogram.

Does anyone know any paper that does something similar? At least conditioning the initial noise to the input.

[george-roussos]

The only relevant paper that I have read up on and comes to mind is this. I will look for that increased STFT loss paper again, I am 99% sure I saw the idea in writing. That, or my brain is mush from trying all the things... I might have also come up with it myself, because I have been trying to train PWGAN on a single speaker (female) and even after 600K steps the voice sounds like it has a slight case of the flu 🤧 and I have been thinking about what you said that the model might be too small to effectively capture all speech discrepancies.

How are the results with FullBand and/or the PWGAN? Is it the PWGAN with resnet? I thought FullBand myself because here it painted a good picture for multispeaker generalization, but the HooliGAN paper just says PWGAN is much more natural.

[erogol]

Full band I just started. PWGAN is still training through 400000 iters. It sounds better than before but not as better as WaveRNN.

Thanks for the links. I'll check and share my take.

[WeberJulian]

Hi,
I'm commenting here because I think it can help for vocoder compatibility.
I did some tests and it turns out that by scaling the mel-spectrogram using torch.nn.functional.interpolate with mode='bilinear', the "MultiBand-Melgan 1.45M" model can vodode without noticable quality loss mel-spectograms from 16k sample rate, even with different audio parameters (like @george-roussos's config "fft_size": 1100 "win_length": 1100 "hop_length": 275)
hope it helps

[erogol]

you mean for universal vocoder or in general. Or you are telling this for different sample-rates ?

[WeberJulian]

you mean for universal vocoder or in general.

I meant in general. I tried with the 1,45M multiband-melgan that you use in this notebook and it worked fine so I don't see why it wouldn't work on a universal vocoder.

Or you are telling this for different sample-rates ?

I'm not sure of what you mean. I tried with very different preprocessor from (sr 16k, fft_size:1100, win_length:1100, hop_length:275) and it worked on the vocoder trained with (sr 22050, fft_size: 1024, win_length: 1024, hop_length:256)

[erogol]

I tried interpolation before but it did not work as well as upsampling net. Also I found it slower compared to upsampling net.

[WeberJulian]

I tried interpolation as well before but it was with pillow an image library and the loss in quality came from the quantisation (image are encoded by ints 0-255). I also did an implementation of interpolation using scipy interpolate but it was really slow due to python interpretation overhead. Have you tried torch's interpolation ? it's pretty much instant for me.
Also, I thought it made for faster training because for upsample net you are forced to resample audio during training which is very CPU intensive and can be a huge overhead. Also, the interpolation solution doesn't force the user to finetune the vocoder model to train upsample net if his audio parameters are not supported.

[erogol]

My observations above was from pytorch interpolation but they may improved it. In anyways we can introduce an option to let user decide what to do. Maybe you can send a PR make it configurable in config.json.

However, interpolation does not solve for example inputs with different sampling rates. Does it? Because interpolation does not recover the difference between sampling rates. It should be learned by the network. Am I wrong ?

It is the main problem in universal vocoder to adapt to different sampling rate voice samples.

[WeberJulian]

I'm not sure I get what you're saying. I used two audio processors. One used for the original vocoder (22K) and one with the same audio parameters but with a different sr (16k). I resampled a wav to 16k with librosa, computed the smaller mel with the second audio processor, stretched the mel with interpolate, and passed it through the vocoder to compare the results and noticed no loss in quality

I used timeit to mesure time taken by torch interpolate to take a 16k-mel from & 4 second clip to a 22k-mel and it took only 70ms on CPU to interpolate 100 times. I don't think it would be the bottleneck in any pipeline.

timeit('torch.nn.functional.interpolate(torch.rand((1,1,80,400)), scale_factor=1.2)', setup='import torch',number=100)

I'm gonna check/share the code and samples later today of tomorrow if you think it's interesting

[erogol]

but you trained the vocoder with 16khz right? If you did, this is what I mean. So you cannot you this model with a TTS model trained with 22khz for instance. This is a problem for an universal vocoder however it is fine for a single speaker vocoder. Hope it is more clear now.

Yes I'd be happy to see your PR to enable interpolation as an alternative to upsampling net if this is what you mean.

[WeberJulian]

No, I didn't retrained nor fine-tuned the vocoder. I used the 22k vocoder from this notebook with mel interpolated from a 16k wav.

[erogol]

Ohh, then it is interesting. Can you set up a colab notebook so that I can also check? I thought that would not work, but apparently I am wrong.

[WeberJulian]

Here is the Colab. Here it works with a different sample rate but feel free to tweek other audio parameters like fft_size or num_mels. Personally, I don't here any noticeable quality loss going from the vocoded original to 16k resampled stretched vocoded one. Tell me what do you think of it.

[erogol]

great let me play with it and get back to you.

[erogol]

I've done some changes to improve the performance but yes interpolation really works :). It is great and it means that I trained new vocoder for different sampling rates for non-sense.

https://colab.research.google.com/drive/19Jifj5WjWD0-icAr20HFLnSdjRrPxbyN?usp=sharing

Let's put it in another issue and add it to somewhere in the code.

[WeberJulian]

it means that I trained new vocoder for different sampling rates for non-sense.

You and me both. I trained a Unet based mel2mel to solve the sample rate problem only to discover that it was doing (almost) identity all along.

I was thinking, maybe I could recycle this idea of mel2mel to do voice cloning. I have a TTS mono speaker that works well. instead of retraining the whole TTS model, only train mel2mel to convert mel generated from common voice sentences by the TTS to mel from common voice wavs. I just need to concatenate the speaker embedding (from CorentinJ's model) to the single vector representation in the middle of the model. I don't have much experience in DL, do you think it could work or will I be wasting my time ?

Thanks

[george-roussos]

Is the fullband melgan doing better in multispeaker than parallel wavegan? I am reading the melgan paper again and maybe I should try and reproduce their multispeaker experiment with melgan (I only tried it with PWGAN and it was not good) -- 6 speakers 10 hours each. Maybe melgan's discriminator would do better. But baseline melgan has a low MOS.

[erogol]

Don't take MOS scores granted. They are not as objective as other DL problems. The best way is to try and see.

I am running FullBand-MelGAN and it is the most promising model so far other than WaveRNN. It is still training.

[erogol]

FullbandMelgan seems to work quite well after 450K iterations. I'll train this weekend more and release it .

[WeberJulian]

FullbandMelgan seems to work quite well after 450K iterations. I'll train this weekend more and release it .

Great, thanks ! Eager to try this out

[george-roussos]

This sounds amazing. Thanks. I just binned another pwgan run. 😂 I am quite literally at my wits end. I think I trained more than 15 models this month.

[george-roussos]

Would it make sense to have Eren's FullBand Melgan with mean-var and another FullBand Melgan without mean-var? So the first one would be for single-speaker models (LJSpeech etc.) and the second one would be for TTS trained without mean-var (single speaker and a multispeaker model, if trained). I can train the FullBand without mean-var for 1024/256/22050. 🙂

[erogol]

It'd be nice to try a model without mean-var. However, I don't see how they are different for single and multi speaker cases.

[george-roussos]

Great! I meant that the vocoder would be suitable for TTS that is trained without mean-var in general. I thought of multispeaker TTS because mean-var is not applied there (?). And then, any single-speaker not trained on mean-var.

Is one GPU feasible, or does it need more? And I saw there is a fork of fullband on TTS_experiments, should I use that?

[erogol]

mean-var is specific for the vocoder model. It has nothing to do with the TTS model. So it can explicitly applied after TTS and before vocoder.

[george-roussos]

Yes, but don't both need to be trained on the same normalization scheme to be compatible? Mean-var or not. So, for example, if I tried a vocoder that was trained using scale_stats.npy and a TTS that was trained without scale_stats.npy, it wouldn't work.

[WeberJulian]

It'd be nice to try a model without mean-var. However, I don't see how they are different for single and multi speaker cases.

I think @george-roussos is referring to the comment for the stats_path parameter in the config .json file:

// DO NOT USE WITH MULTI_SPEAKER MODEL. scaler stats file computed by 'compute_statistics.py'. If it is defined, mean-std based notmalization is used and other normalization params are ignored

why is it that we can't use mean-var normalization on a multi-speaker model ?

[erogol]

@george-roussos you need to denormalize the mel-spec out from TTS model and renormalize it with vocoder stats. Then it causes no problem.

[george-roussos]

@erogol What! Of course I missed that. Thanks so much for the information. Then a vocoder trained on mean-var should do it okay.

[george-roussos]

I tried this with

mel_postnet_spec = ap._denormalize(mel_postnet_spec)
waveform = vocoder_model.inference(torch.FloatTensor(ap._normalize(mel_postnet_spec).T).unsqueeze(0))
waveform = waveform.flatten()

but it didn't work. The voice comes out like it's underwater. The TTS is trained without scale_stats and the vocoder is trained with them. But maybe I am doing it wrong.

[WeberJulian]

Isn't it because you use the same audio processor to denormalize and normalize ? (ap) Try normalizing with the vocoder's ap

[george-roussos]

I did! First I tried
waveform = vocoder_model.inference(torch.FloatTensor(ap_vocoder._normalize(mel_postnet_spec).T).unsqueeze(0))

but I get RuntimeError: [!] Mean-Var stats does not match the given feature dimensions.

[WeberJulian]

Because I feel like normalizing and denormalizing with the same ap is the identity operation

[WeberJulian]

but I get RuntimeError: [!] Mean-Var stats does not match the given feature dimensions.

Did you use the same audio parameters ? fft_size, hop_size... (exept of course normalization)

[george-roussos]

Yes, they are the same

[WeberJulian]

Yes, they are the same

because in the code, the error arrises when num_mels doesn't match

if hasattr(self, 'mel_scaler'):
                if S.shape[0] == self.num_mels:
                    return self.mel_scaler.transform(S.T).T
                elif S.shape[0] == self.fft_size / 2:
                    return self.linear_scaler.transform(S.T).T
                else:
                    raise RuntimeError(' [!] Mean-Var stats does not match the given feature dimensions.')

maybe you did a .T not at the right time causing the num_mels to not match ? try printing the shape to see if it matches

[george-roussos]

Very weird. Mels are 80 both places. And I am only doing a T in normalization.

[WeberJulian]

Very weird. Mels are 80 both places. And I am only doing a T in normalization.

Weird indeed, maybe try printing S_denorm.shape[0] and self.num_mels just after the if hasattr(self, 'mel_scaler'): line in the library to see what's happening. (in both the _normalize and _denormalize fucntion)

[george-roussos]

The shape of the spectrogram is not the same as the num_mels, after normalization. I put the print in the normalize function (which is called after the denorm) and it prints 98 instead of 80.

[WeberJulian]

Could you share the config.json of your tts model please ?

[george-roussos]

{
"github_branch":"* dev-gst-embeddings",
"model": "Tacotron2",
"run_name": "emma-fa-original",
"run_description": "tacotron2 nsb",

// AUDIO PARAMETERS
"audio":{
    // stft parameters
    "fft_size": 1100,         // number of stft frequency levels. Size of the linear spectogram frame.
    "win_length": 1100,      // stft window length in ms.
    "hop_length": 275,       // stft window hop-lengh in ms.
    "frame_length_ms": null, // stft window length in ms.If null, 'win_length' is used.
    "frame_shift_ms": null,  // stft window hop-lengh in ms. If null, 'hop_length' is used.

    // Audio processing parameters
    "sample_rate": 22050,   // DATASET-RELATED: wav sample-rate.
    "preemphasis": 0.98,     // pre-emphasis to reduce spec noise and make it more structured. If 0.0, no -pre-emphasis.
    "ref_level_db": 20.0,     // reference level db, theoretically 20db is the sound of air.

    // Silence trimming
    "do_trim_silence": false,   // enable trimming of slience of audio as you load it. LJspeech (true), TWEB (false), Nancy (true)
    "trim_db": 60,          // threshold for timming silence. Set this according to your dataset.

    // Griffin-Lim
    "power": 1.5,           // value to sharpen wav signals after GL algorithm.
    "griffin_lim_iters": 60,// #griffin-lim iterations. 30-60 is a good range. Larger the value, slower the generation.

    // MelSpectrogram parameters
    "num_mels": 80,         // size of the mel spec frame.
    "mel_fmin": 0.0,        // minimum freq level for mel-spec. ~50 for male and ~95 for female voices. Tune for dataset!!
    "mel_fmax": 8000.0,     // maximum freq level for mel-spec. Tune for dataset!!
    "spec_gain": 20.0,

    // Normalization parameters
    "signal_norm": true,    // normalize spec values. Mean-Var normalization if 'stats_path' is defined otherwise range normalization defined by the other params.
    "min_level_db": -100,   // lower bound for normalization
    "symmetric_norm": true, // move normalization to range [-1, 1]
    "max_norm": 4.0,        // scale normalization to range [-max_norm, max_norm] or [0, max_norm]
    "clip_norm": true,      // clip normalized values into the range.
    "stats_path": null //"/Users/george/emma_mean_var/scale_stats.npy"    // DO NOT USE WITH MULTI_SPEAKER MODEL. scaler stats file computed by 'compute_statistics.py'. If it is defined, mean-std based notmalization is used and other normalization params are ignored
},

// VOCABULARY PARAMETERS
// if custom character set is not defined,
// default set in symbols.py is used
// "characters":{
//     "pad": "_",
//     "eos": "~",
//     "bos": "^",
//     "characters": "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz!'(),-.:;? ",
//     "punctuations":"!'(),-.:;? ",
//     "phonemes":"iyɨʉɯuɪʏʊeøɘəɵɤoɛœɜɞʌɔæɐaɶɑɒᵻʘɓǀɗǃʄǂɠǁʛpbtdʈɖcɟkɡqɢʔɴŋɲɳnɱmʙrʀⱱɾɽɸβfvθðszʃʒʂʐçʝxɣχʁħʕhɦɬɮʋɹɻjɰlɭʎʟˈˌːˑʍwɥʜʢʡɕʑɺɧɚ˞ɫ"
// },

// DISTRIBUTED TRAINING
"distributed":{
    "backend": "nccl",
    "url": "tcp:\/\/localhost:54321"
},

"reinit_layers": [],    // give a list of layer names to restore from the given checkpoint. If not defined, it reloads all heuristically matching layers.

// TRAINING
"batch_size": 32,       // Batch size for training. Lower values than 32 might cause hard to learn attention. It is overwritten by 'gradual_training'.
"eval_batch_size":40,
"r": 7,                 // Number of decoder frames to predict per iteration. Set the initial values if gradual training is enabled.
"gradual_training": [[0, 7, 64], [1, 5, 64], [50000, 3, 32], [130000, 2, 32]], //set gradual training steps [first_step, r, batch_size]. If it is null, gradual training is disabled. For Tacotron, you might need to reduce the 'batch_size' as you proceeed.
"loss_masking": true,         // enable / disable loss masking against the sequence padding.
"ga_alpha": 10.0,        // weight for guided attention loss. If > 0, guided attention is enabled.
"apex_amp_level": null, // level of optimization with NVIDIA's apex feature for automatic mixed FP16/FP32 precision (AMP), NOTE: currently only O1 is supported, and use "O1" to activate.

// VALIDATION
"run_eval": true,
"test_delay_epochs": 3,  //Until attention is aligned, testing only wastes computation time.
"test_sentences_file": "/home/george/tests.txt",  // set a file to load sentences to be used for testing. If it is null then we use default english sentences.

// OPTIMIZER
"noam_schedule": false,        // use noam warmup and lr schedule.
"grad_clip": 1.0,              // upper limit for gradients for clipping.
"epochs": 1000,                // total number of epochs to train.
"lr": 0.0001,                  // Initial learning rate. If Noam decay is active, maximum learning rate.
"wd": 0.000001,                // Weight decay weight.
"warmup_steps": 4000,          // Noam decay steps to increase the learning rate from 0 to "lr"
"seq_len_norm": true,         // Normalize eash sample loss with its length to alleviate imbalanced datasets. Use it if your dataset is small or has skewed distribution of sequence lengths.

// TACOTRON PRENET
"memory_size": -1,             // ONLY TACOTRON - size of the memory queue used fro storing last decoder predictions for auto-regression. If < 0, memory queue is disabled and decoder only uses the last prediction frame.
"prenet_type": "original",           // "original" or "bn".
"prenet_dropout": false,       // enable/disable dropout at prenet.

// TACOTRON ATTENTION
"attention_type": "original",  // 'original' or 'graves'
"attention_heads": 4,          // number of attention heads (only for 'graves')
"attention_norm": "softmax",   // softmax or sigmoid.
"windowing": false,            // Enables attention windowing. Used only in eval mode.
"use_forward_attn": true,     // if it uses forward attention. In general, it aligns faster.
"forward_attn_mask": true,    // Additional masking forcing monotonicity only in eval mode.
"transition_agent": true,     // enable/disable transition agent of forward attention.
"location_attn": true,         // enable_disable location sensitive attention. It is enabled for TACOTRON by default.
"bidirectional_decoder": false,  // use https://arxiv.org/abs/1907.09006. Use it, if attention does not work well with your dataset.
"double_decoder_consistency": true,  // use DDC explained here https://erogol.com/solving-attention-problems-of-tts-models-with-double-decoder-consistency-draft/
"ddc_r": 7,                           // reduction rate for coarse decoder.

// STOPNET
"stopnet": true,               // Train stopnet predicting the end of synthesis.
"separate_stopnet": true,      // Train stopnet seperately if 'stopnet==true'. It prevents stopnet loss to influence the rest of the model. It causes a better model, but it trains SLOWER.

// TENSORBOARD and LOGGING
"print_step": 50,       // Number of steps to log training on console.
"tb_plot_step": 100,    // Number of steps to plot TB training figures.
"print_eval": false,     // If True, it prints intermediate loss values in evalulation.
"save_step": 10000,      // Number of training steps expected to save traninpg stats and checkpoints.
"checkpoint": true,     // If true, it saves checkpoints per "save_step"
"tb_model_param_stats": false,     // true, plots param stats per layer on tensorboard. Might be memory consuming, but good for debugging.

// DATA LOADING
"text_cleaner": "emma_cleaners",
"enable_eos_bos_chars": false, // enable/disable beginning of sentence and end of sentence chars.
"num_loader_workers": 8,        // number of training data loader processes. Don't set it too big. 4-8 are good values.
"num_val_loader_workers": 8,    // number of evaluation data loader processes.
"batch_group_size": 0,  //Number of batches to shuffle after bucketing.
"min_seq_len": 1,       // DATASET-RELATED: minimum text length to use in training
"max_seq_len": 400,     // DATASET-RELATED: maximum text length

// PATHS
"output_path": "/home/george/",

// PHONEMES
"phoneme_cache_path": "/home/george/TTS/emma_phones",  // phoneme computation is slow, therefore, it caches results in the given folder.
"use_phonemes": true,           // use phonemes instead of raw characters. It is suggested for better pronounciation.
"phoneme_language": "sv",     // depending on your target language, pick one from  https://github.com/bootphon/phonemizer#languages

// MULTI-SPEAKER and GST
"use_speaker_embedding": false,      // use speaker embedding to enable multi-speaker learning.
"use_external_speaker_embedding_file": false, // if true, forces the model to use external embedding per sample instead of nn.embeddings, that is, it supports external embeddings such as those used at: https://arxiv.org/abs /1806.04558
"external_speaker_embedding_file": "../../speakers-vctk-en.json", // if not null and use_external_speaker_embedding_file is true, it is used to load a specific embedding file and thus uses these embeddings instead of nn.embeddings, that is, it supports external embeddings such as those used at: https://arxiv.org/abs /1806.04558
"use_gst": true,                    // use global style tokens
"gst":  {                           // gst parameter if gst is enabled
    "gst_style_input": "/Users/george/Desktop/gst_ref/59.wav",
    //{"7": -0.15}, //emphasis on words
    //{"4": 0.15, "5": 0.30, "6": 0.30},
    //"/Users/george/Desktop/gst_ref/59.wav",        // Condition the style input either on a 
                                    // -> wave file [path to wave] or 
                                    // -> dictionary using the style tokens {'token1': 'value', 'token2': 'value'} example {"0": 0.15, "1": 0.15, "5": -0.15}  
                                    // with the dictionary being len(dict) <= len(gst_style_tokens).
    "gst_embedding_dim": 256,       
    "gst_num_heads": 4,
    "gst_style_tokens": 10
},  

// DATASETS
"datasets":   // List of datasets. They all merged and they get different speaker_ids.
    [
        {
            "name": "ljspeech",
            "path": "/home/george/emma_shortened_sils",
            "meta_file_train": "metadata.csv", // for vtck if list, ignore speakers id in list for train, its useful for test cloning with new speakers
            "meta_file_val": "val.csv"
        }
    ]
    }

[WeberJulian]

And can you print S_denorm.shape[1] instead of [0] ? thanks

[george-roussos]

It is 80.

[WeberJulian]

Well I think that I was correct assuming that you were missing a .T
Your 98 correspond to the time dimention not the num_mels, if you change the text by adding some words it would increase

[WeberJulian]

@erogol Do you plan to release the model today or does it need a bit more training ? Can't wait to try it :)

FullbandMelgan seems to work quite well after 450K iterations. I'll train this weekend more and release it .

[george-roussos]

I am missing it where, in denormalize? Because I did try to add it, but it did not work.

[WeberJulian]

What's the error when you do:

waveform = vocoder_model.inference(torch.FloatTensor(ap_vocoder._normalize(mel_postnet_spec.T).T).unsqueeze(0))

[george-roussos]

What's the error when you do:

waveform = vocoder_model.inference(torch.FloatTensor(ap_vocoder._normalize(mel_postnet_spec.T).T).unsqueeze(0))

If I do mel_postnet_spec = ap._denormalize(mel_postnet_spec), I get RuntimeError: Given groups=1, weight of size [80, 80, 1], expected input[1, 92, 80] to have 80 channels, but got 92 channels instead. If I do mel_postnet_spec = ap._denormalize(mel_postnet_spec.T), I get RuntimeError: [!] Mean-Var stats does not match the given feature dimensions.

[erogol]

you can see the model release under wiki/released models.

I close this issue. Thanks for your discussion and help. Feel free to comment here if you encounter with a problem.

[george-roussos]

Thank you! Can't believe we finally have a universal GAN!

For everyone else, to run it I had to take the fullband melgan model from TTS_experiments/vocoder/models and the definition in generic_utils.py (thanks Eren). I wasn't able to get the interpolation to work though, I am probably doing it wrong. @WeberJulian did you get it to work with 16khz or 22khz?

[WeberJulian]

Great thank you @erogol !
@george-roussos I'm trying this out today and if I get it to work, I'll probably do a PR :)

[erogol]

@george-roussos you should check the latest glow-TTS example which uses interpolation for sampling rate.

https://colab.research.google.com/drive/1NC4eQJFvVEqD8L4Rd8CVK25_Z-ypaBHD?usp=sharing

[WeberJulian]

So I got the generic vocoder model to work with the 21k sr TTS model from this notebook thanks to interpolation.

But I observed somthing weird, the voice sounds better if I don't denormalze with the TTS's ap and then normalize with the vocoder's ap. Am I doing something wrong @erogol ? Or is it just a matter of subjective preference ?

mel_postnet_spec = mel_postnet_spec.T
#mel_postnet_spec = ap._denormalize(mel_postnet_spec)
mel_postnet_spec = torch.tensor(mel_postnet_spec)
mel_postnet_spec = interpolate(mel_postnet_spec, (1,21050/24000)).numpy()
#mel_postnet_spec = ap_voco._normalize(mel_postnet_spec)

without norm/denorm : sample
with norm/denorm : sample

[WeberJulian]

And by the way @george-roussos I got the same error as you RuntimeError: Given groups=1, weight of size [80, 80, 1], expected input[1, 92, 80] to have 80 channels, but got 92 channels instead but it was fixed either by placing the .T like I did in the previous snippet or either by the interpolation, I'm not to sure.

[erogol]

So I got the generic vocoder model to work with the 21k sr TTS model from this notebook thanks to interpolation.

But I observed somthing weird, the voice sounds better if I don't denormalze with the TTS's ap and then normalize with the vocoder's ap. Am I doing something wrong @erogol ? Or is it just a matter of subjective preference ?

mel_postnet_spec = mel_postnet_spec.T
#mel_postnet_spec = ap._denormalize(mel_postnet_spec)
mel_postnet_spec = torch.tensor(mel_postnet_spec)
mel_postnet_spec = interpolate(mel_postnet_spec, (1,21050/24000)).numpy()
#mel_postnet_spec = ap_voco._normalize(mel_postnet_spec)

without norm/denorm : sample
with norm/denorm : sample

you need to change the display sample-rate to have a more fair comparison. That is why your samples sound deeper.

In the Glow-TTS notebook, it sounds better for me with denorm

[WeberJulian]

you need to change the display sample-rate to have a more fair comparison. That is why your samples sound deeper.

You mean the rate parameter of the IPython.display.display function ?
Yeah you're right I forgot to change it but it doesn't explain the difference between the two because both use the 24kHz vocoder model you released. I just tested it with the right display sr and I got the same difference between the two. But I guess it's just a matter of preference

[george-roussos]

Thanks guys! I was able to get interpolation to work on a TTS trained with mean-var, but no luck with TTS without mean-var :( Thanks for the vocoder Eren, appreciate it a great deal.

[george-roussos]

Hey @erogol did you train the fullband melgan on 1 GPU with a batch size of 48? I am trying to train a run without mean-var on a V100 and it is soooooooo slow... Within 9 hours it's done, like, 25K steps. If I reduce the batch size to 16 it goes back to usual GAN training speeds, but I am afraid it might not be enough for learning. What would you suggest?

[WeberJulian]

@george-roussos have you checked your CPU usage ? If you're on the P3 aws instance, you could be CPU bottlenecked.

[george-roussos]

I use a google instance. Have you tried it with 48? Is it supposed to be as fast as 16?

[WeberJulian]

Ok, no I haven't try this specifically, but I remeber beeing heavily CPU bound on a deep learning task because the v100 instance had only 8 vCPUs. So it's worth checking both CPU and GPU usage (htop and nvidia-smi respectively)

[george-roussos]

That is how many I have! I have 8. Does it need more? Oh man. 😩

[WeberJulian]

No not necessarily, it depends on the task. Just check your CPU/GPU usage to know ^^

[george-roussos]

Thanks. I tried with 16 vCPUs instead but it was the exact same time. 2K steps in one hour with a batch size of 48.

[erogol]

Hey @erogol did you train the fullband melgan on 1 GPU with a batch size of 48? I am trying to train a run without mean-var on a V100 and it is soooooooo slow... Within 9 hours it's done, like, 25K steps. If I reduce the batch size to 16 it goes back to usual GAN training speeds, but I am afraid it might not be enough for learning. What would you suggest?

yes 1GPU and 48 batch size.

I agree that you might be bottlenecked by CPU.

You can enable the memory feature cache (use_cache:True) in training to load the data faster if it is not already enabled.

[george-roussos]

Thanks, well that sucks. How many CPUs do you have? I tried with 16 but it was the exact same time (1 hour for 2000 steps). use_cache is enabled yeah

[erogol]

what are step_time and loader_time in your console logs?

[george-roussos]

It varies. It can either be something like step_time: 0.23 loader_time: 5.4650 or something like step_time: 0.25 loader_time: 0.0044. These numbers are using 8 vCPUs and 1 GPU (V100) and htop shows 100% on all CPUs. I also tried 4 and 8 number of workers in config.json but it didn't help.

[WeberJulian]

If cpu usage is 100% on all cores, you're CPU bottolnecked. Your GPU usage is probably low (you can check by typing watch -n 0.1 nvidia-smi). Increasing the number of workers won't help sorry

[george-roussos]

I thought the same (yes GPU usage is low). But when I tried 16 cores instead, it was still the same times (16 CPUs also showed 100% usage)

[SanjaESC]

It varies. It can either be something like step_time: 0.23 loader_time: 5.4650 or something like step_time: 0.25 loader_time: 0.0044. These numbers are using 8 vCPUs and 1 GPU (V100) and htop shows 100% on all CPUs. I also tried 4 and 8 number of workers in config.json but it didn't help.

I ran into the same issue with the CPU being the bottleneck. Using the prefix OMP_NUM_THREADS=1 as described here resolved my issue. I'm using 4 workers at most, most of the time i set them to 2.

[george-roussos]

Yes! This seems to do things (now it's not 100% usage CPUs). I tried it so many times but I guess I wasn't putting it in the right order. Thanks. 🥇