Tacotron2: How to use r9y9's pretrained wavenet vocoder to synthesize speech?

[NewEricWang]

Hi, I use r9y9's pre-trained wavenet vocoder to synthesize speech, but I get noisy audio(
20180510_mixture_lj_checkpoint_step000320000_ema.wav.zip
). My GL-vocoder generated audio is normal(
test.wav.zip
).

I use the following code to save the mel-spectrum generated by tacotron2:
np.save("./works/test_mel.npy", mel_outputs_postnet.data.cpu().numpy()[0].T, allow_pickle=False)

And I use the following code to convert the mel-spectrum generated by tacotron2 into one matched with r9y9's pre-trained wavenet vocoder:
filter_length = 1024
hop_length = 256
win_length = 1024
sampling_rate = 22050
mel_fmin = 125
mel_fmax = 7600

taco_stft_other = TacotronSTFT(
filter_length, hop_length, win_length,
sampling_rate=sampling_rate, mel_fmin=mel_fmin, mel_fmax=mel_fmax)

#Project from Spectrogram to r9y9's WaveNet Mel-Spectrogram
mel_minmax = taco_stft_other.spectral_normalize(
torch.matmul(taco_stft_other.mel_basis, spec_from_mel))

np.save("./works/test_mel_wnet.npy", mel_minmax.data.cpu().numpy()[0].T, allow_pickle=False)

And the code below is used to synthesize speech:
python synthesis.py --preset=./pre-trained_models/20180510_mixture_lj_checkpoint_step000320000_ema.json --conditional=/qwork1/hlw74/ttswork/tacotron2/works/test_mel_wnet.1.npy ./pre-trained_models/20180510_mixture_lj_checkpoint_step000320000_ema.pth ./works

[rafaelvalle]

After changing the mel-basis, you have to apply the same pre-processing as in Ryuchi's repo, something like:

# methods from from Ryuchi's repo
mel = audio._amp_to_db(mel) - hparams.ref_level_db
if not hparams.allow_clipping_in_normalization:
    assert mel.max() <= 0 and mel.min() - hparams.min_level_db >= 0
mel = audio._normalize(mel)

[NewEricWang]

@rafaelvalle , Thank you for your professional guidance！ I add your given code above in synthesis.py, but the speech that I get is still noise(
20180510_mixture_lj_checkpoint_step000320000_ema.wav.zip
)!
The original mel-spectrum generated by tracotron2 is ：
test_mel.npy.zip

The mel-spectrum converted from tracotron2's mel-spectrum for r9y9's pre-trained wavenet vocoder is:
test_mel_wnet.npy.zip

[NewEricWang]

@rafaelvalle, This is the mel-spectrum produced by tracotron2:
test_mel.npy.zip

Could you help me to convert it into the mel-spectrum for r9y9's pre-trained wavenet vocoder?
I have tried many methods to do it, but I always failed to synthesize normal sound by r9y9's pre-trained wavenet vocoder. This is an example sound produced by r9y9's pre-trained wavenet vocoder:
20180510_mixture_lj_checkpoint_step000320000_ema.wav.zip

Thank you!

[vijaysumaravi]

Hi,

I tried generating audio using the method above but I still get just noise. Please let me know if I am doing something wrong.

My code:

tacotron Inference.py:

mel_outputs, mel_outputs_postnet, _, alignments = model.inference(sequence)
mel = dynamic_range_decompression(mel_outputs_postnet)
np.save('spectrograms/taco_melspec.npy', mel_outputs_postnet.data.cpu().numpy()[0])

np.save('spectrograms/taco_melspec.npy', mel.data.cpu().numpy()[0].T)

conversion of Tacotron2-mel-spec to Wavenet-mel-spec to :

load mel file output by Tacotron 2

mel = torch.autograd.Variable(torch.from_numpy(
np.load('spectrograms/taco_melspec.npy'))[None,:])

followed by your code as is.

...

Project from Spectrogram to r9y9's WaveNet Mel-Spectrogram

mel_minmax = taco_stft_other.spectral_normalize(
torch.matmul(taco_stft_other.mel_basis, spec_from_mel))

np.save('spectrograms/wavenet_melspec.npy', mel_minmax.data.cpu().numpy().transpose())

Then pre-processing from Ryuchi's code:

   mel = np.load('/home/vijay/speech_tts/singletaco/spectrograms/wavenet_melspec.npy')

# methods from from Ryuchi's repo
mel = audio._amp_to_db(mel) - hparams.ref_level_db
if not hparams.allow_clipping_in_normalization:
    assert mel.max() <= 0 and mel.min() - hparams.min_level_db >= 0
mel = audio._normalize(mel)

np.save('spectrograms/wavenetmelspec.npy', mel)

and then I use this wavenetmelspec.npy in the synthesis:

python synthesis.py --preset=models/20180510_mixture_lj_checkpoint_step000320000_ema.json --conditional=spectrograms/wavenetmelspec.npy models/20180510_mixture_lj_checkpoint_step000320000_ema.pth generate/

am I doing something wrong?

[pravn]

I use @r9y9's wavenet as well, which is also how my mels are preprocessed. As @rafaelvalle says, we have to deal with it accordingly. One way of doing that is to use Ryuichi's wavenet as a module (recommended in his wavenet page) and then we can import that module.

import audio #From Ryuchi's module
class MelSpectrogram():
    def __init__(self):

        self.method = hparams.method
        self.n_mels = hparams.num_mels
        self.fmax = hparams.fmax
        self.n_fft = hparams.fft_size
        self.hop_length = hparams.hop_size
        self.sr = hparams.sample_rate
        self.rescaling_max = hparams.rescaling_max
        self.rescaling = hparams.rescaling



 def melspectrogram(self, y, sr):
        if self.method == 'librosa':
            #m: (n_bins, seq_len)
            #y = y / np.abs(y).max() * self.rescaling_max
            m = librosa.feature.melspectrogram(y, sr, n_mels=self.n_mels, n_fft=self.n_fft,
                                               fmax=self.fmax,
                                               hop_length=self.hop_length)


        if self.method =='r9y9':
            #call r9y9's spectrogram routines
            #first normalize
            if self.rescaling:
                y = y / np.abs(y).max() * self.rescaling_max

            #now call audio.melspectrogram
            # (D, N)
            m = audio.melspectrogram(y).astype(np.float32)

            #plot_mel(m)

        return m