Tts: Use WORLD vocoder

[reuben]

[erogol]

Since I cannot find a handy WORLD interface for feature extraction, it has no progress. If anyone finds something out pls let me know.

Simple WaveNet is really slow to be used especially for the test time. New Parallel WaveNet paper seems interesting but it is also a big work to train proposed teacher-student Framework. Before starting this phase, I expect to finish some other experiments.

[m-toman]

I've used this python wrapper once: https://github.com/JeremyCCHsu/Python-Wrapper-for-World-Vocoder

Alternatively there is this helper script in Merlin that might be useful: https://github.com/CSTR-Edinburgh/merlin/blob/master/misc/scripts/vocoder/world/extract_features_for_merlin.py

[erogol]

@m-toman thnks for pointers. How was the experience with the first one. I've tried it on a single audio file but synthesis on the extracted features was creating aberrations on the generated audio.

Do you also have any results with World? Would you say things are better with it?

[m-toman]

@erogol
I did not try your.codebase yet to be honest. But my experience with world:

Like in the merlin codebase I also replaced F0 extraction with REAPER, which was a huge improvement, at least over the older method in WORLD - Dio.

I've only briefly tried the internal spectrum compression method but did not have any luck with it and resorted back to using MGCs/MFCCs.

It's fast enough for live synthesis on mobile (especially via the streaming implementation)

It generally sounds pretty good, buts I suspect we can get better results with neural vocoders in future (although they typically use the acoustic features for conditioning as well). But the fast Wavenet method seems a bit bloated to me.
I'd love to try WaveRNN at some point (https://arxiv.org/abs/1802.08435) but currently can't find the time.

[erogol]

@m-toman thanks for sharing your bits&pieces. I thinks soon I gonna try WORLD and put the results here.

[erogol]

Wavenet with these values impractical to use https://github.com/r9y9/wavenet_vocoder/issues/28#issuecomment-369190907

[stevemurr]

Hi @erogol , I'd like to help you integrate the WORLD vocoder. It seems we need a way to extract the params needed for the vocoder. One option for extraction and synthesis could be pyworld library.

import pyworld as pw
import soundfile as sf

# numpy_array, sample_rate
x, fs = sf.read(path)
# f0, spectrogram, aperiodicities
f0, sp, ap = pw.wav2world(x, fs)

# numpy_array
y = pw.synthesize(f0, sp, ap, fs, pw.default_frame_period)

sf.write("/path/to/audio.wav", y, fs)

Since the linear spectrogram is already being predicted by the post net, it seems we need to also predict f0 and aperiodicities. Is it possible to pass f0 and aperiodicity targets and predict them using a post net as well?

[erogol]

@stevemurr thanks for the post. I also just start to use pyworld and test its performance in different settings. I can share the notebook of these experiments if you are also interested.

I did extract the features for WORLD and now writing a dataloader. After I finish this, I can push the branch to your provision.

I think what you suggest it possible but I am not sure about the quality. Only way to check is to try :). It would be worthy since WORLD is much faster compared to Griffin-Lim with no quality sacrifice in general. The only down side is to extract the features at first for any dataset since it is too slow to be done on the fly.

[stevemurr]

@erogol Intriguing! I'd love to check out the notebook with your comparisons. I'm very curious about the viability of WORLD as an intermediate choice as a vocoder until fast neural vocoders become ubiquitous.

I've achieved comparable results to Google's paper using https://github.com/Rayhane-mamah/Tacotron-2 and r9y9's WaveNet implementation. The downside being I had to train the wave net to roughly 1.6 million steps which took a couple weeks on a single 1080. The upside being it's useful for offline synthesis.

[erogol]

@stevemurr FYInterest: https://gist.github.com/erogol/92cdeca0e12c9ea3e79e518111b354c7
https://github.com/mozilla/TTS/tree/world_new with only a WORLD feature extraction script. So the next thing is to writing a data loader. I am up to this as soon as I finish my current experiment.

[erogol]

What I observe, encoding with f0 tunning by harvest is the best.

[erogol]

Data loader added...

[stevemurr]

@erogol Thanks for the notebook examples and I agree with your assessment harvest. Taking a look at the data loader, it seems some modifications to train.py will do the trick. Let me know if there are any specific tasks you need help with for the WORLD integration.

[erogol]

@stevemurr I coded the train.py with a small lazy testing. Now I try to update the network architecture to allocate WORLD features. One of the questions, how to replace intermediate mel-spectrogram prediction. Do you think it makes sense to use mel-scale spectral envelope at that stage?

I also try to normalize WORLD features in a non-disruptive way for efficient training. Any ideas on this?

[stevemurr]

@erogol Sorry for the late reply!

I'd like to share a tutorial from one of r9y9's repos - he describes a traditional TTS process using linguistic features and acoustic features to build a duration model and an acoustic model - this can hopefully serve as a guide in targeting WORLD. Since we are using the encoder/decoder architecture we can hopefully learn the alignments from text to WORLD targets. A portion of the tutorial does cover data preparation for WORLD including normalization strategies.

Before training neural networks, we need to normalize data. Following Merlin’s demo script, we will apply min/max normalization for linguistic features and mean/variance normalization to duration/acoustic features. You can compute necessary statistics using nnmnkwii.util.minmax and nnmnkwii.util.meanvar. The comptuation is online, so we can use the functionality for any large dataset.

Following this I assume we should attempt mean/variance normalization for the WORLD features - a function is provided for this in his nnmnkwii library at nnmnkwii.util.meanvar.

how to replace intermediate mel-spectrogram prediction. Do you think it makes sense to use mel-scale spectral envelope at that stage?

Is there a difference between spectrogram and spectral envelope?

I have an idea for using multiple encoder and decoders that I wanted to get your thoughts on - hopefully it's not too crazy sounding :). Currently the network predicts a Mel spectrogram which works in discovering correct alignments with the text. What if we leave the current architecture as is but create a series of encoder/decoders after the Mel is predicted. We then encode the predicted Mel and decode it as the target f0 and do the same for ap. In training we would probably want to pass the ground truth Mel to the f0 and ap encoders. Let me know your thoughts on this or if you have a more optimal solution.

[erogol]

@stevemurr unfortunately I am busy with the other experiments for now, will return to this thread after a while.

[Maxxiey]

Hi,@stevemurr,I am thinking another way to integerate tacotron with WORLD vocoder.What if I just replace the Mel spectogram with WORLD parameters(using pyworld to extract them) and let the model predict these parameters directly? I am not sure if it can work and I am going to give it a try.Dose this idea ever occur to you and have you succeed in integerating tacotron with WORLD vocoder?

Thanks~

[m-toman]

@Maxxiey I suspect this is a reasonable approach, after all that's what systems like Merlin do, or also the a bit older papers by Heiga Zen (although with the Vocaine vocoder) - just put mcep, f0, bap (potentially V/UV flag depending on which F0 extractor you use) in a single vector, usually with delta and delta-delta features (+MLPG afterwards, not sure if necessary with Taco).

Other options:
I've hooked up the WaveRNN repo with another Taco implementation here: https://github.com/m-toman/tacorn
Then of course we have the nvidia wavenet implementation here https://github.com/NVIDIA/nv-wavenet
and interesting slides about wavenet on CPU: http://on-demand.gputechconf.com/gtc/2017/presentation/s7544-andrew-gibiansky-efficient-inference-for-wavenet.pdf

[Maxxiey]

@m-toman Thank you very much for your reply.Your opnion encourages me,a newbie in TTS field, a lot.
The idea to put f0,spectral_envelope and aperiodicity into a single vector is quite straightforward and I think that someone had already working on it: https://github.com/geneing/deepvoice3_pytorch.I am going to try this in the coming days, hope to get somewhere(whether the result is good or not).Thank you again for your help~

[erogol]

@m-toman thanks for visiting TTS. Yes I was planing to do that. Even I wrote a script to extract all WORLD features but could not find time to go further.

DeepVoice3 paper uses WORLD and reports very close results to NN based vocoder. It would be more preferential over WaveNet since it'd be easier to train and perform inference. I believe WORLD would give better results, at least without network in the loop, WORLD performs better recovery then Griffin-Lim algorithm.

This branch is outdated but it might be useful for you to take a look at https://github.com/mozilla/TTS/tree/world_new/scripts

[erogol]

@m-toman How was the quality and run-time with WaveRNN ?

[erogol]

@Maxxiey you can also checkout this https://gist.github.com/erogol/92cdeca0e12c9ea3e79e518111b354c7

[m-toman]

@erogol I only tried the adapted model by fatchord... While I don't have actual numbers, I would say it was probably about 1 minute for a longer sentence instead of 10 minutes with the Wavenet implementation by r9y9. On a GTX 1080Ti.

The GTA samples produced during the training were pretty good (https://www.dropbox.com/sh/2gtunx8d1r92fqb/AADh9CJEtvHnQ7YlwNClk8X5a?dl=0&m=) , I wasn't that happy with the actual end to end synthesis results, but perhaps it was the fault of the Tacotron model - didn't train it very long, perhaps I can produce a couple samples soon.

My main issue is that so much more work is going on with Wavenet. And on the other side of the spectrum - we don't have to train WORLD for every speaker and it also easily compiles and runs fast enough on lots of platforms. That's why I think it would certainly be interesting to try out.
At the moment I also don't have the time but perhaps later on, in case @Maxxiey hasn't already done it then ;).

By chance, do you know what the main differences in your tacotron implementation vs https://github.com/Rayhane-mamah/Tacotron-2 are? (except of course Tensorflow vs PyTorch and the Wavenet integration)

[Maxxiey]

@erogol Thanks, this will come in handy~

[erogol]

@m-toman I've not checked https://github.com/Rayhane-mamah/Tacotron-2 but I don't use Tacotron2 model. The only similarity is to use Location Sensitive Attention the rest is the same old Tacotron. I found no improvement to use other alternative layers proposed in TC2. It worked for them probably since they end the system with WaveNet instead of GL.

[m-toman]

Quick update, I've uploaded two samples here: https://github.com/m-toman/tacorn
They took about 2 minutes on a GTX1080Ti and sound probably OK for the quick test I did (no tuning at all, just fire & forget).

But I now think that WORLD might be a better option to go for, for fast synthesis and also avoiding training/hyperparameter-tuning of two different models.

[Maxxiey]

@m-toman Thanks for sharing those samples and I think they sound good. Witch vocoder did you use to get them, WORLD or WaveNet?

[m-toman]

@m-toman Thanks for sharing those samples and I think they sound good. Witch vocoder did you use to get them, WORLD or WaveNet?

This uses the modified WaveRNN here: https://github.com/fatchord/WaveRNN

[m-toman]

Perhaps you already know, but there's another option for neural vocoding in town now:
https://github.com/NVIDIA/waveglow

[erogol]

@m-toman thx for sharing. I already had my own implementation before the release but this model has 200M parameters, such a beast. I am going to try but it is not a very practical solution.

TTS has only 7M parameters :)

[m-toman]

Didn't look at it in detail but I've seen in the issues that the GTX1080Ti was only able to train a batch size of 1, so I already assumed that it is quite hefty. Oh my... at first I discarded it because the abstract stated "synthesis with 500 Hz" which is not really impressive. But seems to be a typo as in the paper it says 500kHz on some (big, fat - I assume) GPU.

I wonder why in the samples they used r9y9s WaveNet and not their own implementation.

[m-toman]

Another interesting paper: https://arxiv.org/pdf/1811.06292.pdf
tl;dr they used more or less vanilla WaveRNN but trained it on a about 70 speakers to make it universal.

I'll be back on synthesis soon and would look into the original WaveRNN implementation again. Which parts from Tacotron2 are missing so we could hook up a neural vocoder?
Or did you make any progress on that?

[erogol]

@m-toman thanks for the link.It is interesting.

So far, I've also obtained the easiest working vocoder with WaveRNN. It is much easier, faster to train and much smaller. Here is the fork I work on https://github.com/erogol/WaveRNN . With this, I was able to generate great examples with vanilla mel-specs. However, I couldn't make it work with TTS mel-specs. I guess it requires higher quality mel-specs.

Given this, right now I train TTS with r=1. It takes a lot longer to train but seem promising. Loss values are already much lower than r=5 (which is default)

I also train WaveGlow with TTS mel-specs but again results are not good so far.

If you like to go for Tacotron2, there is a branch. It's supposed to work just fine but I couldn't align the attention. In the paper it is not clear to me how they handle the attention part. If they keep and RNN in attention part as in Tacotron1 or they do something else. It is also 10x larger than Tacotron, making things slower. (TTS master 7M vs Tacotron2 120M)

One last try was that I trained TTS + WaveRNN (~15M params, r=5) end-to-end. It gave the best results so far but still not acceptable. I plan to try with r=1.

Let me know if you need help at any part.

[m-toman]

Here is some info on a successful integration of WORLD https://github.com/Rayhane-mamah/Tacotron-2/issues/304
Sounds good but rather typical vocodery...

I wasn't completely happy with the alternative wavernn model with the 9 bit quantization because it was a bit noisy.

I tried this cleaned up and extended version with the 10 bit version.
https://github.com/m-toman/WaveRNN-Pytorch
with mixed results...
Oh, all of them trained from GTA Mel specs of Rayhane-mamahs taco2 implementation.

Yeah waveglow is huge, although in their issues there is a discussion about reducing parameters. Interesting as it seems to work with their rather small taco2 implementation without lots of tricks..?

So to sum up I still haven't reached a real conclusion on this topic. Although I assume that something like the universal neural vocoder paper posted above is probably more future-proof than WORLD.

[erogol]

@m-toman thx for sharing your progress. I am still dealing with #50 to train Tacotron in r=1 setting and training current TTS with different languages soon to be delivered here.

How small their taco2 implementation compared to the reference. Last time I checked the implementation was almost the same.

[m-toman]

Oh, I was comparing it to the implementation by Rayhane-mamah which is a pretty large codebase with lots of settings, tricks and experiments to get it to work.
With it I got LJ and a German test voice trained without issues, even with batch size 16 and r=1, from there finetuning another dataset generally works pretty well.

Although I suspect much of the size is just because of the tensorflow boilerplate and the Wavenet integration.

Still it seems that the Nvidia version is just a straightforward 1:1 implementation and apparently works fine with neural vocoders. But perhaps this perception is wrong as I haven't really experimented with it.

So you think #50 is because you did not fully adopt the taco 2 architecture?

[erogol]

tacotron2 has the same problem. That's why in even NVIDIA implementation they keep the dropout in the training mode always.

[erogol]

@m-toman do you remember how large the batch size could you use the most with TF taco2 implementation assuming you have a single 1080ti?

[m-toman]

That's interesting, I just checked both other implementations and yes, they fix dropout to train mode.
Although Rayhane-mamah states here
https://github.com/Rayhane-mamah/Tacotron-2/blob/master/tacotron/models/modules.py#L247-L248
`

The paper discussed introducing diversity in generation at inference time by using a dropout of 0.5 only in prenet layers (in both training and inference).

`
Perhaps he just never tried it otherwise and never came across the alignment problem.

But this is not related to the issues for using a neural vocoder with this repo, right? (so if you just kept dropout on in inference there would still be other issues to be solved?)

Regarding batch size, for Rayhane-mamah I had to throw out all sentences with a length of more than 10 seconds (800 mel frames), then I was able to use batch size 32 and it just fit into the 11GB of the GTX1080Ti.
I only briefly trained the nvidia implementation for a couple epochs on the LJ set, but it worked without throwing out sentences with batch size 32 (again it was pretty close with the 11GB). Didn't check how long the longest wav in the LJ dataset is.

I suspect the repo of Rayhane-mamah requires more memory because he uses an additional layer to predict linear spectrograms from the mel specs for better Griffin-Lim results (as described here https://github.com/Rayhane-mamah/Tacotron-2/wiki/Spectrogram-Feature-prediction-network#post-processor-net-optional)

[carlfm01]

Hi guys, just sharing this https://jmvalin.ca/papers/lpcnet_icassp2019.pdf, I'd love to read your opinions.
Here's the code https://github.com/mozilla/LPCNet
I know @azraelkuan and @candlewill used tacotron with lpcnet, maybe they could share its testing notes.

https://github.com/mozilla/LPCNet/issues/4#issuecomment-455222377

[tsungruihon]

Does any friend have ever tried merging mozilla TTS with world vocoder successfully?

[tsungruihon]

@m-toman @erogol It seems that facebookresearch/loop uses WORLD as a vocoder(https://github.com/facebookresearch/loop).

[m-toman]

I didn't really like the non commercial license ;).
Meanwhile there is this implementation: https://github.com/Rayhane-mamah/Tacotron-2/issues/304

I found the samples to sound very vocoder-typical and probably even worse than the Nvidia griffin lim samples here:
https://nv-adlr.github.io/WaveGlow
Although I couldn't reproduce them by downloading their pretrained model and using their griffin lim notebook - sounded significantly worse.

LPCNet looks promising.

Lastly I'm currently testing the architecture from the Amazon paper stated above but at the moment the training got stuck. Also trying h-merus implementatiin he forked from my repo and put lots of work in.
Well... Unfortunately training takes really long and I currently don't have much time to work on it.

[tsungruihon]

@m-toman thanks my friend. I am now looking for a way to connect mozilla TTS and World vocoders.

[carlfm01]

Looks like people is investing time trying to make easy to work with tacotron and LPCNet
https://github.com/MlWoo/LPCNet

Listen to the examples of the comment:

https://github.com/mozilla/LPCNet/issues/33#issue-414513757

[tsungruihon]

Hey guys, I struggled to made WORLD as a vocoder, following @erogol and @begeekmyfriend and it seems that i have made it work, but the alignment seems not well.
Thanks @erogol and @begeekmyfriend.
@begeekmyfriend have wrote a tensorflow version, maybe you could check the link here.
My modification WORLD code is in here named tts_world. Hope this could bring some help to the one who want to use WORLD as vocoder like me.
Cause i just spend 2 days in the modification, the code just contain the necessary part of training and validation. I haven't made everything seem perfect, like i haven't wrote the synthesizer, hope someone could help together.
Now the alignment and loss graph are like below. The result seems not good. Maybe i need to play with the parameter file.

[tsungruihon]

i bridged world with tts. now the alignment images are below. It seems that the model began to align, but still too slow.

train alignment

eval alignment

[erogol]

@tsungruihon it looks erroneous to me. Are you sure your input and output set correctly at any part of your network?

[begeekmyfriend]

It really strange that whether it will learn alignment with WORLD acoustic features that might depend on dataset. For some dataset it never converge but for others it learns quickly. I doubt the acoustic features of WORLD vocoder very sensitive for training.

[tsungruihon]

@erogol @begeekmyfriend here is the latest alignment graph. Yesterday i have tried to generate audio using pyworld , but the wav array's numbers are all zeros. Today i will check it out.

[erogol]

@tsungruihon it looks better. How many epochs?

[tsungruihon]

@erogol @begeekmyfriend nearly 140k steps.
But now the strangest thing is that i could not synthesize audio. I compared target and predict output given the same input like text = 'la4 wu3 you2 tu4'. The loss between target and prediction is small, but when i denormalize lf0 to f0, the values of f0_pred are quite large.

world_targets = torch.cat([lf0s_input.unsqueeze_(-1), mgcs_input, baps_input], dim=-1)
world_out, alignments, stop_tokens = model.forward(chars_var.long(), world_targets)
criterion = L1LossMasked()
print(criterion(world_out, world_targets, mgcs_length))
[output]
tensor(0.4070, grad_fn=<DivBackward0>)

Below is how i extract lf0 output, mgc_outputs and bap_outputs like @begeekmyfriend did. Here the n_mgc is 60.

lf0_outputs = world_out[:, :, 0]
mgc_outputs = world_out[:, :, 1 : 1 + hp.n_mgc]
bap_outputs = world_out[:, :, 1 + hp.n_mgc:]

lf0_targets = world_targets[:, :, 0]
mgc_targets = world_targets[:, :, 1 : 1 + hp.n_mgc]
bap_targets = world_targets[:, :, 1 + hp.n_mgc:]

print(sum(sum(lf0_outputs - lf0_targets)))
[output]
tensor(-491.7143, grad_fn=<AddBackward0>)

[tsungruihon]

@erogol @begeekmyfriend i just plot the lf0_predict, lf0_target, mgc_predict, mgc_target , bap_predict and bap_target, trying to figure out what's going on. The result tells me something wrong.

[begeekmyfriend]

@tsungruihon Maybe you need some post processing before synthesis. See my code. By the way, you can do some resynth with WORLD vocoder only and record the feature values for checking out. https://github.com/Rayhane-mamah/Tacotron-2/files/2713952/world_vocoder_resynth_scripts.zip Here is resynth script.

[tsungruihon]

@begeekmyfriend sure my friend. I followed your code like below

lf0_outputs = world_out[:, :, 0]
mgc_outputs = world_out[:, :, 1:61]
bap_outputs = world_out[:, :, 61:]

lf0 = lf0_outputs[0].data.numpy()
lf0 = np.where(lf0 < 1, 0.0, lf0)
f0_pred = f0_denormalize(lf0)

mgc_targets = mgc_outputs[0].data.numpy()
sp_pred = sp_denormalize(mgc_targets)

bap_targets = bap_outputs[0].data.numpy()
ap_pred = ap_denormalize(bap_targets, lf0)

But when i ran wav = pw.synthesize(f0_pred, sp_pred, ap_pred, 48000), the jupyter notebook kernel restart automatically every time..

[begeekmyfriend]

@tsungruihon There is core dump in pw.synthesize method. Please save those feature values and use resynth script for test.

[tsungruihon]

@erogol @begeekmyfriend i would like to share the prediction of the model, lf0, mgc and bap and the denormalize result f0, sp and ap. It still show Segmentation fault (core dumped).

world_features_prediction.zip

[begeekmyfriend]

mandarin_male_world.zip
The training is only with 80K steps and still undergoing. In my humble opinion, We need to some normalization jobs for WORLD vocoder acoustic feature values for quicker alignment. If it comes to core dump in WORLD library, just go on training @tsungruihon.
BTW, I am using Tacotron-2 with this branch.

[tsungruihon]

Thanks my friend. Really grateful and appreciated.! @begeekmyfriend

[begeekmyfriend]

WORLD feature extraction from GanTTS helps convergence. Any feedback is welcome! https://github.com/begeekmyfriend/Tacotron-2/commit/e40a7b73ac31d299d731439fbabe8921b231a739

[begeekmyfriend]

Here is biaobei mandarin demo from T2 + WORLD. The f0 feature value prediction is tough for this model.
xmly_biaobei_world.zip

[mrgloom]

@begeekmyfriend
I wonder why did you not used default setting like in pyworld demo? There is something specific for for TTS task?
https://github.com/JeremyCCHsu/Python-Wrapper-for-World-Vocoder/blob/master/demo/demo.py

[begeekmyfriend]

The part of feature extraction is derived from gantts project.

[begeekmyfriend]

Never recommend this vocoder anymore. The feature values are too sensitive to be predicted. In my humble opinion mel spectrogram plus neural network vocoder such is WaveRNN is the most proper solution for TTS so far. I give up.

By the way the implementation of WORLD + Tacotron2 is still kept in my fork branch

[carlfm01]

@begeekmyfriend did you ever tried using LPCNET+Tacotron2?

[carlfm01]

@tsungruihon I saw you comment on how to connect LPCNET and Tacotron, this guide maybe will be useful for you https://github.com/MlWoo/LPCNet/commit/324b2128e13debbb6439b53d76e110cbda2b3db6

[tsungruihon]

@carlfm01 thanks a lot.!

[begeekmyfriend]

I have tried WaveRNN.
wavernn_mandarin_male_22050.zip

[mrgloom]

Test on robustness of smoothing features from WORLD vocoder(as I understand L1 loss will introduce smoothness of predicted time series):

Based on: https://github.com/JeremyCCHsu/Python-Wrapper-for-World-Vocoder/blob/master/demo/demo.py

As source this file used: https://google.github.io/tacotron/publications/tacotron2/demos/romance_gt.wav
From tacotron2 demo page: https://google.github.io/tacotron/publications/tacotron2/

Results: reconstruct_smooth_features_example.zip

Here I just use box filter:

import os
from shutil import rmtree

import numpy as np
import soundfile as sf
import pyworld as pw
from scipy import signal

import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt

def savefig(filename, figlist, log=True):
    EPSILON = 1e-8
    n = len(figlist)
    f = figlist[0]
    if len(f.shape) == 1:
        plt.figure()
        for i, f in enumerate(figlist):
            plt.subplot(n, 1, i+1)
            if len(f.shape) == 1:
                plt.plot(f)
                plt.xlim([0, len(f)])
    elif len(f.shape) == 2:
        plt.figure()
        for i, f in enumerate(figlist):
            plt.subplot(n, 1, i+1)
            if log:
                x = np.log(f + EPSILON)
            else:
                x = f + EPSILON
            plt.imshow(x.T, origin='lower', interpolation='none', aspect='auto', extent=(0, x.shape[0], 0, x.shape[1]))
    else:
        raise ValueError('Input dimension must < 3.')
    plt.savefig(filename)


def reconstruct_smooth_features():
    output_dir = 'temp'
    if os.path.isdir(output_dir):
        rmtree(output_dir)
    os.mkdir(output_dir)

    x, fs = sf.read('utterance/romance_gt.wav')
    print('x.shape', x.shape) #
    print('fs', fs) #

    f0, sp, ap = pw.wav2world(x, fs)
    print('f0.shape', f0.shape) #
    print('sp.shape', sp.shape) #
    print('ap.shape', ap.shape) #

    def smooth_1d(y, n=3):
        box = np.ones(n) / n
        y_smooth = np.convolve(y, box, mode='same')
        return y_smooth

    def smooth_2d(y, n=3):
        box = np.ones((n,n)) / (n*n)
        y_smooth = signal.convolve(y, box, mode='same')
        return y_smooth

    y = pw.synthesize(f0, sp, ap, fs, pw.default_frame_period)
    sf.write(os.path.join(output_dir, 'original_reconstruction.wav'), y, fs)

    for kernel_size in [3,5,7,9,15,31,65]:
        print('-'*60)
        f0_smooth = smooth_1d(np.copy(f0), n=kernel_size)
        sp_smooth = smooth_2d(np.copy(sp), n=kernel_size)
        ap_smooth = smooth_2d(np.copy(ap), n=kernel_size)

        savefig(os.path.join(output_dir, str(kernel_size).zfill(4)+'_f0.png'), [f0, f0_smooth])
        savefig(os.path.join(output_dir, str(kernel_size).zfill(4)+'_sp.png'), [sp, sp_smooth])
        savefig(os.path.join(output_dir, str(kernel_size).zfill(4)+'_ap.png'), [ap, ap_smooth])

        y_smooth = pw.synthesize(f0_smooth, sp_smooth, ap_smooth, fs, pw.default_frame_period)
        sf.write(os.path.join(output_dir, str(kernel_size).zfill(4)+'_smoothed_reconstruction.wav'), y_smooth, fs)

        print('kernel_size:', kernel_size)
        print('np.max(np.abs(f0-f0_smooth))', round(np.max(np.abs(f0-f0_smooth)), 2))
        print('np.max(np.abs(sp-sp_smooth))', round(np.max(np.abs(sp-sp_smooth)), 2))
        print('np.max(np.abs(ap-ap_smooth))', round(np.max(np.abs(ap-ap_smooth)), 2))
        print('np.max(np.abs(y-y_smooth))', round(np.max(np.abs(y-y_smooth)), 2))


if __name__ == '__main__':
    reconstruct_smooth_features()

[mrgloom]

Abouth the same test for griffin-lim algorithm, even at kernel_size 5 it already produce bad results.

Results: griffin_lim_smooth_mel_reconstruction.zip

import os
import shutil

from scipy import signal
import numpy as np

from utils.audio import AudioProcessor

if __name__ == "__main__":

    ap = AudioProcessor(
        num_mels = 80,
        num_freq =  1025,
        sample_rate = 22050,
        frame_length_ms = 50,
        frame_shift_ms = 12.5,
        preemphasis = 0.98,
        min_level_db = -100,
        ref_level_db = 20,
        power = 1.5,
        griffin_lim_iters = 60,
        signal_norm = True,
        symmetric_norm = False,
        max_norm = 1,
        clip_norm = True,
        mel_fmin = 0.0,
        mel_fmax = 8000.0,
        do_trim_silence = False
    )

    def smooth_2d(y, n):
        box = np.ones((n,n)) / (n*n)
        y_smooth = signal.convolve(y, box, mode='same')
        return y_smooth

    for kernel_size in [3,5,7,9,15,31,65]:
        for griffin_lim_iters in [30,60,90]:
            print('-' * 60)
            print('kernel_size', kernel_size)
            print('griffin_lim_iters', griffin_lim_iters)

            ap.griffin_lim_iters = griffin_lim_iters

            output_dir = 'griffin_lim_iters_'+str(griffin_lim_iters)+'_kernel_size_'+str(kernel_size)
            os.makedirs(output_dir)

            wav_filepath = './example_data/LJ001-0001.wav'

            wav = ap.load_wav(wav_filepath)
            mel = ap.melspectrogram(wav)
            shutil.copy(wav_filepath, os.path.join(output_dir, 'original.wav'))

            wav_reconstructed = ap.inv_mel_spectrogram(mel)
            ap.save_wav(wav_reconstructed, os.path.join(output_dir, 'original_reconstructed.wav'))

            mel_smooth = smooth_2d(mel, kernel_size)
            wav_smooth_reconstructed = ap.inv_mel_spectrogram(mel_smooth)
            ap.save_wav(wav_smooth_reconstructed, os.path.join(output_dir, 'smooth_reconstructed.wav'))