Tensorflowtts: How long does "mb-melgan" take to converge?

[LLianJJun]

How long does "mb-melgan" take to converge?

The default config settings and v100 single gpu is used. (batch_size=64, batch_max_size=8192)

It seems that the training takes too long.

[train]: 8%|▊ | 305492/4000000 [57:06:29<1897:39:04, 1.85s/it]

Is "1.8s ~ 2s per iter" normal?

[dathudeptrai]

How long does "mb-melgan" take to converge?

The default config settings and v100 single gpu is used. (batch_size=64, batch_max_size=8192)

It seems that the training takes too long.

[train]: 8%|▊ | 305492/4000000 [57:06:29<1897:39:04, 1.85s/it]

Is "1.8s ~ 2s per iter" normal?

it is not normal, it should run around 3-4it/s. Did you enable mixed precision ?, when discriminator enable you should disble it. Group convolution Mixed precision not yet fully supported and it will cause slow training.

[LLianJJun]

How long does "mb-melgan" take to converge?
The default config settings and v100 single gpu is used. (batch_size=64, batch_max_size=8192)
It seems that the training takes too long.
[train]: 8%|▊ | 305492/4000000 [57:06:29<1897:39:04, 1.85s/it]
Is "1.8s ~ 2s per iter" normal?

it is not normal, it should run around 3-4it/s. Did you enable mixed precision ?, when discriminator enable you should disble it. Group convolution Mixed precision not yet fully supported and it will cause slow training.

After 200000 iter, I disable mixed precision.
I used the default config settings.
Is there a problem with the initial setup? Such cudnn, cuda driver...

and, about 17G of GPU memory is being used.
This also seems to be a problem.

CUDA_VISIBLE_DEVICES=6 python examples/multiband_melgan/train_multiband_melgan.py \ --train-dir ./data/dump/train/ \ --dev-dir ./data//dump//valid/ \ --outdir ./examples/multiband_melgan/exp/train.multiband_melgan.v1/ \ --config ./examples/multiband_melgan/conf/multiband_melgan.v1.yaml \ --use-norm 1 \ --resume "./examples/multiband_melgan/exp/train.multiband_melgan.v1/checkpoints/ckpt-200000"

[dathudeptrai]

How long does "mb-melgan" take to converge?
The default config settings and v100 single gpu is used. (batch_size=64, batch_max_size=8192)
It seems that the training takes too long.
[train]: 8%|▊ | 305492/4000000 [57:06:29<1897:39:04, 1.85s/it]
Is "1.8s ~ 2s per iter" normal?

it is not normal, it should run around 3-4it/s. Did you enable mixed precision ?, when discriminator enable you should disble it. Group convolution Mixed precision not yet fully supported and it will cause slow training.

After 200000 iter, I disable mixed precision.
I used the default config settings.
Is there a problem with the initial setup? Such cudnn, cuda driver...

CUDA_VISIBLE_DEVICES=6 python examples/multiband_melgan/train_multiband_melgan.py \ --train-dir ./data/dump/train/ \ --dev-dir ./data//dump//valid/ \ --outdir ./examples/multiband_melgan/exp/train.multiband_melgan.v1/ \ --config ./examples/multiband_melgan/conf/multiband_melgan.v1.yaml \ --use-norm 1 \ --resume "./examples/multiband_melgan/exp/train.multiband_melgan.v1/checkpoints/ckpt-200000"

did you cache the dataset (cache is True in config) ?. Other reasons maybe come frome ur server :)) when many people use the same disk, ram :)).

[LLianJJun]

yes, "allow_cache: true"
It seems to be my server issue. I'll find the cause.

[ZDisket]

@dathudeptrai

when discriminator enable you should disble it

Any plans to add that to the multi-band melgan example readme? Seems pretty important, it sped up my V100 from 1.3it/s to 5.1it/s

[dathudeptrai]

@ZDisket okay i will add it. (https://github.com/TensorSpeech/TensorFlowTTS/tree/master/examples/melgan#some-important-notes), seem i added it before in Melgan :D