Tts: ValueError: Unsupported dtype when calling self.dict_to_tb_figure

[WeberJulian]

I use the dev branch.
At the 5k step mark (which correspond to my save_step), I get this error:

--> STEP: 71/703 -- GLOBAL_STEP: 5000
     | > decoder_loss: 0.50926  (0.39530)
     | > postnet_loss: 1.28198  (1.03700)
     | > stopnet_loss: 0.21789  (0.21245)
     | > decoder_coarse_loss: 0.61804  (0.53054)
     | > decoder_ddc_loss: 0.00180  (0.00346)
     | > decoder_diff_spec_loss: 0.30351  (0.25957)
     | > postnet_diff_spec_loss: 1.08807  (0.90843)
     | > decoder_ssim_loss: 0.47350  (0.34447)
     | > postnet_ssim_loss: 0.53051  (0.40000)
     | > loss: 1.48395  (1.20202)
     | > align_error: 0.98525  (0.96950)
     | > avg_spec_length: 136.6
     | > avg_text_length: 39.6
     | > step_time: 0.6146
     | > loader_time: 0.00
     | > current_lr: 0.0001
 > CHECKPOINT : /home/ubuntu/model_vm/fr-5-speakers-November-22-2020_09+55PM-8d3554d/checkpoint_5000.pth.tar
 ! Run is kept in /home/ubuntu/model_vm/fr-5-speakers-November-22-2020_09+55PM-8d3554d
Traceback (most recent call last):
  File "train_tacotron.py", line 691, in <module>
    main(args)
  File "train_tacotron.py", line 603, in main
    global_step, epoch, scaler, scaler_st, speaker_mapping)
  File "train_tacotron.py", line 305, in train
    tb_logger.tb_train_figures(global_step, figures)
  File "/home/ubuntu/TTS/TTS/utils/tensorboard_logger.py", line 60, in tb_train_figures
    self.dict_to_tb_figure(f"{self.model_name}_TrainFigures", figures, step)
  File "/home/ubuntu/TTS/TTS/utils/tensorboard_logger.py", line 44, in dict_to_tb_figure
    self.writer.add_figure('{}/{}'.format(scope_name, key), value, step)
  File "/home/ubuntu/anaconda3/envs/TTS/lib/python3.6/site-packages/tensorboardX/writer.py", line 713, in add_figure
    self.add_image(tag, figure_to_image(figure, close), global_step, walltime, dataformats='CHW')
  File "/home/ubuntu/anaconda3/envs/TTS/lib/python3.6/site-packages/tensorboardX/utils.py", line 36, in figure_to_image
    image = render_to_rgb(figures)
  File "/home/ubuntu/anaconda3/envs/TTS/lib/python3.6/site-packages/tensorboardX/utils.py", line 23, in render_to_rgb
    canvas.draw()
  File "/home/ubuntu/anaconda3/envs/TTS/lib/python3.6/site-packages/matplotlib/backends/backend_agg.py", line 407, in draw
    self.figure.draw(self.renderer)
  File "/home/ubuntu/anaconda3/envs/TTS/lib/python3.6/site-packages/matplotlib/artist.py", line 41, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/TTS/lib/python3.6/site-packages/matplotlib/figure.py", line 1864, in draw
    renderer, self, artists, self.suppressComposite)
  File "/home/ubuntu/anaconda3/envs/TTS/lib/python3.6/site-packages/matplotlib/image.py", line 131, in _draw_list_compositing_images
    a.draw(renderer)
  File "/home/ubuntu/anaconda3/envs/TTS/lib/python3.6/site-packages/matplotlib/artist.py", line 41, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/TTS/lib/python3.6/site-packages/matplotlib/cbook/deprecation.py", line 411, in wrapper
    return func(*inner_args, **inner_kwargs)
  File "/home/ubuntu/anaconda3/envs/TTS/lib/python3.6/site-packages/matplotlib/axes/_base.py", line 2747, in draw
    mimage._draw_list_compositing_images(renderer, self, artists)
  File "/home/ubuntu/anaconda3/envs/TTS/lib/python3.6/site-packages/matplotlib/image.py", line 131, in _draw_list_compositing_images
    a.draw(renderer)
  File "/home/ubuntu/anaconda3/envs/TTS/lib/python3.6/site-packages/matplotlib/artist.py", line 41, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/TTS/lib/python3.6/site-packages/matplotlib/image.py", line 644, in draw
    renderer, renderer.get_image_magnification())
  File "/home/ubuntu/anaconda3/envs/TTS/lib/python3.6/site-packages/matplotlib/image.py", line 929, in make_image
    magnification, unsampled=unsampled)
  File "/home/ubuntu/anaconda3/envs/TTS/lib/python3.6/site-packages/matplotlib/image.py", line 495, in _make_image
    A_resampled = _resample(self, A_scaled, out_shape, t)
  File "/home/ubuntu/anaconda3/envs/TTS/lib/python3.6/site-packages/matplotlib/image.py", line 197, in _resample
    image_obj.get_filterrad())
ValueError: Unsupported dtype

here is my config (don't mind the run name and description, it's single speaker and m-ailabs)

[SanjaESC]

Could be a problem with using mixed precision, see also here https://github.com/matplotlib/matplotlib/issues/15432

[WeberJulian]

I think you're right, so the solution would be to cast here

  File "train_tacotron.py", line 305, in train
    tb_logger.tb_train_figures(global_step, figures)

the figures with something like:

figures = [figure.astype(np.float32) for figure in figures]
tb_logger.tb_train_figures(global_step, figures)

[WeberJulian]

Myfix consisting of adding a bunch of .astype(np.float32) works, but I don't know if it's clean enough to go in a PR. Do we let the fix in the train_tacotron.py file or do we implement something else inside the tb-logger ?

[WeberJulian]

With the same config, I get this error at 10.4k

[WARNING] NaN or Inf found in input tensor.
[WARNING] NaN or Inf found in input tensor.
 ! Run is kept in /home/ubuntu/model_vm/fr-5-speakers-November-23-2020_01+41PM-8d3554d
Traceback (most recent call last):
  File "train_tacotron.py", line 691, in <module>
    main(args)
  File "train_tacotron.py", line 603, in main
    global_step, epoch, scaler, scaler_st, speaker_mapping)
  File "train_tacotron.py", line 183, in train
    text_lengths)
  File "/home/ubuntu/anaconda3/envs/TTS/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/TTS/TTS/tts/layers/losses.py", line 380, in forward
    raise RuntimeError(f" [!] NaN loss with {key}.")
RuntimeError:  [!] NaN loss with decoder_loss.

Can this come from AMP as well ?

I find the tensorboard suspect as it seems that the gradient hit 0 cause all the losses stagnated.

EDIT: I forgot about the gradnorm graph and it seem that I was right about the gradients. @erogol could this be link to the switch from nvidia/apex to native pytorch amp ?

[erogol]

that's a great call @SanjaESC . I was thinking about this problem but mixed precision makes a ton of sense.

[SanjaESC]

Myfix consisting of adding a bunch of .astype(np.float32) works, but I don't know if it's clean enough to go in a PR. Do we let the fix in the train_tacotron.py file or do we implement something else inside the tb-logger ?

This would work, but I think your mentioned approach in the tb-logger should be better. Only cast to float32 if needed, so something like this:

def dict_to_tb_figure(self, scope_name, figures, step):
    for key, value in figures.items():
        value = value if not value.dtype == np.float16 else value.astype(np.float32)
        self.writer.add_figure('{}/{}'.format(scope_name, key), value, step)

[WeberJulian]

Thanks @SanjaESC, you're right it looks much better.

But there still is a training problem even with this fix. The training doesn't work with AMP: true.
These are the graphs with the exact same config (except AMP)

[erogol]

Myfix consisting of adding a bunch of .astype(np.float32) works, but I don't know if it's clean enough to go in a PR. Do we let the fix in the train_tacotron.py file or do we implement something else inside the tb-logger ?

This would work, but I think your mentioned approach in the tb-logger should be better. Only cast to float32 if needed, so something like this:

def dict_to_tb_figure(self, scope_name, figures, step):
    for key, value in figures.items():
        value = value if not value.dtype == np.float16 else value.astype(np.float32)
        self.writer.add_figure('{}/{}'.format(scope_name, key), value, step)

This won't work since value here is a figure. I fix it in plot_spectrogram function.

[WeberJulian]

Myfix consisting of adding a bunch of .astype(np.float32) works, but I don't know if it's clean enough to go in a PR. Do we let the fix in the train_tacotron.py file or do we implement something else inside the tb-logger ?

This would work, but I think your mentioned approach in the tb-logger should be better. Only cast to float32 if needed, so something like this:

def dict_to_tb_figure(self, scope_name, figures, step):
    for key, value in figures.items():
        value = value if not value.dtype == np.float16 else value.astype(np.float32)
        self.writer.add_figure('{}/{}'.format(scope_name, key), value, step)

This won't work since value here is a figure. I fix it in plot_spectrogram function.

Oh yeah that's right, I forgot why I didn't fix it here in the first place.

And for the training stability problem ? Should I open an other issue about it as it's not directly related ?

[erogol]

I am updating dev branch with the fix

[erogol]

now it should be fixed