Tensorflowtts: Is multi-gpu training possible?

Created on 2 Jul 2020 · 27Comments · Source: TensorSpeech/TensorFlowTTS

I am trying to train tacotron2 with my own dataset which is a little bit larger than LJSpeech and getting OOM errors with batch size 32 and 1 gpu as log below.

Is there any way to use multi-gpu training with the current code? Is reducing the batch size only option? I am not familiar with tf2.2 so can't come up with multi-gpu implementation right away, it is very simple with PyTorch though.

2020-07-02 04:54:41.501441: I tensorflow/core/common_runtime/bfc_allocator.cc:1002] Sum Total of in-use $
hunks: 10.13GiB                                                                                       
2020-07-02 04:54:41.501455: I tensorflow/core/common_runtime/bfc_allocator.cc:1004] total_region_allocat$
d_bytes_: 10874802688 memory_limit_: 10874802688 available bytes: 0 curr_region_allocation_bytes_: 21749$
05376                                                                                                    2020-07-02 04:54:41.501476: I tensorflow/core/common_runtime/bfc_allocator.cc:1010] Stats:               Limit:                 10874802688                                                                       InUse:                 10874794496                                                                       
MaxInUse:              10874802688                   
NumAllocs:                  825693                                                              
MaxAllocSize:            627179520                                                                       

2020-07-02 04:54:41.503330: W tensorflow/core/common_runtime/bfc_allocator.cc:439] *********************$
******************************************************************************  
2020-07-02 04:54:41.503383: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at cast_op$
cc:109 : Resource exhausted: OOM when allocating tensor with shape[32,256] and type float on /job:localh$
st/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc                    
Traceback (most recent call last):
  File "examples/tacotron2/train_tacotron2.py", line 594, in <module>
    main()
  File "examples/tacotron2/train_tacotron2.py", line 586, in main
    resume=args.resume,
  File "examples/tacotron2/train_tacotron2.py", line 392, in fit
    self.run()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_tts/trainers/base_trainer.py", line 72, in run
    self._train_epoch()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_tts/trainers/base_trainer.py", line 94, in _tr$
in_epoch
    self._train_step(batch)
  File "examples/tacotron2/train_tacotron2.py", line 117, in _train_step
    charactor, char_length, mel, mel_length, guided_attention
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __$
all__
    result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 644, in _ca
ll                                                                                                       
    return self._stateless_fn(*args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 2420, in __cal$
__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1665, in _filt$
red_call
    self.captured_inputs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1746, in _call$
flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 598, in call
    ctx=ctx)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_ex$
cute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
  (0) Resource exhausted:  OOM when allocating tensor with shape[32,512] and type float on /job:localhos$
/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[{{node tacotron2/StatefulPartitionedCall/decoder/while/body/_848/decoder_cell/Sum}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upo$
_oom to RunOptions for current allocation info.

         [[tacotron2/StatefulPartitionedCall/decoder/while/LoopCond/_1042/_144]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upo$
_oom to RunOptions for current allocation info.

  (1) Resource exhausted:  OOM when allocating tensor with shape[32,512] and type float on /job:localhos$
/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[{{node tacotron2/StatefulPartitionedCall/decoder/while/body/_848/decoder_cell/Sum}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upo$
_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored. [Op:__inference__one_step_tacotron2_1349493]

Function call stack:
_one_step_tacotron2 -> _one_step_tacotron2

[train]:   0%|                                                                | 0/200000 [04:56<?, ?it/s$

Feature Request 🤗 enhancement 🚀

Source

jun-danieloh

All 27 comments

@jun-danieloh we will support multi-gpu soon. it take some time :)). At this time, you need move long samples to validation folder :)) so the training dataset is not too long then you can be train with large batch_size

dathudeptrai on 2 Jul 2020

@dathudeptrai I can't come up with any way to parse long samples within training dataset. It consists of 40k wav files. Do you have anything in mind? I am planning to reduce the batch size.