Tacotron-2: `python train.py --model=Tacotron-2` stucks

[a8568730]

Hi, thank you for this great project!

We prepared the LJ-dataset, and follow the preprocess and training instructions for practice.
This is our dockerfile:

FROM tensorflow/tensorflow:latest-gpu-py3

RUN apt-get update
RUN apt-get install -y libasound-dev portaudio19-dev libportaudio2 libportaudiocpp0 ffmpeg libav-tools wget git vim
RUN pip install --upgrade pip

RUN wget http://data.keithito.com/data/speech/LJSpeech-1.1.tar.bz2
RUN tar -jxvf LJSpeech-1.1.tar.bz2

RUN git clone https://github.com/Rayhane-mamah/Tacotron-2.git
RUN mv LJSpeech-1.1/ Tacotron-2/

WORKDIR Tacotron-2
RUN pip install -r requirements.txt

Today is the 3rd training day, and the training process seems to stuck.
It would be very appreciated if there are some advice to help us find out problems.

No error messages, and the latest update time of output dir logs-Tacotron-2/ is yesterday.

• OS: Ubuntu 18.04.1
• Python: 3.5.2
• Tensorflow: 1.9.0
• GPU status:

$ watch nvidia-smi

Every 2.0s: nvidia-smi                 

Fri Aug 10 15:59:53 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.77                 Driver Version: 390.77                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1070    Off  | 00000000:01:00.0  On |                  N/A |
|  0%   36C    P8    17W / 230W |   7874MiB /  8116MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1176      G   /usr/lib/xorg/Xorg                            65MiB |
|    0      1850      G   /usr/bin/gnome-shell                          91MiB |
|    0      5239      C   python                                      7685MiB |
|    0     31673      G   /opt/teamviewer/tv_bin/TeamViewer             27MiB |
+-----------------------------------------------------------------------------+

[a8568730]

We press ctrl+c to stop the process, and it prints the error message below:

Writing summary at step 8500
Step    8503 [2.763 sec/step, loss=0.34506, avg_loss=0.35066]
Generated 32 train batches of size 32 in 11.061 sec
Step    8536 [2.766 sec/step, loss=0.34923, avg_loss=0.35019]
Generated 32 train batches of size 32 in 11.045 sec
Step    8567 [2.737 sec/step, loss=0.34346, avg_loss=0.34899]
Generated 32 train batches of size 32 in 10.737 sec
Traceback (most recent call last):ss=0.35169, avg_loss=0.34880]
  File "train.py", line 133, in <module>
^C  File "train.py", line 127, in main
    train(args, log_dir, hparams)
  File "train.py", line 52, in train
    checkpoint = tacotron_train(args, log_dir, hparams)
  File "/notebooks/Tacotron-2/tacotron/train.py", line 308, in tacotron_train
    return train(log_dir, args, hparams)
  File "/notebooks/Tacotron-2/tacotron/train.py", line 180, in train
    log(message, end='\r', slack=(step % args.checkpoint_interval == 0))
  File "/notebooks/Tacotron-2/infolog.py", line 26, in log
    print(msg, end=end)
KeyboardInterrupt
Exception in thread background:
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1322, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled
         [[Node: datafeeder/input_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](datafeeder/input_queue, _arg_datafeeder/inputs_0_1, _arg_datafeeder/input_lengths_0_0, _arg_datafeeder/mel_targets_0_3, _arg_datafeeder/token_targets_0_5, _arg_datafeeder/linear_targets_0_2, _arg_datafeeder/targets_lengths_0_4)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/notebooks/Tacotron-2/tacotron/feeder.py", line 166, in _enqueue_next_train_group
    self._session.run(self._enqueue_op, feed_dict=feed_dict)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 900, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1135, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1316, in _do_run
    run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1335, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled
         [[Node: datafeeder/input_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](datafeeder/input_queue, _arg_datafeeder/inputs_0_1, _arg_datafeeder/input_lengths_0_0, _arg_datafeeder/mel_targets_0_3, _arg_datafeeder/token_targets_0_5, _arg_datafeeder/linear_targets_0_2, _arg_datafeeder/targets_lengths_0_4)]]

Caused by op 'datafeeder/input_queue_enqueue', defined at:
  File "train.py", line 133, in <module>
    main()
  File "train.py", line 127, in main
    train(args, log_dir, hparams)
  File "train.py", line 52, in train
    checkpoint = tacotron_train(args, log_dir, hparams)
  File "/notebooks/Tacotron-2/tacotron/train.py", line 308, in tacotron_train
    return train(log_dir, args, hparams)
  File "/notebooks/Tacotron-2/tacotron/train.py", line 127, in train
    feeder = Feeder(coord, input_path, hparams)
  File "/notebooks/Tacotron-2/tacotron/feeder.py", line 85, in __init__
    self._enqueue_op = queue.enqueue(self._placeholders)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/data_flow_ops.py", line 346, in enqueue
    self._queue_ref, vals, name=scope)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 3977, in queue_enqueue_v2
    timeout_ms=timeout_ms, name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3414, in create_op
    op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1740, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

CancelledError (see above for traceback): Enqueue operation was cancelled
         [[Node: datafeeder/input_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](datafeeder/input_queue, _arg_datafeeder/inputs_0_1, _arg_datafeeder/input_lengths_0_0, _arg_datafeeder/mel_targets_0_3, _arg_datafeeder/token_targets_0_5, _arg_datafeeder/linear_targets_0_2, _arg_datafeeder/targets_lengths_0_4)]]


Exception in thread background:
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1322, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled
         [[Node: datafeeder/eval_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](datafeeder/eval_queue, _arg_datafeeder/inputs_0_1, _arg_datafeeder/input_lengths_0_0, _arg_datafeeder/mel_targets_0_3, _arg_datafeeder/token_targets_0_5, _arg_datafeeder/linear_targets_0_2, _arg_datafeeder/targets_lengths_0_4)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/notebooks/Tacotron-2/tacotron/feeder.py", line 174, in _enqueue_next_test_group
    self._session.run(self._eval_enqueue_op, feed_dict=feed_dict)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 900, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1135, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1316, in _do_run
    run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1335, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.CancelledError: Enqueue operation was cancelled
         [[Node: datafeeder/eval_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](datafeeder/eval_queue, _arg_datafeeder/inputs_0_1, _arg_datafeeder/input_lengths_0_0, _arg_datafeeder/mel_targets_0_3, _arg_datafeeder/token_targets_0_5, _arg_datafeeder/linear_targets_0_2, _arg_datafeeder/targets_lengths_0_4)]]

Caused by op 'datafeeder/eval_queue_enqueue', defined at:
  File "train.py", line 133, in <module>
    main()
  File "train.py", line 127, in main
    train(args, log_dir, hparams)
  File "train.py", line 52, in train
    checkpoint = tacotron_train(args, log_dir, hparams)
  File "/notebooks/Tacotron-2/tacotron/train.py", line 308, in tacotron_train
    return train(log_dir, args, hparams)
  File "/notebooks/Tacotron-2/tacotron/train.py", line 127, in train
    feeder = Feeder(coord, input_path, hparams)
  File "/notebooks/Tacotron-2/tacotron/feeder.py", line 97, in __init__
    self._eval_enqueue_op = eval_queue.enqueue(self._placeholders)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/data_flow_ops.py", line 346, in enqueue
    self._queue_ref, vals, name=scope)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 3977, in queue_enqueue_v2
    timeout_ms=timeout_ms, name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3414, in create_op
    op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1740, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

CancelledError (see above for traceback): Enqueue operation was cancelled
         [[Node: datafeeder/eval_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_INT32, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](datafeeder/eval_queue, _arg_datafeeder/inputs_0_1, _arg_datafeeder/input_lengths_0_0, _arg_datafeeder/mel_targets_0_3, _arg_datafeeder/token_targets_0_5, _arg_datafeeder/linear_targets_0_2, _arg_datafeeder/targets_lengths_0_4)]]

[Rayhane-mamah]

Hello @a8568730, this is more likely related to your machine than to our code.
When using ctrl-c and retrying does it resume correctly?

[a8568730]

Understood.
We retry now, hope it's good news. :)

And also, thank you for being able to warm-start!
We make sure our latest checkpoint file in ./logs-Tacotron-2/taco_pretrained/ and then run the training script:

python train.py --model='Tacotron-2'

and it's perfectly resuming training from the checkpoint!

Loading checkpoint logs-Tacotron-2/taco_pretrained/tacotron_model.ckpt-5000
...

[Rayhane-mamah]

Alright, glad that's working out for your :) You can change the checkpoint interval by adding the --checkpoint_interval when launching train.py if you want to make it save more frequently. Example:

python train.py --model='Tacotron-2 --checkpoint_interval=1000

That saves checkpoints every 1000 steps. If you want the change to be permanent, you can apply it in this line:

https://github.com/Rayhane-mamah/Tacotron-2/blob/d13dbba16f0a434843916b5a8647a42fe34544f5/train.py#L105

I am closing the issue as it doesn't seem to be related to the repo's code itself. Feel free to open any further issues when you encounter any problems ;) Thanks for using our work!

[a8568730]

@Rayhane-mamah
Thank you for advice, we resume the training process and now running normally. <3
Would you mind to share the insight to us, how you find the problem is from our machine rather than code? We haven't figured out the machine problem yet.

[Rayhane-mamah]

The problem doesn't seem to be happening with other people and I am not able to reproduce, so the odds are small that it is code related :)