Modernmt: Neural training starts with warnings

[EtienneAb3d]

GPU training seems to be running.

Strangely, it is using only between 50% and 80% of GPU (not stable), on my 1080Ti card.

I got these warnings at screen:

INFO: (6 of 7) Preparing training data...                              DONE (in 1h 43m 42s)
INFO: (7 of 7) Neural decoder training...                             /home/lm-dev8/MMT2/ModernMT/build/lib/pynmt/nmmt/NMTEngineTrainer.py:177: UserWarning: self and other not broadcastable, but have the same number of elements.  Falling back to deprecated pointwise behavior.
  num_correct_t = pred_t.data.eq(targ_t.data).masked_select(targ_t.ne(Constants.PAD).data).sum()
/home/lm-dev8/MMT2/ModernMT/build/lib/pynmt/nmmt/NMTEngineTrainer.py:177: UserWarning: self and mask not broadcastable, but have the same number of elements.  Falling back to deprecated pointwise behavior.
  num_correct_t = pred_t.data.eq(targ_t.data).masked_select(targ_t.ne(Constants.PAD).data).sum()

PS: it seems that GPU usage is slowly growing up. After 30mn, it is much more between 95% and 100%.

[davidecaroselli]

Hi @EtienneAb3d

the GPU Utilization is in line with our internal tests. Regarding the warnings: yes, we are using deprecated functions from pytorch. This is not an issue right now so we decided to leave where they are for the moment: it does not interfere with the training process.

[davidecaroselli]

I noticed a notification with a message from you that I cannot find here, I suppose you removed it. Anyways, you can check more in details the training process with:

tail -f runtime/<engine_name>/logs/training.log

Unfortunately it is not possible to estimate remaining time, because the termination condition depends on the performance of the engine on the validation set and that is not predictable.

[EtienneAb3d]

Yes, I removed my message, because I found the answer by myself.

Certainly, here are some questions that will be explain in the documentation, when up to date...

I see that the log contains many informations about the network structure and its parameters.

Where may I tune the network structure, and the training parameters ?

What is the "acc" value ? BLEU ? Something else ?

What are the stop conditions ?

[nicolabertoldi]

q1: Where may I tune the network structure, and the training parameters ?
Actually, the network structure (number of layers, type and number of cells, etc) cannot be tuned,
You can tune only the initial learning rate by means of the "mmt tune" command.

q2: What is the "acc" value ? BLEU ? Something else ?
The acc metrics reported in the log is essentially the WER between the hyp generated by the network and the ref. This stands for both training acc and validation acc.

q3: What are the stop conditions ?
In the new version of MMT (v2.0), training terminates if the perplexity on the validation set does not improve for N consecutive validations. N is the parameter 'early_stop' and its default is 10.
Note that a validation is performed every X batches. X is the parameter 'validation_steps' and its default is 10000.

[nicolabertoldi]

a clarification on q1:
as mentioned before the network structure and training parameters are not really tunable, in the sense that you cannot automatically find the best setting for them.
but you can set your desired network structure and training parameters by editing a configuration file, which can be load when you start the training

./mmt create en it examples/data/train --neural --metadata config_file

'config_file' can contain any of the following parameters:
layers = 2
rnn_type = LSTM
rnn_size = 500
word_vec_size = 500
brnn_merge = sum
brnn = True
dropout = 0.3
max_grad_norm = 5
input_feed = 1
optim = sgd
tuning_max_learning_rate = 0.2
tuning_max_epochs = 10

If a parameter is not set, its default is used.

The name of parameters are quite expressive, but if you need more info about them, fill free to ask.

The parameters driving the training process (learning_rate, learning_rate_decay, etc.) are currently hardcoded, but we are working to make them settable in the command line.

[AndRossi]

Hello @EtienneAb3d,

Following your issue, I have written a neural network configuration page.

Basically, it describes the parameters that @nicolabertoldi has written in his answer, so it should be helpful for all users that need to manually set the network structure and params.

Since this issue seems solved, I am going to close it; however, for any further doubt or problem, do not hesitate to reopen it! :-)

Andrea