Opennmt-py: how to set a proper -train_steps during training a seq2seq model

[alphadl]

background: because of the changing of the parameters in the latest OpenNMT-py version , the parameter "-epochs " has changed to "-train_steps".

question: If I need to train a seq2seq rather than transformer model by using opennmt recently, how to determine the number of training epochs? 13 or 100000 ? （Because these two Numbers are not in the same order of magnitude, I am quite confused)

[vince62s]

it all depends on your training dataset. You can do the math in sentence batch mode. in token mode it's a bit trickier.
if you have 1000000 segments, and use batch_size 64, it means you have 15625 steps to finish 1 epoch.
However if you use accum X or more than GPU, it's also different.

[alphadl]

I see , thanks a lot~

[alphadl]

I assume that segments in your answer means the number of sentence pairs ~ so , If I have 500w sentence pairs and the batch_size is 128, each epoch needs 500w/128 ≈ 4w steps ?

[alphadl]

btw, can you explain the meaning of "-max_generator_batches" , is it useful during decoding?

[vince62s]

yes to your first question.
max_generator_batches is just a mechanism at training time to parallelize the soft max calculation. (no use at inference)

[alphadl]

okay , thanks ~~~

another question :

when I use multi-gpu option, an error was reported:

Traceback (most recent call last):
File "OpenNMT-py-path/train.py", line 37, in
main(opt)
File "OpenNMT-py-path/train.py", line 22, in main
multi_main(opt)
File "OpenNMT-py-path/onmt/train_multi.py", line 20, in main
mp = torch.multiprocessing.get_context('spawn')
AttributeError: 'module' object has no attribute 'get_context'

actually, I looked up the officially document of PyTorch 0.4.1 [https://pytorch.org/docs/stable/multiprocessing.html], they haven't provided the attribute "get_context", how could I tackle this problem if I want to use multi-gpu

[vince62s]

Provide the command line
However the forum is better suited for question

[merc85garcia]

Hello,
I am trying the Transformer model in OpenNMT-py I am using the last version. When I use the option -train_steps in order to train a model longer, the training process does not start, after charging the train dataset everything stops without giving an error. Do you have any idea of why this is happening? Thank you!

[kingkf]

"AttributeError: 'module' object has no attribute 'get_context'"===> I met the same question, I solved it by using python3. I hope can help you

[merc85garcia]

Hi,
I would like to know how to set one epoch using Transformer model. I use only 1 GPU, the batch_size is 4096 tokens and my sequences have seq_length=100 tokens. Does this mean that in each update I will process 41 sequences? This is not what I get in the training logs, I guess it is is because my sequences batches can have less than 100 tokens.

Is there any method to know one epoch in advance using Transformer? Thank you!

[francoishernandez]

the batch_size is 4096 tokens and my sequences have seq_length=100 tokens. Does this mean that in each update I will process 41 sequences?

There is a 'pooling' mechanism to optimize training performance.
Basically:

• the equivalent of pool_factor * batch_size examples are drawn from the dataset
• these examples are sorted in length
• batches are formed (with examples of approx. the same size)
• batches are shuffled.

Is there any method to know one epoch in advance using Transformer? Thank you!

The batching scheme does not depend on the architecture.
You can probably estimate the number of steps an epoch represents from the total number of tokens in your data / number of tokens per batch, as batches will be mostly homogeneous in length and without much padding thanks to pooling.
(This is valid for single dataset only, when using several datasets with different weights this is much harder.)
Also, you can see in the logs, when you have rolled over all your shards.