Farm: Why does my Trainer not train?

[Chipandcharge]

Question
In several examples (from your github) the Trainer.train() function does not act at all. No epoch is started. Why?
Unfortunately there is no further context given then the picture, the status bar and the statement that it is killed after a while.

Additional context

[Timoeller]

Hey @Chipandcharge

Sure, we are happy to help. Could you please provide a bit of additional context, so that we can narrow down the bug together. In the future it would greatly improve response times if you could file a bug issue and fill out the fields you think could be relevant:

• What versions of FARM and transformers are you using?
• What is the setup of your system?
• Are you running the examples as they come, or do you add your own data/code?

You could try running the examples in colab, like in this example.

Hope that helps, let get FARM working on your end :D

[Chipandcharge]

Hi @Timoeller,

sure. Solved it by reducing the batch_size. Seems to be a ressource bottleneck.

Thanks!

[Timoeller]

Thanks for resolving the issue yourself.

So you are using the CPU with a large batch size, I guess?
It would be helpful to document roughly what happened. So other people facing the same issue find a solution to their problem. Thanks

[Chipandcharge]

Sure.

Wanted to test Farm and I started by utilizing ner.py example.

Versions:

• transformers: 2.4.0
• FARM: 0.4.1
• Python: 3.7.6

System:

• i7-8705G 3.1 GHz
• 8 GB RAM
• Linux Ubuntu

I didn't do any adjustment to the example provided by deepset. And the error occured. With some researche I figured out, that "killed" is a message from python, if the ressources are not enough. Then I adjusted the batch size step-by-step and ended up using batch size 5.
After all it took a whole night to train the model on the CPU.

Hope this helps a bit. Cheers.

[Timoeller]

Awesome, thanks for the insights.

And yes, training Language Models without GPU is quite time consuming.
Maybe you haven't seen it: In our readme there are links to colab notebooks, where you can utilize GPUs for free. That should speed up computations by about 30-50 times. If you want to use your own data though there might be privacy issues...