Fasttext: How should the text file be structured for training?

[PhilipMay]

Hi.
How should the text file be structured for training?
Are there new lines allowed?
Are blank lines ok?
Should there be one sentence per line?

Thanks
Philip

[a11apurva]

Hi @PhilipMay ,

Following is the structure of a file used for supervised training.

 __label__A This is sentence 1. 
 __label__B This is sentence 2. 

For unsupervised training the file can have lines without labels.

This is sentence 1. 
This is sentence 2. 

• Yes, new lines are allowed.
• Not sure about blank lines, but you can do a quick experiment and check.
• Please see #545 for you last question.

Best,
Anubhav

[PhilipMay]

@a11apurva ok. Thanks. :-)

[PhilipMay]

@a11apurva: For unsupervised training: sould there be a space infront of the .?
More like this:

This is sentence 1 .
This is sentence 2 .

[EdouardGrave]

Hi @PhilipMay,

In general, it is a good idea to apply some sort of pre-processing / text normalization on your data before feeding it to fastText. This will make the learned classifier more robust. Examples of such pre-processing include tokenization, lowercasing, numbers/date normalization, etc. The kind of pre-processing you should apply is application (and language) dependent, which is why we decided not to include it in fastText. For example, casing might be important for some applications, but not for others.

Here are a few third party tokenizers: the Stanford Tokenizer, the Moses Tokenizer.

Best,
Edouard.

[fclesio]

@EdouardGrave I think we can close this one as we have the answer provided by @a11apurva

[PhilipMay]

Ok. Thanks.

[djstrong]

In word embeddings training, are new lines changing anything, i.e. a context crosses new lines?

[djstrong]

A context is not crossing new lines (#518).