Modernmt: how shall tell MMT to use the TMX data for training, not only corpora file pairs?

[mindaugasDEV]

Hello,

Instead of using training monolingual file pairs, I have tried to use a TMX file with proper language code.
However, the command ./mmt create en it example/data/train did not find TMX file there (as I deleted the file pairs and only left bilingual en-it TMX).

The script gave a message: ERROR Illegal Argument: you project does not include EN-IT data

Main question: how shall we use TMX files as training data?

Additional questions:
Can we also use monolingual file pairs together with TMX files in training?
Can the language codes in TMX file (or in extensions of file pairs) contain regional codes, eg. EN-GB (for English British), EN-US, or PT-BR, PT-PT? My TMX did not contain that regional code, was a simple EN IT TMX, therefor the reason for error message lies somewhere else.

Thanks!

Regards,
Mindaugas

[davidecaroselli]

Hi @mindaugasDEV ,

regarding the TMX import, you are doing everything correctly. The problem is a bug in the scripts that check for the presence of a correct corpus; that's why you are hitting that error message. I'm working to fix that, thanks for the report!

Can we also use monolingual file pairs together with TMX files in training?

Yes, you can use both monolingual files for the language model and bilingual parallel corpus (.en + .it) in the same folder with your TMX.

Can the language codes in TMX file (or in extensions of file pairs) contain regional codes, eg. EN-GB (for English British), EN-US, or PT-BR, PT-PT?

Yes, but please note that the current implementation trims the language code to the first two letters: in this way you can use TMX with lang "EN-GB" and "EN-US" in the same engine, they will be considered as "EN" TMX.

Cheers,
Davide

[davidecaroselli]

@mindaugasDEV the fix has been applied to the master branch: you can now train an engine with a folder that contains only TMX files.

In order to have the fixed version, you can build from sources (you can find instructions in the INSTALL.md file).
If you want instead to apply a quick fix, you can replace the scripts folder with the new one and everything will work, the fix only changed python scripts.

You can download a zip archive with the new sources directly from github: Download ZIP

[mindaugasDEV]

I tried (overwrote SCRIPTS folder with new files), then recompiled.

First it complained that no TMX14.dtd is found (then I added the missing file, also attached here

), later it gave the below message:
ERROR Unexpected exception:
SAXParseException('error in processing external entity reference',)
Traceback (most recent call last):
File "./mmt", line 440, in
main()
File "./mmt", line 406, in main
main_create(args)
File "./mmt", line 383, in main_create
debug=args.debug, steps=args.training_steps, split_trainingset=args.split_corpora)
File "/home/matecat/ModernMT/scripts/engine.py", line 84, in build
bilingual_corpora, monolingual_corpora = ParallelCorpus.splitlist(source_lang, target_lang, roots=roots)
File "/home/matecat/ModernMT/scripts/mt/init.py", line 54, in splitlist
corpora = ParallelCorpus.list(directory)
File "/home/matecat/ModernMT/scripts/mt/init.py", line 33, in list
corpora = [ParallelCorpus.build(name, root, langs) for name, langs in name2langs.iteritems()]
File "/home/matecat/ModernMT/scripts/mt/__init.py", line 39, in build
return TMXParallelCorpus(name, root) if langs is None else FileParallelCorpus(name, root, langs)
File "/home/matecat/ModernMT/scripts/mt/__init.py", line 158, in init
ParallelCorpus.init(self, name, root, self.get_langs(name, root))
File "/home/matecat/ModernMT/scripts/mt/__init.py", line 151, in __get_langs
parser.parse(f)
File "/usr/lib/python2.7/xml/sax/expatreader.py", line 107, in parse
xmlreader.IncrementalParser.parse(self, source)
File "/usr/lib/python2.7/xml/sax/xmlreader.py", line 123, in parse
self.feed(buffer)
File "/usr/lib/python2.7/xml/sax/expatreader.py", line 214, in feed
self._err_handler.fatalError(exc)
File "/usr/lib/python2.7/xml/sax/handler.py", line 38, in fatalError
raise exception
xml.sax._exceptions.SAXParseException: examples/data/train/tmx14.dtd:1:0: error in processing external entity reference
tmx14.zip

[mindaugasDEV]

Our TMX is created with Xbench program (www.xbench.net), I noticed that tmx14.dtd version on the web differ. Maybe you can try with your TMX file and then include DTD version in the MMT release @davidecaroselli :)

[davidecaroselli]

Hi @mindaugasDEV

can you share the TMX in order to let me reproduce the bug? Of course you can "anonymize" it if it's a private TM. I suspect that's some kind of different XML header that is producing the issue because the fix worked without problems with the TMX files I use for training the engines.

Cheers

[davidecaroselli]

@mindaugasDEV

finally I've finished to fix the TMX support. Thanks to your feedback, I was able to fix a series of small bugs that prevented the tmx import to complete successfully.
Now the system is able to import your TM (changing the lang from LT to another supported language).

In order to test the fixed version of the software, you can follow the instructions here in order to build ModernMT from source.

Thanks again for your contribution.

• Davide