Modernmt: fa_build fails

[jgrivolla]

Using the latest Docker image I always get a failure on fa_build:

$ docker run -it --publish 8045:8045 modernmt/master bash
ubuntu@b197a1dda1e7:~/mmt$ ./mmt create en it examples/data/train --train-steps 6000

=========== TRAINING STARTED ===========

ENGINE:  default
CORPORA: 3 corpora
LANGS:   en > it

WARNING: No GPU for Neural engine training, the process will take very long time to complete.

INFO: (1 of 6) Corpora cleaning...                                     DONE (in 3s)
INFO: (2 of 6) Corpora pre-processing...                               DONE (in 2s)
INFO: (3 of 6) Aligner training...                                    
ERROR Unexpected exception: None

Traceback (most recent call last):
  File "./mmt", line 612, in <module>
    main()
  File "./mmt", line 558, in main
    actions[command](args)
  File "./mmt", line 145, in main_create
    builder.build()
  File "/home/ubuntu/mmt/cli/mmt/engine.py", line 632, in build
    self._build(resume=False)
  File "/home/ubuntu/mmt/cli/mmt/engine.py", line 704, in _build
    method(self, args, skip=skip, log=log_stream, delete_on_exit=self._delete_on_exit)
  File "/home/ubuntu/mmt/cli/mmt/engine.py", line 498, in __call__
    self._f(*args, **kwargs)
  File "/home/ubuntu/mmt/cli/mmt/engine.py", line 813, in _train_aligner
    self._aligner.build(corpora, log=log)
  File "/home/ubuntu/mmt/cli/mmt/engine.py", line 104, in build
    osutils.shell_exec(command, stdout=log, stderr=log)
  File "/home/ubuntu/mmt/cli/libs/osutils.py", line 57, in shell_exec
    raise ShellError(str_cmd, return_code, stderr_dump)
cli.libs.osutils.ShellError: Command '/home/ubuntu/mmt/build/bin/fa_build -s en -t it -i /home/ubuntu/mmt/runtime/default/tmp/training/preprocessed_corpora/train -m /home/ubuntu/mmt/engines/default/models/aligner/en__it.mdl -I 4' failed with exit code 127

It seems to be a problem with libboost. Here's the full log:

ubuntu@b197a1dda1e7:~/mmt$ cat runtime/default/logs/training.log 
2019-03-29 15:54:30,007 [INFO] - Training started: engine=default, corpora=3, lang_pair=en-it
2019-03-29 15:54:30,007 [INFO] - Training step "clean_tms" (1/7) started
2019-03-29 15:54:30,008 [DEBUG] - java -Xmx28902m -cp /home/ubuntu/mmt/build/mmt-3.3.1.jar -Dmmt.home=/home/ubuntu/mmt -Djava.library.path=/home/ubuntu/mmt/build/lib eu.modernmt.cli.CleaningPipelineMain -s en -t it --output /home/ubuntu/mmt/runtime/default/tmp/training/clean_corpora --input /home/ubuntu/mmt/examples/data/train
2019-03-29 15:54:33,542 [INFO] - Training step "clean_tms" completed in 3s
2019-03-29 15:54:33,542 [INFO] - Training step "preprocess" (2/7) started
2019-03-29 15:54:33,543 [DEBUG] - java -cp /home/ubuntu/mmt/build/mmt-3.3.1.jar -Dmmt.home=/home/ubuntu/mmt -Djava.library.path=/home/ubuntu/mmt/build/lib eu.modernmt.cli.TrainingPipelineMain -s en -t it --output /home/ubuntu/mmt/runtime/default/tmp/training/preprocessed_corpora/train --input /home/ubuntu/mmt/runtime/default/tmp/training/clean_corpora --dev /home/ubuntu/mmt/runtime/default/tmp/training/preprocessed_corpora/extracted_validation --test /home/ubuntu/mmt/engines/default/test_data
2019-03-29 15:54:35,667 [DEBUG] - java -cp /home/ubuntu/mmt/build/mmt-3.3.1.jar -Dmmt.home=/home/ubuntu/mmt -Djava.library.path=/home/ubuntu/mmt/build/lib eu.modernmt.cli.TrainingPipelineMain -s en -t it --output /home/ubuntu/mmt/runtime/default/tmp/training/preprocessed_corpora/validation --input /home/ubuntu/mmt/runtime/default/tmp/training/preprocessed_corpora/extracted_validation
2019-03-29 15:54:36,388 [INFO] - Training step "preprocess" completed in 2s
2019-03-29 15:54:36,389 [INFO] - Training step "train_aligner" (3/7) started
2019-03-29 15:54:36,389 [DEBUG] - /home/ubuntu/mmt/build/bin/fa_build -s en -t it -i /home/ubuntu/mmt/runtime/default/tmp/training/preprocessed_corpora/train -m /home/ubuntu/mmt/engines/default/models/aligner/en__it.mdl -I 4
/home/ubuntu/mmt/build/bin/fa_build: error while loading shared libraries: libboost_locale.so.1.58.0: cannot open shared object file: No such file or directory
2019-03-29 15:54:36,429 [ERROR] - Unexpected exception
Traceback (most recent call last):
  File "/home/ubuntu/mmt/cli/mmt/engine.py", line 704, in _build
    method(self, args, skip=skip, log=log_stream, delete_on_exit=self._delete_on_exit)
  File "/home/ubuntu/mmt/cli/mmt/engine.py", line 498, in __call__
    self._f(*args, **kwargs)
  File "/home/ubuntu/mmt/cli/mmt/engine.py", line 813, in _train_aligner
    self._aligner.build(corpora, log=log)
  File "/home/ubuntu/mmt/cli/mmt/engine.py", line 104, in build
    osutils.shell_exec(command, stdout=log, stderr=log)
  File "/home/ubuntu/mmt/cli/libs/osutils.py", line 57, in shell_exec
    raise ShellError(str_cmd, return_code, stderr_dump)
ShellError: Command '/home/ubuntu/mmt/build/bin/fa_build -s en -t it -i /home/ubuntu/mmt/runtime/default/tmp/training/preprocessed_corpora/train -m /home/ubuntu/mmt/engines/default/models/aligner/en__it.mdl -I 4' failed with exit code 127

This is on a machine without GPU, but so far wherever I have tried (e.g. on our cluster on nodes with GPU) it I had the same result. Shouldn't boost be provided within the image, or is this an issue with the (nvidia-)docker installation?

[jgrivolla]

Other libboost libraries are included in the image, but not libboost_locale:

ubuntu@b197a1dda1e7:~/mmt$ find . -name libboost*
./build/lib/libboost_system.so.1.58.0
./build/lib/libboost_program_options.so.1.58.0
./build/lib/libboost_thread.so.1.58.0
./build/lib/libboost_serialization.so.1.58.0
./build/lib/libboost_filesystem.so.1.58.0
./build/lib/libboost_iostreams.so.1.58.0

[mtresearcher]

apt-get install libboost-all-dev should fix this

[davidecaroselli]

Hi @jgrivolla

you're right, this is a problem with the binary release pack. I'm working right now in fixing the issue and updating the docker image.

Thanks!

[davidecaroselli]

Hi @jgrivolla

we have just pushed the fix. Now the docker image "latest", as well as the re-uploaded "v3.3.1" should work as expected.

Please try it yourself and let me know if it works in your environment.

[jgrivolla]

That part seems to work now. So I guess the fa_build issue is solved.

_Edit: I originally reported a problem with t2t-datagen but that turned out to be due to my local configuration (GPU was not working)._