Modernmt: MMT4Monitor

[EtienneAb3d]

Waiting for an indication on a tool to monitor the MMT4 training, I wrote my own. Here is the small piece of Java code, using gnuplot to draw the learning curves:

import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;

public class MMT4Monitor {

    public MMT4Monitor() {
    }

    public void process(String aLogPath) throws Exception {
        BufferedReader aBR = new BufferedReader(
                new InputStreamReader(new FileInputStream(aLogPath)
                ,"UTF8"));
        BufferedWriter aBW = new BufferedWriter(
                new OutputStreamWriter(new FileOutputStream(aLogPath+".series")
                ,"UTF8"));

        String aLine = null;
        int aCountValid = 0;
        double aLoss = 0.0;
        double aPPL = 0.0;
        while((aLine = aBR.readLine()) != null){
            if(!aLine.startsWith("| epoch ")){
                continue;
            }
            if(aLine.indexOf("valid on")>=0){
//              System.out.println(aLine);
                String[] aToks = aLine.split("[|]");
                int aEpoch = Integer.parseInt(aToks[1].replaceAll("[^0-9.]", ""));
                double aVLoss = Double.parseDouble(aToks[3].replaceAll("[^0-9.]", ""));
                double aVPPL = Double.parseDouble(aToks[5].replaceAll("[^0-9.]", ""));
                aBW.write(aCountValid++
                        +"\t"+aEpoch
                        +"\t"+aVLoss
                        +"\t"+aVPPL
                        +"\t"+aLoss
                        +"\t"+aPPL
                        +"\n");

            }
            if(aLine.matches("[|] *epoch *[0-9]+: *[0-9]+ */ *[0-9]+ *loss=.*")){
//              System.out.println(aLine);
                String[] aToks = aLine.replaceAll(" +", " ").split("[ ,]");
                aLoss = Double.parseDouble(aToks[6].replaceAll("[^0-9.]", ""));
                aPPL = Double.parseDouble(aToks[10].replaceAll("[^0-9.]", ""));

            }
        }
        aBW.flush();
        aBW.close();

        aBW = new BufferedWriter(
                new OutputStreamWriter(new FileOutputStream(aLogPath+".plt")
                ,"UTF8"));
        aBW.write("set grid\n"
                + "set xlabel 'Validations'\n"
                + "set format y \"%6.2f\"\n"
                + "set ytics 1 nomirror tc lt 2\n"
                + "set ylabel 'LOSS' tc lt 2\n"
                + "set yrange [0:]\n"
                + "set format y2 \"%6.2f\"\n"
                + "set y2label 'PPL' tc lt 3\n"
                + "set y2range [0:20]\n"
                + "set y2tics 1 nomirror tc lt 3\n"
                + "plot \""+new File(aLogPath).getName()+".series\""
                + " using 1:2 with lines t \"EPOCH\""
                + " , \"\" u  1:3 with lines t \"V-LOSS\""
                + " , \"\" u  1:4 with lines t \"V-PPL\""+ " axes x1y2"
                + " , \"\" u  1:5 with lines t \"LOSS\""
                + " , \"\" u  1:6 with lines t \"PPL\""+ " axes x1y2"
                + "\n"
                + "pause -1\n"
                );
        aBW.flush();
        aBW.close();

        aBW = new BufferedWriter(
                new OutputStreamWriter(new FileOutputStream(aLogPath+".sh")
                ,"UTF8"));
        aBW.write("cd \""+new File(aLogPath).getParent()+"\" \n");
        aBW.write("gnuplot "+aLogPath+".plt"
                +"\n");
        aBW.flush();
        aBW.close();

        aBR.close();

        final Process proc = Runtime.getRuntime().exec(new String[]{"sh", aLogPath+".sh"});
        proc.waitFor();

    }

    public static void main(String[] args) {
        try{
            String aLogPath = "/mnt/SSD/MMT4beta/modernmt-versions-v4.0/runtime/DEFR_Cew/logs/training.log";
            new MMT4Monitor().process(aLogPath);
        }
        catch(Throwable t){
            t.printStackTrace(System.err);
        }
    }

}

[EtienneAb3d]

For a more comfortable ending, add --persist option to the gnuplot command:
aBW.write("gnuplot --persist "+aLogPath+".plt"+"\n");
And remove the waiting line:
// proc.waitFor();

[davidecaroselli]

With last commit I have added Tensorboard support:

./mmt create en it examples/data/train/ --tensorboard-port 8066

During training ModernMT will automatically launch a tensorboard instance on specified port.

[mindaugasDEV]

Thanks, @davidecaroselli , tensorboard works fine, I can confirm.

However, it does not show perplexity, although training.log contains PPL value.
1) What shall be customized by the user to see PPL it in Tensorboard dashboard at localhost:8066? It shows many variables, but not PPL.
2) Can you add BLEU value to tensorboard? In MMT3 tensorboard was displaying approx_bleu_score variable nicely and it was the main indicator for me to decide when to stop training (although in discussion it was decided that PPL was the main indicator ).

The last line from training log. Tensorboard dashboard shows most variable except PPL (and no BLEU))

| epoch 080: 5600 / 26137 loss=3.883, nll_loss=2.257, ppl=4.78, wps=11747, ups=9, wpb=1352.994, bsz=108.049, num_updates=2.07042e+06, lr=4.39542e-06, gnorm=5.928, clip=0.000, oom=0.000, wall=29076, train_wall=219891

[davidecaroselli]

Hi @mindaugasDEV

with the switch from Tensorflow to Fairseq the training process is totally changed. I have notice a lack of infos in Tensorboard too, but this is something we cannot directly control because it depends on fairseq framework. We are using the latest version (0.6.2), I hope there will be more infos in the upcoming releases.

BTW the loss is as good as perplexity to decide when to stop the training so you can safely for this purpose.