Yolact: visualizing loss curve

[explorer359]

[dbolya]

In case you closed this by accident, I'll paste earlier instructions I wrote:

To view training performance over time, log your console output to a file. Then from the project's root directory run
python -m "scripts.plot_loss" <my_log_file>
to plot the loss over time.

To plot validation mAP over time, use
python -m "scripts.plot_loss" <my_log_file> val

[jhtao1860]

hi where is the log file？

[dbolya]

@jhtao1860 I meant a log of the console output. You have to generate that yourself by running your command like this:
python train.py (...) >>"<my_log_file>" 2>&1
>>"<my_log_file>" will append the console output to <my_log_file> and 2>&1 will redirect any errors to standard out so that it can append those to that same file. Use >"<my_log_file>" if you want to overwrite instead of append.

[jhtao1860]

@dbolya Thank you！ Another question: "loss_types = ['B', 'C', 'M', 'P', 'D', 'E', 'S']" ,what do the 'P', 'D', 'E', 'S' mean?

[dbolya]

@jhtao1860 The key is in multibox_loss.py:

# Loss Key:
#  - B: Box Localization Loss
#  - C: Class Confidence Loss
#  - M: Mask Loss
#  - P: Prototype Loss
#  - D: Coefficient Diversity Loss
#  - E: Class Existence Loss
#  - S: Semantic Segmentation Loss

The ones that aren't used, I found to not improve performance. If you want to enable them you'll have to wade through all the settings in config.py.

[VinniaKemala]

@dbolya
Hi,
I get the following error when I run this command: python -m "scripts.plot_loss" logs/custom_yolact_plus_im550_resnet50.log
What could be the reason for such an error?

Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/srv/algo/yolact/scripts/plot_loss.py", line 78, in <module>
    plot_train(data['train'])
  File "/srv/algo/yolact/scripts/plot_loss.py", line 56, in plot_train
    if data[0]['s'] is not None:
IndexError: list index out of range

[dbolya]

@VinniaKemala So that script accepts stdout dumps, not the new *.log files. I still haven't gotten around to writing a similar script for the new log format, but the new logger is much more powerful. I'll write a short guide here that I'll transition into a longer one later:

Start a python shell (or notebook) in the root yolact folder. To start, grab a visualizer with:

from utils.logger import LogVisualizer
vis = LogVisualizer()

Each log file is split into training "sessions" where every time you restart / resume from training with the same config it opens up a new session. To check what sessions a log file has, simply run:

vis.sessions('logs/<your_log_file>.log')

This will print out all the sessions in the log file along with how many iterations (entries) and the time elapsed in training that session. If you have resumed from training, you can also use multiple sessions, so pick out the session ids you want to use and then load that session (or sessions) with:

vis.add('logs/<your_log_file>.log', session=<session_id>) # If multiple sessions, use a list of ids

If you want to compare multiple different logs / sessions, you can simply add more logs to the visualizer in the same way.

Then, to plot the total training loss you can go:

vis.plot('train', 'x.data.iter * s.data.batch_size', 'x.data.loss.T', smoothness=1000)

You can adjust the smoothness parameter as desired. The parameters are entry type ("train"), x axis (here, the number of total images by using iterations * batch_size), and the y axis (the total loss, you can change this to any of the short codes for the other losses, e.g. "M" for mask).

For whatever reason I wrote a whole query system for this but I guess I'll explain that in the full guide instead ¯\_(ツ)_\/¯

To plot the validation accuracy over training, you can go:

vis.plot('val', 'x.data.epoch', 'x.data.mask["all"]')

This will plot the 0.5:0.05:0.9 mAP metric for mask. You can change the x axis to box["all"] for the same but box mAP, and you can also put numbers instead of all for different IoUs (e.g., x.data.mask["50"] for 0.5 IoU).

You can also do bar plots for a specific epoch with

vis.bar('val', 'x.data.mask', x_idx=-1)

which will plot all the mask IoU APs for the last entry in validation (use 0 for first, etc. and use box instead of mask for box APs).

There's a lot more you can do with this, but I haven't gotten around to writing a full guide. And hmm I should include this as a notebook huh.

[VinniaKemala]

@dbolya Thank you for your elaboration! It's working!

[stbjelcevic]

@dbolya any updates on a full guide to using the visualizer? What you have above is a great start!

[Traderain]

@dbolya you should pin this