Machinelearning: How to get training progress info?

[seabluescn]

Hi，Suppose I have 10,000 training data. When I run Fit method, it takes a long time. During this period, how can I get real-time training progress information?
Thank You!

[PeterPann23]

Hi,

I have opened 2 issue on this topic that are currently under review #3685 and #3358.

You will find some comments in these threads that perhaps can be used for some sort of activity monitoring.

The short answer however is no, even though there are some ways to get feed back, none are really helping determine what's going on.

You have

1. Use the event MLContext.Log that will give you some text that you can use to parse

2.a You can, if the network you choose use early stopping to help mitigate wasting resources (doesn't tell you your progress though)

2.b Stop a training session periodically, test the current metrics and continue as shown here. Again not really progress reporting

1. Hoockup an WithOnFitDelegate that get's called by the network, most of them however do not loop and only call it when trained, some of them do. Have a look at the Microsoft.ML.LearningPipelineExtentsions line 138 for more info

2. Pull the source and alter the source code and publish progress using a delegate and or instrument using WMI

[PeterPann23]

also have a look at the answers in #3749
could help your issue as well

[justinormont]

As mentioned in https://github.com/dotnet/machinelearning/issues/3235, MLContext.Log() doesn't have a verbosity selection, so it's more of a firehose.

If a verbosity argument is added to MLContext.Log(), the log output from there should be human readable to see general progress.

I believe it's still hidden within the firehose of output and once the verbosity is scaled down, you should see messages like:

LightGBM objective=multiclassova
[7] 'Loading data for LightGBM' finished in 00:00:15.6600468.
[8] 'Training with LightGBM' started.
..................................................(00:30.58)    0/200 iterations
..................................................(01:00.9) 1/200 iterations
..................................................(01:31.2) 2/200 iterations
..................................................(02:01.4) 2/200 iterations
..................................................(02:31.9) 3/200 iterations
..................................................(03:02.5) 4/200 iterations
..................................................(03:32.9) 4/200 iterations
..................................................(04:03.6) 5/200 iterations
..................................................(04:34.4) 5/200 iterations
..................................................(05:04.8) 6/200 iterations

And naively extrapolating, there's around 2.7 hours left in the LightGBM training.

[seabluescn]

I tried the MLContext.Log event but found that I could not filter out the information I wanted.

[seabluescn]

I found a way to use CustomMapping: write a CustomMappingFactory subclass, input arbitrary, output as a fixed value such as 1.0f, and input it as a feature, then the Action method of the class will be called, you can write debugging code in this method.
See: https://github.com/seabluescn/Study_ML.NET/tree/master/MulticlassClassification_Mnist