Machinelearning: How to dump intermediate data in pipeline?

[WladdGorshenin]

Hi,

What would be the best way to get intermediate data in a pipeline? I'd like to debug data transformation steps.

Looking forward for reply

[rauhs]

I just want to note that I'd also very much appreciate a way to dump the intermediate data view in the pipeline. The more transformations I do the less I'm certain about how this all plays together. Even a simple way to just print the data as JSON would be great. Then one can feed the pipeline 2-3 artificial instances and see the pipeline result.

[Zruty0]

It is great to see interest in this scenario, as this is something we specifically want to enable in #583 via #581

[WladdGorshenin]

Hi,

any updates on the topic?

[Zruty0]

We are working on the new API: https://github.com/dotnet/machinelearning/projects/9

After this is done, you will be able to inspect the intermediate results of pipeline steps much similar to this test:
https://github.com/dotnet/machinelearning/blob/master/test/Microsoft.ML.Tests/Scenarios/Api/Estimators/FileBasedSavingOfData.cs

We will add SaveAsText extension to save data as text too.
We also have some plans to build custom Visual Studio visualizers for IDataView, which would enable you to see the values in the debugger.

[abgoswam]

We now have a couple of unit tests showcasing this functionality . Please check out New_Visibility and Visibility

DRI RESPONSE : Planning to close this issue (resolved)

[Ivanidzo4ka]

Please see:
https://github.com/dotnet/machinelearning/blob/master/docs/code/MlNetCookBook.md#how-do-i-look-at-the-intermediate-data