Docs: Image recognition sample for ML.NET

[dotMorten]

I'd love to see a sample that trains and recognizes images (ie object detection)

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: e855e630-1a30-d138-4ac6-bfeefebf9aae
• Version Independent ID: afc0a7d1-d3c2-acf2-d4ea-d2565e22b237
• Content: ML.NET tutorials
• Content Source: docs/machine-learning/tutorials/index.md
• Service: dotnet-ml
• GitHub Login: @JRAlexander
• Microsoft Alias: johalex

[mairaw]

Thanks for your suggestion @dotMorten. @aditidugar @asthana86 @JRAlexander this is a request for a new tutorial for ML.NET. Please take a look.

[aditidugar]

Hey @dotMorten! We currently don’t support image recognition in ML.NET, but it is on our roadmap for future improvements. Once this is supported, we will revisit publishing a sample on this topic.

[JRAlexander]

Thanks, @aditidugar, for the quick follow up!

[TechnikEmpire]

@mairaw What is this blocked by?

[mairaw]

@TechnikEmpire as far as I know, ML.NET still doesn't support image recognition, so that's why I tagged as blocked. @CESARDELATORRE @asthana86 can you confirm that's still the case? Thanks!

[TechnikEmpire]

@mairaw Ok thanks for clarifying, I interpreted [blocked] to mean there was some other numbered issue that this feature was waiting on to be resolved.

[huanbd]

@mairaw I see in a newer version, exits nuget name Microsoft.ML.ImageAnalytics, and in Microsoft.ML have classes ImageLoader, ImageResizer, ImagePixelExtractor. Is that available for Image recognition ?
I am anxious to see sample about Image recognition.

[BattMonkey]

@huanbd @dotMorten
Not sure if this is what you're looking for, however the https://github.com/dotnet/machinelearning-samples has an image classification project (TensorFlowMLNETInceptionv3ModelScoring).
Were it takes a set of image files and the Inception model, then outputs it's predictions of what the images are.
More info can be located at: https://blogs.msdn.microsoft.com/dotnet/2018/09/12/announcing-ml-net-0-5/
Snippet of output
"

---

• Predictions

---

  ImagePath: ..\DeepLearning_TensorFlowMLNETInceptionv3ModelScoring\bin\Debug\netcoreapp2.1\assets\images\broccoli.jpg predicted as broccoli
  ImagePath: ..\DeepLearning_TensorFlowMLNETInceptionv3ModelScoring\bin\Debug\netcoreapp2.1\assets\images\bucket.png predicted as bucket
  ImagePath: ..\DeepLearning_TensorFlowMLNETInceptionv3ModelScoring\bin\Debug\netcoreapp2.1\assets\images\canoe.jpg predicted as canoe
  ImagePath: ..\DeepLearning_TensorFlowMLNETInceptionv3ModelScoring\bin\Debug\netcoreapp2.1\assets\images\snail.jpg predicted as snail
  ImagePath: ..\DeepLearning_TensorFlowMLNETInceptionv3ModelScoring\bin\Debug\netcoreapp2.1\assets\images\teddy1.jpg predicted as teddy
  ImagePath: ..\DeepLearning_TensorFlowMLNETInceptionv3ModelScoring\bin\Debug\netcoreapp2.1\assets\images\teddy5.jpg predicted as teddy

---

---

• Metrics for Image Classification

---

• Log Loss: 0

---

"

[TechnikEmpire]

Where can we make recommendations? It would be nice to see Microsoft leverage some of their own tech to make image recognition mass-marketable. Doing these things on the CPU for any real task is so slow that it's useless.

Leveraging c++amp or even the upcoming windows ML capabilities to accelerate this task would work, for example.

[CESARDELATORRE]

For image recognition, we are integrating ML.NET with TensorFlow. Specifically, you're be able to just score/run an existing trained TensorFlow model from ML.NET.

We have a preliminary version of this integration between ML.NET and TensorFlow since ML.NET 0.5, as explained in this Blog Post I wrote/published on Sept. 12th 2018:

https://blogs.msdn.microsoft.com/dotnet/2018/09/12/announcing-ml-net-0-5/

You can also kind of have "simple transfer learning" by extending an existing ML.NET pipeline that might be using a multi-class cassification with TensorFlow as a transform that scores images for each case, as this sample:

https://github.com/dotnet/machinelearning-samples/tree/master/samples/csharp/examples/DeepLearning_TensorFlowMLNETInceptionv3ModelScoring

We're also providing GPU support/usage when scoring/using TensorFlow models.

For training custom TensorFlow models you need to use other TensorFlow platforms, such as Keras.
In the future, we might support training TensorFlow models from ML.NET, but that is out of our current scope/backlog for the upcoming version.

Hope that helps. 👍

[TechnikEmpire]

Is the GPU coming via opencl or cuda? Thanks for the replies, I'll check the blog post again for that.

[CESARDELATORRE]

It'll be based on CUDA and CuDNN. Note that this is still in "work in progress" state.
See related PR from us, still not merged into ML.NET:
https://github.com/dotnet/machinelearning/pull/890

[TechnikEmpire]

Ah ok that's unfortunate. Given Nvidia's invasive clickwrap license requirement and of course being bound to a hardware vendor, it's not commercially viable. I don't understand why Microsoft can't use their own open technology to accelerate this task but if Microsoft wants to be an Nvidia shill then I guess that's the direction.

Thanks for the additional information.

[veikkoeeva]

@TechnikEmpire It would perhaps be interesting to integrate CNTK too, as noted at https://github.com/dotnet/machinelearning/issues/67. It has a .NET Standard library, but otherwise it's a native one (see intro page). They're on CUDA when it comes to HW acceleration, but as far as I know, other vendors and HW platforms than GPU are on the table.

Oh, they have image recognition samples online.

[TechnikEmpire]

Good call. I did some work with Intel and their mkl-dnn project where we enabled openmp support for msvc and it improved performance considerably. I opened a ticket on cntk to re-add the mkl-dnn backend to the CPU only version but who knows of they've done that yet.

That back end should make CPU only usage more feasible. I don't know about cntk having GPU acceleration other than Nshillia (also known as NAntitrust or NAntifreemarket) but if it's opencl and doesn't target opencl 1.2 it's useless for mass market too. That's why Nvidia has made sure to freeze opencl support at 1.2 so we're stuck in 2009 for features forever.

[TechnikEmpire]

Re CNTK accelerated CPU support: https://github.com/Microsoft/CNTK/issues/3266

[Cloudmersive]

Any update on this?

[JRAlexander]

We are publishing the above PR with an image classification tutorial with TensorFlow. If this doesn't help, @dotMorten, please reopen the issue. Object Detection is still on the list.