Machinelearning: ImageLoadingEstimator for TensorFlow scoring should allow in-memory image streams as input in addition to images from files on drive

[CESARDELATORRE]

Right now the only way for ML.NET to load images is via ImageLoadingEstimator, which can load them only from disk files (as confirmed by @yaeldekel and Pete a few weeks ago).

However, it is a very common scenario in applications, such as a web app, where users submit images through Http, then the DataView/pipeline would load in-memory image streams (either BitMap, byte[], Image) instead of loading images from files in a folder on a disk/drive.

That's the right way to do it for many scenarios in web apps and services (Web APIs).
And for instance, you can do that when using TensorFlowSharp in C#. But we cannot in ML.NET, as of today.

When implementing this feature improvement in ML.NET, there could be the following two approaches:

• Modify schema comprehension to be able to map Bitmap fields/properties to Image columns of a data view.

• Add another version of ImageLoading transformer that loads/decodes the image from a byte vector, rather than from a disk file identified by path.

In any case, this is an important scenario to implement because not being able to load images from in-memory streams and only from files can be a big handicap in performance for on-line scenarios like the ones mentioned.

With the current implementation in ML.NET, the only workaround is to save the upcoming image from http and in-memory into a temporary file on the disk and load it from there. But that is a very "coarse/poor" workaround, not performant at all for a real application in production.

The following is a sample app I created for this online scenario where the user uploads an image from the browser into a service (Web API) and ultimately you get it as an in-memory image stream.

SEE CODE HERE:

https://github.com/CESARDELATORRE/TensorFlowImageClassificationWebAPI

• That web form uploads the image through Http into a service (Web API) in the server-side. At that point, the image is an in-memory image stream.

• In this implementation the sample app works because I implemented a workaround so the submitted image is temporarily stored as a file, then loaded from the file into the DataView through the pipeline...)

Basically, when the C# method in the Web API gets the image as an in-memory stream it should be able to load it directly in the DataView. The following code is an example:

        // Controller's method from Web API 
        [HttpPost]
        [ProducesResponseType(200)]
        [ProducesResponseType(400)]
        [Route("classifyimage")]
        public async Task<IActionResult> ClassifyImage(IFormFile imageFile)
        {
                if (imageFile.Length == 0)
                    return BadRequest();

                // WORKAROUND: Save image into a temporal file
                //Save the temp image image into the temp-folder 
                string fileName = await _imageWriter.UploadImageAsync(imageFile, _imagesTmpFolder);
                string imageFilePath = Path.Combine(_imagesTmpFolder, fileName);

                // Use image filename as the workaround...
                // Rest of the implementation with ML.NET API for scoring TensorFlow model...
                // ...

        }

To sum up:

I believe it is "a must" for ML.NET to be able to load in-memory image streams into the DataView to use those images when scoring TensorFlow models (in addition "from files") because of the mentioned on-line and in-memory scenarios that are pretty common.

[zeahmed]

I think #1609 is duplicate of this. Can we close #1609?

[CESARDELATORRE]

Let's close https://github.com/dotnet/machinelearning/issues/1609 issue when this new issue is added to 0.10 backlog, ok?

But I agree, since this issue https://github.com/dotnet/machinelearning/issues/2121 is related, but I'm providing further info about the impacted scenarios in this issue.

[mareklinka]

The project I'm currently working on is looking to use ML.NET in a scenario very similar to the one outlined in this issue. And while in our case performance is not that much of a factor, having the capability to score in-memory images would be a great simplification.

+1

[glebuk]

Actually ML.NET supports working with in-memory images already.
Loading images from files is just one way to get images into the system. You can also feed them with System.Drawing.Bitmap objects, wrapped in Microsoft.ML.ImageAnalytics.ImageType or directly from pixel matrices.
While ImageLoader works with paths of images, other Microsoft.ML.ImageAnalytics transforms work with ImageType, which is a wrapper of Bitmap. You just need to create an IDataView that contains ImageType column. You can convert form ImageType to n-dimensional vectors via ImagePixelExtractorTransformer and back via VectorToImageTransform
Please take a look at ImageTests.cs for more examples.

[CESARDELATORRE]

@glebuk @yaeldekel - I cannot find in https://github.com/dotnet/machinelearning/blob/master/test/Microsoft.ML.Tests/ImagesTests.cs a clear example of a pipeline loading the image from a memory stream.

In all the cases I see in those tests the images are loaded into the pipeline from files.

Gleb, Yael, can you create a specific sample/test that simply do that? (load the image from an in-memory stream)

For instance, after doing the following code in regular .NET Core, what are the specific steps to load the image into the pipeline to be able to use the TensorFlow estimator?

using System.Drawing;
using (var image = new Bitmap(System.Drawing.Image.FromFile(inputPath)))
{
    // TO DO:

I could get that Bitmap from Http, etc. The important point is to know how to load that memory stream image into the pipeline with a simple sample code.

Or in the case of an ASP.NET Core Web API I get the image like the following:

        // Controller's method from Web API 
        [HttpPost]
        [ProducesResponseType(200)]
        [ProducesResponseType(400)]
        [Route("classifyimage")]
        public async Task<IActionResult> ClassifyImage(IFormFile imageFile)
        {
            // TO DO:

I can convert from IFormFile to other image types, like here:
https://code.msdn.microsoft.com/How-to-resize-image-after-c8fce9b4

But we need a simple sample on how to load the in-memory image (BitMap?) into the pipeline and then score a TensorFlow model.
That is not clearly provided/tested on these tests: https://github.com/dotnet/machinelearning/blob/master/test/Microsoft.ML.Tests/ImagesTests.cs

[damir-usmanov]

Here's the resource i found the most useful:

https://docs.microsoft.com/en-us/dotnet/api/microsoft.ml.transforms.tensorflowmodel.scoretensorflowmodel?view=ml-dotnet-preview

See examples section.