Ml5-library: Return probability with featureExtractor classification

[jrmedd]

I've been working up a sketch for a game incorporating transfer learning, using featureExtractor. I'd find it really handy to be able to return an object with classes and probabilities, just as I can with imageClassifier.

Syntactically, I'm not sure if there's merit to using .classify() to return a single result, andpredict() to return a object with classes/probabilities? I notice @shiffman bumped into this during one of his livestreams, and while https://github.com/ml5js/ml5-library/issues/192 is still open, maybe there's an opportunity to keep both.

[shiffman]

Oh interesting suggestion! When doing the tutorials I found myself expecting the labels and probability scores given that the ml5.ImageClassifier('MobileNet') returned these values. I think for simplicity's sake it's probably best to use classify() for both the MobileNet classifier and the ml5.FeatureExtractor (via MobileNet) classifier. . .and have both return an array of labels and probabilities.

But the idea of having two methods — one for detail, and one for just the top label — is an interesting one!

[davenep]

@jrmedd @shiffman
I think this feature will be very helpful. For example, if I have 5 categories and when testing an image, I want to display the probability of the image falling in each category.. just like the predict function in image classifier. It would be very useful in feature extractor too. I would love to know if anyone has found a way to do this. Thank you.

[shiffman]

This also aligns with the LSTM discussion in #221 as I think we are moving more towards returning an object for most ml5 methods.

[a4ankit]

We have done training of dataset and also able to load that custom model for 'FeatureExtractor.classification()', and I'm able to get results with a label. So, Is there any possibility to get a probability number with labels. As of now it only giving us labels.

[shiffman]

Thanks for bringing this up again @a4ankit. I think we should go ahead and implement this. Any volunteers? Maybe @yining1023 or @NHibiki? Revisiting this with #192 makes sense.

[a4ankit]

Thanks for response @shiffman, as I get to know there is another method 'FeatureExtractor.regression()' which is giving continous values by using regressor.addImage() and regressor.train() functions. We have following points which we need to know:

1. Is this used for the probability/confidence values of an input tag(image)?

2. Is it returning us the confidence values from the standard model(MobileNet) or custom model? if its return values from custom model, then is there any function to load custom model in regression() as we have saved a custom model using regression.save() function.

3. With in a Dataset, Does we can use 'FeatureExtractor.classification()' and 'FeatureExtractor.regression()' functions simultaneously for classifying and probability/confidence both of an single instance/inputHTML_Image_Tag?

[yining1023]

I think we could make the following examples:
Classification on labels:

featureExtractor = ml5.featureExtractor('MobileNet', modelReady);
classifier = featureExtractor.classification(video, videoReady);
classifier.addImage('dog');
classifier.addImage('cat');
classifier.classify(gotResults);
function gotResults(err, result) {
  const label = result.label; // 'dog'
  const confidences = result.confidencesByLabel; // {'dog': 0.999, 'cat': 0.001}
}

Classification on classIndex: (I'm not sure if many people will use it, but if they can also pass class index instead of a string label)

featureExtractor = ml5.featureExtractor('MobileNet', modelReady);
classifier = featureExtractor.classification(video, videoReady);
classifier.addImage(0);
classifier.addImage(1);
classifier.classify(gotResults);
function gotResults(err, result) {
  const class = result.class; // 0
  const confidences = result.confidences; // [0.999, 0.001]
}

For featureExtractor.regression(), it will still output a number that is between 0 - 1;

[shiffman]

Thanks @yining1023! I think we should probably match what the ml5.ImageClassifier does already (or change what it does). It'll make the most sense for users if the data format and terminology remains the same. At the moment, this what we get back from MobileNet:

Some questions:

1. Should we use "label" instead of "className"? (I think yes?)
2. Should we use "confidence" instead of "probability"? (I think yes? Note we have discussed using "confidence" with PoseNet as per #245.)

If so, both the feature extractor classifier and image classifier (plain MobileNet) would work as follows:

featureExtractor = ml5.featureExtractor('MobileNet', modelReady);
classifier = featureExtractor.classification(video, videoReady);
classifier.addImage('dog');
classifier.addImage('cat');
classifier.classify(gotResult);
function gotResults(err, result) {
  const label = result[0].label; // 'dog'
  const confidence = result[0].confidence;  // 0.99
} 

[shiffman]

@a4ankit I'm not sure I fully understand your question, but I don't believe there is a "confidence" value associated with our regression example. There is only the continuous floating point number returned by the "predictor". Still, I could imagine aligning the API by returning an object that contains the value, i.e.

regressor.predict(gotResult);
function gotResult(err, result) {
  const value = result.value;
}

I think the precedent that every single ml5 callback function returns an object is a good one and gives us the most flexibility moving forward.

[yining1023]

Ah, got it, it makes sense! Will work on it.
I will update feature extractor classifier and image classifier (plain MobileNet) to have result[0].label and result[0].confidence, and also update regression example to have result.value.

[a4ankit]

Thanks, @shiffman and @yining1023. For more convenient, can you please share the estimated enhancement date as we need to implement this for a running project.

[brunoneumann]

Hello guys! Sorry for the question... but there's a way to test this feature? I'm still getting just the label when I execute:
classifier.classify(image, (err, results) => { console.log(results); });

[shiffman]

Hi @brunoneumann we are working on maybe getting a release out today! Otherwise you'd have to build the library yourself to test as it's not in the most recent published version. We are trying to document this process better and welcome questions or ideas!

[brunoneumann]

Hello @shiffman! Wow, maybe today, that's amazing!!
I recently start a project I'm testing Ml5 and things are doing great. I'll let you guys know if I have more questions. Thanks for your efforts and for this incredible tool.

[thirukumars]

after creating our custom model,how we add it in live image classifier.With custom models we can only detect within the customly created model.if i want the custom model to detect with all other pre-trained model,how do i do this?
any one help me out

[joeyklee]

I believe this issue should now be resolved as the featureExtractor returns a label and confidence.

Thanks all for your participation here!