Rasa: Wrong Intent Classification using SVM suggestions needed

[jithurjacob]

Hey guys,

Let's say when we are using SVM to classify the intents, if a completely random text is entered by user it will also be tagged to some intent.

For eg. if a user enters "I love you" it will be assigned to one of the intents. How awkward it would be the if the bots response is which cuisine are you looking for?

I have been trying to solve this issue, need your inputs on this.

From my experience we can't rely on the confidence score given by the SVM, as for wrong predictions completely out of the vocabulary it is giving very high confidence score.

[amn41]

Yes I would take confidence scores with a grain of salt. Remember we are doing classification in 300 dimensions with very few examples. A good way to deal with this is to include lots of these examples in your training, with an out_of_scope label or similar.

[jithurjacob]

we have tried this as well. what will eventually happen is that number of samples in out_of_scope would be more than the number of intents. Thus the svm would get biased towards out_of_scope even for an expected intent

[alfredfrancis]

try LinearSVC with OneVsRestClassifier

[jithurjacob]

We have tried it. The same issue will arise SVC will classify the string to a particular intent even if the user input completely unrelated.

[alfredfrancis]

why don't you wrap SVC with onevsrest classifier ? you will get a False if input string doesn't belongs to any of the trained classes.

check this

[jithurjacob]

@alfredfrancis onevsrest doesn't provide this functionality as far as I know.

For example, use your chat bot framework and as shown in your youtube video train it for booking a cab. Now enter some random inputs like "I love ice cream" it will be classified to book_cab intent only.

[farouk-hf]

Hey guys,
i also have the same problem and it would be very helpful if someone can provide us with a solution
thx

[jithurjacob]

One of my friend has said that recast.ai is the only one in which he has seen this functionality. As far as I understoood from their application, they might not be using ML or DL but pure NLP. Could someone take a look at this ?

[tmbo]

I created a more general feature request handling this. #387