Fasttext: Is fastText support multi-label classification with sigmoid?

[Derekkk]

I know this issue has been asked for several times but I'm still confused now.

So here I want to do a classification task using fastText, and each sample may belong to one or more classes, which is multi-label classification. I guess using sigmoid activation function instead of softmax would solve the problem, so I wonder if there is any related options provided in fastText, or is there any other solutions for multi-label classification? Massive thanks!

[EdouardGrave]

Hi @Derekkk,

You can actually use the softmax when performing multi-label classification (and this is the recommended method for now). At test time you can specify the number of label that should be predicted for each example, or the threshold above which labels should be predicted, using the following command:

./fasttext predict model.bin test.txt <K> <T>

This will predict at most K labels, which have a score larger than T. By default, K=1 and T=0.0. If you want to predict all the labels which are above a given threshold, you should set K to the number of classes in your problem.

You can also try to train the model with sigmoid activation instead of the softmax, by using the negative sampling loss, with the command line option -loss ns. This will correspond to do one-vs-all training, where the negative labels are randomly sampled. However, this option is not supported at test time yet, since the softmax is still used to compute to label scores. We will probably implement full support for this in the near future.

Best,
Edouard

[Derekkk]

Hi @EdouardGrave,

Thanks a lot for your answer. As far as I know, using softmax will give a probability distribution, and the sum of probabilities of all labels will be 1. Let's say we want to do a multi-label classification with 5 classes, and the true label for a sample X is Y_1 and Y_2. In test time, after applying softmax function the result might be [0.7, 0.2, 0.05, 0.03, 0.02] for [Y_1, Y_2, Y_3, Y_4, Y_5]. In this case, it's not easy to choose a threshold. If I set threshold to be 0.5, the label it return will be only Y_1, not Y_1 and Y_2.

I haven't read the source code in detail, but just want to check if this case can be solved by the method you refer in your answer? Thanks again for your help!

[EdouardGrave]

Hi @Derekkk,

If each example of your dataset contains many labels, you should pick a threshold smaller than 0.5. For example, I used this method on a dataset containing approximately three labels per example, and the threshold giving the best results was 0.1. One way to chose the threshold is to find the best one on a validation set.

I hope this answer your question.

Best,
Edouard

[Derekkk]

Hi @EdouardGrave ,

It seems c-v would be very important in the multi-label classification with softmax. Still hope in the future sigmoid option can be added since it is very useful, and also makes this more convenient and extendable. Thanks again for your answer.

Best,

[livida123]

I kinda have the same issue,I don't know whether the fasttext.supervised() will run all the labels to train the model or it just randomly pick one ? Same for classifier.test(), if it just randomly choose 1, the top k precision could be very low since it ignores others. So if you figured this out, please let me know. This troubles me a lot. And it appears that it only support 3 loss functions(softmax, ns, hs).

[jonDel]

Just use keras, much more friendly, supported and easy to integrate into your pipeline. See the example here: https://github.com/keras-team/keras/blob/master/examples/imdb_fasttext.py

[0x0004]

@jonDel So how did the result compare with fasttext?

[Celebio]

Hi all,
The commit 8850c51b972ed68642a15c17fbcd4dd58766291d enables the classification with n independent sigmoids.

Actually you can obtain similar performance results with softmax loss. But with ova loss, it is easier to obtain decent performance, just set k to -1 (meaning unlimited number of predictions) and threshold to 0.5 for example :
/fasttext test model_cooking.bin cooking.valid -1 0.5

Best regards,
Onur

[daisukelab]

Hi Onur or @Celebio,
Thanks for announcement, I tried and it works very good for my use.
It's great, thanks again!

[Celebio]

You are welcome @daisukelab.

A section on multi-label classification has been added to the tutorial.

Thank you all for your feedback!

Regards,
Onur