Fasttext: Precision and Recall values Equals

[loretoparisi]

When evaluating a text classifier model, I always have that P and R have the same values like

{ N: '2191', P: '0.693', R: '0.693' }

This happens for k=1, no matter of the test size N, or training set size.

This issue was depicted https://github.com/facebookresearch/fastText/issues/212 as well.

[cpuhrsch]

Hello @loretoparisi ,

Thank you for your post. It looks like you're using a third party library to use fastText. Unfortunately we can't help you with this. You're probably best advised to follow up with the maintainer of the package, unless you can reproduce an error with the fasttext binary itself (which is part of this repository). Feel free to reopen this issue at any time if this is indeed the case.

Thank you,
Christian

[giacbrd]

This is actually related to fastText evaluation approach: https://github.com/facebookresearch/fastText/issues/93

[loretoparisi]

@giacbrd thank you I will take a look!
@cpuhrsch I think it's what Giacomo pointed out, since my wrapper is just a process fork of fasttext binary, but called by node.

[cpuhrsch]

Hello @loretoparisi, @giacbrd,

As @EdouardGrave mentioned in #93, we compute the multi-class precision and recall. We want to make sure our code stays as simple as possible. Merging https://github.com/facebookresearch/fastText/pull/35 adds additional complexity and introduces a feature, that could be computed after the fact by using predict. It'll increase the amount of code to maintain and is not orthogonal to the core functionality of this library (or the functionality of the Matrix class). I understand that this can be frustrating, as it requires a bit more work on your end, but please also understand that we aim to do a good as job as we can in providing fast and efficient models and that this means keeping a minimal codebase.

As a post-processing option you could use sklearn's confusion matrix option. For example, you could compute the output of each test sentence using the predict subcommand, parse the results and compare it to the label's mentioned in the .test file you used to compute the P and R values you reference above. If your test results need to be online you could use predict's stdin option. All of this could then be wrapped into a small tool and added to your PATH variable (e.g. ft_cm <predict> <test>).

In fact I'm adding an example of this, please let me know if this is what you're looking for (or if there are any mistakes in it).

Consider the following script

#! /data/users/cpuhrsch/anaconda3/bin/python

import argparse
import numpy as np
from sklearn.metrics import confusion_matrix


def parse_labels(path):
    with open(path) as f:
        return np.array(list(map(lambda x: int(x[9:]), f.read().split())))


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description='Display confusion matrix.')
    parser.add_argument('test', help='Path to test labels')
    parser.add_argument('predict', help='Path to predictions')
    args = parser.parse_args()
    test_labels = parse_labels(args.test)
    pred_labels = parse_labels(args.predict)
    eq = test_labels == pred_labels
    print("Accuracy: " + str(eq.sum() / len(test_labels)))
    print(confusion_matrix(test_labels, pred_labels))

Consider it's usage and output for the classification example.

cut -f 1 -d ',' data/dbpedia.test > data/dbpedia.testlabel
./fasttext predict result/dbpedia.bin data/dbpedia.test > pexp
./ft_cm.py data/dbpedia.testlabel pexp
Accuracy: 0.984628571429
[[4801   36   11    5    8   28   50    5    0    1    3    7    6   39]
 [  34 4925    1    0    5    0   32    0    0    0    0    0    0    3]
 [  20    2 4868   13   57    0    2    0    0    0    0   15    6   17]
 [   1    1   23 4960   12    0    0    0    0    1    0    2    0    0]
 [   6    2   63    9 4910    3    3    0    0    0    0    1    0    3]
 [  18    0    1    0    3 4960    9    3    0    0    2    0    2    2]
 [  58   29    1    0    3   12 4864   23    2    0    0    1    1    6]
 [   0    0    0    0    0    0   21 4968    8    1    1    0    0    1]
 [   1    0    0    0    0    0    7   16 4975    0    1    0    0    0]
 [   2    0    2    0    1    3    0    5    0 4961   24    0    0    2]
 [  12    1    1    0    0    3    0    4    0   37 4942    0    0    0]
 [   3    0   15    1    0    2    0    0    0    0    0 4952   14   13]
 [   7    2    4    0    0    6    0    1    0    0    0   11 4940   29]
 [  36    7   17    1    3    5    4    2    0    4    0    4   19 4898]]

Thanks,
Christian

[loretoparisi]

@cpuhrsch Thank you so much 🥇

I perfectly understand your point about the PR, and that one is the reason why I have chosen FastText rather than other tensor based libraries, etc.
I think that your script is exactly what we were looking for!

[loretoparisi]

Hey @cpuhrsch I had an error with your python script:

$ ./confusion.py /root/dataset_test.csv /root/pexp
Traceback (most recent call last):
  File "./confusion.py", line 18, in <module>
    test_labels = parse_labels(args.test)
  File "./confusion.py", line 10, in parse_labels
    return np.array(list(map(lambda x: int(x[9:]), f.read().split())))
  File "./confusion.py", line 10, in <lambda>
    return np.array(list(map(lambda x: int(x[9:]), f.read().split())))
ValueError: invalid literal for int() with base 10: '

I have my pepx file like

$ head -n5 /root/pexp
__label__spam
__label__verified
__label__verified
__label__spam
__label__verified

and the test file it's ok ( I have used it for training).

It seems it's due to the

def parse_labels(path):
    with open(path, 'r') as f:
        return np.array(list(map(lambda x: int(x[9:]), f.read().split())))

where you are getting out he label from the __label__verified, right?

[loretoparisi]

...okI had to convert back my labels first:

awk '{ $1="__label__" tolower($1)}1' /root/dataset_test.csv > /root/dataset_test_norm.csv

but I'm still getting

$ ./confusion.py /root/_dataset_test_norm.csv /root/pexp
Traceback (most recent call last):
  File "./confusion.py", line 18, in <module>
    test_labels = parse_labels(args.test)
  File "./confusion.py", line 10, in parse_labels
    return np.array(list(map(lambda x: int(x[9:]), f.read().split())))
  File "./confusion.py", line 10, in <lambda>
    return np.array(list(map(lambda x: int(x[9:]), f.read().split())))
ValueError: invalid literal for int() with base 10: 'spam'

I don't get why since the labels are ok now in the dataset_test.csv

__label__spam      dkfjkjdfdfk dfkldfkjfdkj kdfkfdkjdfjkdf
__label__verified   in the darkness I am alone
__label__verified   hui yaguang nitu reteshi

Since I was not sure of the parsing of the other cols I have removed them from test file:

cut -d$'\t' -f1 /root/dataset_test_norm.csv > /root/dataset_test_norm_first.csv
$ head -n10 /root/dataset_test_norm_first.csv 
__label__spam
__label__verified
__label__verified
__label__spam
__label__verified
__label__verified

and still getting

$ ./confusion.py /root/dataset_test_norm_first.csv /root/pexp
Traceback (most recent call last):
  File "./confusion.py", line 18, in <module>
    test_labels = parse_labels(args.test)
  File "./confusion.py", line 10, in parse_labels
    return np.array(list(map(lambda x: int(x[9:]), f.read().split())))
  File "./confusion.py", line 10, in <lambda>
    return np.array(list(map(lambda x: int(x[9:]), f.read().split())))
ValueError: invalid literal for int() with base 10: 'spam'

[loretoparisi]

@cpuhrsch I have fixed the python script and now it works. See here.

Thank you!!!!