Simpletransformers: Explanation for prediction

Created on 7 Oct 2019 · 15Comments · Source: ThilinaRajapakse/simpletransformers

I am using this library for sentiment analysis. I have trained the model and now I want to predict the testing dataset. When I am using model.predict(['sentence comes here']) of the prediction of an unknown new sentence I am getting an output as follows:

(array([0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1, 0, 1, 1, 0, 1, 0, 0,
0, 1, 0, 1, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 1, 0,
0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 0,
1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 0,
0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0,
1, 0, 0, 1, 0]), array([[ 0.05918646, -0.2312517 ],
[-0.15851352, -0.0500554 ],
[ 0.13287637, -0.25292102],
[ 0.13287637, -0.25292102],
[-0.11805943, -0.09430733],
[-0.19722672, 0.13679701],
[-0.27268714, 0.13469158],
[ 0.71184117, -0.83837014],
[ 0.04452557, -0.21931541],
[ 0.1598964 , -0.33018294],
[-0.21872135, 0.11754164],
[ 0.46354538, -0.52351034],
[ 0.00256112, -0.19036555],
[-0.27268714, 0.13469158],
[-0.27268714, 0.13469158],
[ 0.00256112, -0.19036555],
[-0.19722672, 0.13679701],
[-0.17320369, 0.00332226],
[ 0.1598964 , -0.33018294],
[-0.10593811, -0.06245855],
[ 0.04452557, -0.21931541],
[ 0.1598964 , -0.33018294],
[ 0.31071872, -0.46716788],
[-0.15851352, -0.0500554 ],
[ 0.05918646, -0.2312517 ],
[-0.42780194, 0.4040406 ],
[ 0.00256112, -0.19036555],
[-0.19722672, 0.13679701],
[-0.17320369, 0.00332226],
[ 0.1598964 , -0.33018294],
[ 0.74134415, -0.89872265],
[-0.37553215, 0.2911712 ],
[-0.19722672, 0.13679701],
[-0.27268714, 0.13469158],
[-0.21872135, 0.11754164],
[-0.62663805, 0.67718726],
[ 0.1598964 , -0.33018294],
[ 1.263364 , -1.7161529 ],
[-0.37553215, 0.2911712 ],
[ 0.74134415, -0.89872265],
[ 0.74134415, -0.89872265],
[ 0.71184117, -0.83837014],
[-0.11805943, -0.09430733],
[ 0.1598964 , -0.33018294],
[ 1.263364 , -1.7161529 ],
[ 0.00256112, -0.19036555],
[-0.04172399, -0.13767837],
[-0.37553215, 0.2911712 ],
[ 0.05918646, -0.2312517 ],
[ 0.1598964 , -0.33018294],
[ 1.263364 , -1.7161529 ],
[ 0.05918646, -0.2312517 ],
[-0.37553215, 0.2911712 ],
[-0.21872135, 0.11754164],
[ 0.46354538, -0.52351034],
[-0.10593811, -0.06245855],
[-0.24102493, 0.12610589],
[-0.19722672, 0.13679701],
[-0.11805943, -0.09430733],
[ 0.04452557, -0.21931541],
[ 0.1598964 , -0.33018294],
[ 1.263364 , -1.7161529 ],
[ 0.13287637, -0.25292102],
[-0.37553215, 0.2911712 ],
[ 0.49057966, -0.6595632 ],
[ 0.49057966, -0.6595632 ],
[-0.11805943, -0.09430733],
[-0.21872135, 0.11754164],
[-0.27268714, 0.13469158],
[ 0.1598964 , -0.33018294],
[ 1.263364 , -1.7161529 ],
[-0.21872135, 0.11754164],
[ 0.25952443, -0.29697695],
[-0.37553215, 0.2911712 ],
[-0.17320369, 0.00332226],
[-0.21872135, 0.11754164],
[ 0.25952443, -0.29697695],
[-0.37553215, 0.2911712 ],
[-0.17320369, 0.00332226],
[-0.21872135, 0.11754164],
[ 0.25952443, -0.29697695],
[-0.37553215, 0.2911712 ],
[-0.17320369, 0.00332226],
[ 0.1598964 , -0.33018294],
[ 0.46354538, -0.52351034],
[-0.27268714, 0.13469158],
[-0.27268714, 0.13469158],
[ 0.74134415, -0.89872265],
[ 0.49325183, -0.62816525],
[ 1.1978749 , -1.6458939 ],
[ 1.1978749 , -1.6458939 ],
[ 0.00256112, -0.19036555],
[-0.19722672, 0.13679701],
[-0.21872135, 0.11754164],
[-0.27268714, 0.13469158],
[-0.37553215, 0.2911712 ],
[-0.17320369, 0.00332226],
[ 0.13287637, -0.25292102],
[-0.62663805, 0.67718726],
[-0.37553215, 0.2911712 ],
[-0.10593811, -0.06245855],
[ 1.1978749 , -1.6458939 ],
[ 0.74134415, -0.89872265],
[ 1.1978749 , -1.6458939 ],
[-0.5022752 , 0.5205793 ],
[-0.5022752 , 0.5205793 ],
[-0.5210591 , 0.31223089],
[-0.25475317, 0.12043235],
[-0.38624087, 0.32818356],
[ 0.23968135, -0.35400856],
[-0.5210591 , 0.31223089],
[ 0.19725166, -0.36717686],
[ 0.16669363, -0.35203722],
[-0.24102493, 0.12610589],
[ 1.1978749 , -1.6458939 ]], dtype=float32))

Can you please help me understand this output as I have trained on 0 or 1 sentiment. I was expecting some integer as an output. Please can you help me understand this output?

Thank you

Source

samarthsarin

Most helpful comment

You can apply a softmax.

from scipy.special import softmax

probabilities = softmax(raw_outputs, axis=1)

46

ThilinaRajapakse on 19 Dec 2019

❤2

All 15 comments

It's mentioned in the docs. The first array is the predicted labels. The second array is the raw outputs from the model. If you got this output for a single example, make sure that you are passing in a list and not a single string.

ThilinaRajapakse on 7 Oct 2019

Yes, I am passing it as list only
predictions = model.predict(["I'd like to puts some CD-ROMS on my iPad, is that possible?' — Yes, but wouldn't that block the screen?"])

predictions

I am still finding it difficult to understand what is the sentiment of this sentence

samarthsarin on 7 Oct 2019

Shouldn't it be returning only a single number that depicts the class name?

samarthsarin on 7 Oct 2019

(array([1]), array([[-0.19189453, -0.1743164 ]], dtype=float32))

This is the output I get when I pass the same sentence you provided to the predict method.

ThilinaRajapakse on 7 Oct 2019

Can I please get your email id or LinkedIn id so that I can share my code and if you code help me out in it please?

samarthsarin on 7 Oct 2019

The predict function returns both the class labels and the logits from the model. The reason for this is in case you need to compute the probabilities for each class rather than just a class label.

predictions, raw_outputs = model.predict(["I'd like to puts some CD-ROMS on my iPad, is that possible?' — Yes, but wouldn't that block the screen?"]

This would put the class label in predictions, and the logits in the raw outputs. If you don't need the logits, you can do;

predictions, _ = model.predict(["I'd like to puts some CD-ROMS on my iPad, is that possible?' — Yes, but wouldn't that block the screen?"]

Note that predictions is still a list. So if you are doing it for a single value, you can get it with predictions[0].

ThilinaRajapakse on 7 Oct 2019

It worked. Thanks

samarthsarin on 7 Oct 2019

Is there way to convert the raw outputs to predictions between 0 and 1? like:

[3.04134, -2.42028] will become [0.9237, 0.0763]

kinoute on 19 Dec 2019

You can apply a softmax.

from scipy.special import softmax

probabilities = softmax(raw_outputs, axis=1)

46

ThilinaRajapakse on 19 Dec 2019

❤2

@ThilinaRajapakse You're faster than Clint Eastwood.. Thank you again!

kinoute on 19 Dec 2019

Haha, thanks! (I spend too much time on my computer)

ThilinaRajapakse on 19 Dec 2019

is there any plan to allow question_answering_model to return preds with probs?

`def predict(self, to_predict, n_best_size=None):
"""
Performs predictions on a list of python dicts containing contexts and qas.

    Args:
        to_predict: A python list of python dicts containing contexts and questions to be sent to the model for prediction.
                    E.g: predict([
                        {
                            'context': "Some context as a demo",
                            'qas': [
                                {'id': '0', 'question': 'What is the context here?'},
                                {'id': '1', 'question': 'What is this for?'}
                            ]
                        }
                    ])
        n_best_size (Optional): Number of predictions to return. args['n_best_size'] will be used if not specified.

    Returns:
        preds: A python list containg the predicted answer, and id for each question in to_predict.`

is so, i would like to submit a MR. thanks.

alexucb on 15 May 2020

Yes, I think it's better to return those so that QA is consistent with the other tasks. We should probably return the probabilities both with the predict() and the eval_model() methods.

For eval_model(), we can simply return all_nbest_json which is created here.

For the predict() method, we can use the same functions to get all_nbest_json and return it.

What do you think? I'd be happy to take a look if you open a PR for this. This is something I've been procrastinating on, so it would be really helpful!

ThilinaRajapakse on 15 May 2020

How do you apply softmax if raw_outputs array has nested array for each data point?

[array([[-0.00824738,  0.63574219],
       [ 0.10467529,  0.31518555],
       [-0.39599609,  0.98681641],
       [-0.08673096,  0.70507812],
       [-0.359375  ,  0.78320312]]), 

     array([[0.01482391, 0.58496094],
       [0.04428101, 0.69140625],
       [0.42407227, 0.23718262]]), 
 ........]