Simpletransformers: Unable to use predict() for sentence classification task

[namrathaurs]

I have been trying to use the predict method for making predictions on a data sample using pre-trained XLNet model that has been fine-tuned on a small sample of the data specific to my task. Upon execution, the IndexError is thrown saying the list index is out of range (see below).

Code Snippet:

model = ClassificationModel('bert', 'bert-base-uncased', use_cuda=True)
model.train_model(train_df)

valdn_set = valdn_df.drop("labels", axis=1).values.tolist()
pred_labels, _  = model.predict(valdn_set)

Error:

Running loss: 0.709356Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0                                                             | 0/1 [00:00<?, ?it/s]
Running loss: 0.677126Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0                                                   | 8/748 [00:06<10:31,  1.17it/s]
Running loss: 0.457629Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8192.0                                                   | 25/748 [00:21<10:11,  1.18it/s]
Running loss: 0.558106Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4096.0                                                  | 104/748 [01:27<09:05,  1.18it/s]
Current iteration: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 748/748 [10:32<00:00,  1.18it/s]
Epoch: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [10:33<00:00, 633.39s/it]
Training of bert model complete. Saved to outputs/.

Traceback (most recent call last):
  File "zero_threshold.py", line 83, in <module>
    main()
  File "zero_threshold.py", line 64, in main
    pred_labels, model_outputs = model.predict(valdn_set)
  File "/.local/lib/python3.6/site-packages/simpletransformers/classification/classification_model.py", line 821, in predict
    eval_examples = [InputExample(i, text[0], text[1], 0) for i, text in enumerate(to_predict)]
  File "/.local/lib/python3.6/site-packages/simpletransformers/classification/classification_model.py", line 821, in <listcomp>
    eval_examples = [InputExample(i, text[0], text[1], 0) for i, text in enumerate(to_predict)]
IndexError: list index out of range

As per the documentation, predict() expects a list of strings to be sent to the model, which I did convert my dataframe column to as in the above code. I looked into code lines #812 to #820 (attached below) in simpletransformers/classification/classification_model.py to understand what was causing the issue and I noticed that the code was failing due to the code behavior upon isinstance() being evaluated.

simpletransformers/classification/classification_model.py:

if multi_label:
     eval_examples = [ InputExample(i, text, None, [0 for i in range(self.num_labels)]) for i, text in enumerate(to_predict) ]
else:
     if isinstance(to_predict[0], list):
          eval_examples = [InputExample(i, text[0], text[1], 0) for i, text in enumerate(to_predict)]
     else:
          eval_examples = [InputExample(i, text, None, 0) for i, text in enumerate(to_predict)]

From what I understood from the existing code: the first element of the to_predict variable (a list) is checked to see if it is a list. If it is a list, then it implies that the data is representative of a sentence classification task where the data being fed to the model is just a single sentence. If it is not a list, then it indicates data representing a sentence pair classification task where the data being fed to the model is more than a single sentence. Am I correct in understanding the logic behind the evaluation of the condition based on isinstance()?

If the above explanation I provided is correct, then I think the statements to be executed based on the if condition need to be reversed as follows:

if isinstance(to_predict[0], list):
     ''' do when True (indicating data for sentence classification) '''
     eval_examples = [InputExample(i, text, None, 0) for i, text in enumerate(to_predict)]
else:
     ''' do when False (indicating data for sentence pair classification) '''
     eval_examples = [InputExample(i, text[0], text[1], 0) for i, text in enumerate(to_predict)]

Am I correct in thinking this through or am I missing something that's implied? Please help. Thanks in advance.

[ThilinaRajapakse]

Thank you for the detailed description! Really makes things easy.

The logic is as follows:

• Input the the predict() method should always be a list. This is so that it can be used with both single samples (a single element list in that case) as well as with multiple samples. In this context, single and multiple refers to the number of samples (think rows of a DF) rather than the type of task (single sentence vs sentence pair).
• For single sentence (type) tasks this list will be a list of strings. Each string is analogous to a single row of a dataframe.
• For sentence pair tasks, the list will be a list of lists of the shape (n, 2) where n would be the number of rows/samples. The two "columns" represent text_a and text_b of a sentence pair.

With this format, the code checks whether the first element of the input is a list. If the first element of the list is again a list, that is taken to be a sentence pair task. If it is not, it is taken to be a single sentence task.

Does this clear things up? Or did I miss something in the implementation?

[namrathaurs]

Thank you so much for the clarification. This does clear things up for me in terms of the implementation. I was assuming that within the single sentence task case, the input would also be a list of lists, where each inner list will have a single element. The examples in the readme did not suggest the case of the input having more than one sentence but I believe that having such examples would defy the purpose of predict() to be used for real time prediction on new, unseen data, one example at a time. Now that you have explained the underlying structure of input, it all makes sense and your implementation is to the point.

I had a couple of further questions which may not directly fall under the topic of the previous issue. However, these questions arose when I was debugging my code by looking into your implementation to understand what was happening behind the scene.

1. Within the simpletransformers/classification/classification_model.py script, I noticed that eval() method has been used in the definition of predict() and evaluate() methods. While the evaluate() method definition exists, I do not see any definition for eval(). Even the docstring for load_and_cache_examples() mentioned as being a utility function for train() and eval() methods. I could find train() defined in the script, but nothing for eval(). Is evaluate() the same as eval()? Or, am I missing something with respect to this?

2. Coming to the method definition for evaluate() and predict(), evaluate() has model.eval() call outside of the for loop but predict() has model.eval() call within the for loop. Could you please explain the reason behind this difference? Sorry if this is a naive question as I am still getting started with understanding PyTorch related code.

3. When I use XLNet pretrained models and fine-tune on a small sample of data pertaining to my task, the following warning message is persistently encountered.

/cm/shared/utils/PYTHON/transformers/lib/python3.6/site-packages/sklearn/metrics/classification.py:872: 
RuntimeWarning: invalid value encountered in double_scalars 
mcc = cov_ytyp / np.sqrt(cov_ytyt * cov_ypyp)

Seems to me like something is affecting the calculation of mcc using scikit-learn, which makes the result look weird:

{'mcc': 0.0, 'tp': 431, 'tn': 0, 'fp': 316, 'fn': 0, 'accuracy': 0.5769745649263721, 'f1': 0.731748726655348, 'eval_loss': 0.6876807485489135} 

Despite casting the labels to np.float64 as opposed to np.int, the message still keeps occurring. I have two questions in this regard:

• Do you have any idea/thoughts about what might be causing this to happen? Has anyone else reported this before because I could not find anything in the issue log with respect to this scenario? I took a look at the data sets and everything seems to be normal.
• While I know that MCC is the default performance metric, is it possible to disable the default metric and only use other metrics (such as accuracy or F1-measure)?

Looking forward to your responses! Thanks once again for addressing these concerns. Cheers!

[ThilinaRajapakse]

The predict() method is not necessarily meant to be used one example at a time. It's meant to be used with a list of examples, where it is ok for the list to only contain 1 example. The reason is that restricting it to 1 example at a time would slow down predictions if there was a batch of data to be processed.

The model.eval() method is a Pytorch function that switches a Pytorch model to evaluation mode. The model will perform inference faster when in this mode as opposed to the training mode (which can be activated with model.train()). It is not related to the library's evaluate() method (other than the fact that the model is put into evaluation mode during evaluation). It doesn't really matter whether it is inside the loop or not.

The warning is because the model is predicting the same class for all inputs. Does it still happen if you are using the full dataset?

Atm, it is not possible to disable MCC as you can just ignore it. You can add any other metrics if necessary.

[splevine]

I am trying to use simpletransformers for multiclass text classification with 5 classes (0,1,2,3,4) for single instances. When using predict() for single examples, I am placing the string into a list and running predict, but for some reason no matter what the input is I am getting the same result until their are 5 items in the list. To illustrate, as I add multiple instances of the same string, the correct output is not given until there are 5 items in the list. Regardless of what is in the input list the output array is the same. So any list of length 1 produces array([4]), any list of length 2 produces array([2,2]), and so on. Here are some sample outputs to get an idea of what I'm saying:

Any list of length 1 produces the same result:
model.predict(['Thanks so much!!'])
Output: (array([4]), array([[-1.7466019, -3.0352244, -0.8154188, 1.9429802, 2.499206 ]], dtype=float32))

Any list of length 2 produces the same result:
model.predict(['Thanks so much!!']*2)
Output: (array([2, 2]), array([[ 0.12082468, -0.41065404, 1.2655902 , 0.2366471 , -1.0810448 ], [ 0.12082468, -0.41065404, 1.2655902 , 0.2366471 , -1.0810448 ]], dtype=float32))

model.predict(['Thanks so much!!']*3)
Output: (array([1, 4, 3]), array([[ 2.334078 , 3.026861 , -0.02490483, -1.7674333 , -2.47653 ], [-1.2527776 , -2.376678 , -0.86035436, 1.1691897 , 2.4863505 ], [-1.1579055 , -2.387143 , 0.8597026 , 1.9112996 , 0.51665646]], dtype=float32))

model.predict(['Thanks so much!!']*4)
Output: (array([1, 4, 3, 0]), array([[ 2.334078 , 3.026861 , -0.02490483, -1.7674333 , -2.47653 ], [-1.2527776 , -2.376678 , -0.86035436, 1.1691897 , 2.4863505 ], [-1.1579055 , -2.387143 , 0.8597026 , 1.9112996 , 0.51665646], [ 0.87399036, -0.35889414, 0.8303212 , -0.42555094, -0.73294175]], dtype=float32))

model.predict(['Thanks so much!!']*5)
Output: (array([1, 1, 1, 1, 1]), array([[ 2.334078 , 3.026861 , -0.02490463, -1.767433 , -2.47653 ], [ 2.334078 , 3.026861 , -0.02490463, -1.767433 , -2.47653 ], [ 2.334078 , 3.026861 , -0.02490463, -1.767433 , -2.47653 ], [ 2.334078 , 3.026861 , -0.02490463, -1.767433 , -2.47653 ], [ 2.334078 , 3.026861 , -0.02490463, -1.767433 , -2.47653 ]], dtype=float32))

As far as I can tell the predict function works fine on batches 5 or greater, but does not produce the right output for lists of size 1 to 4.

Is anybody else experiencing this issue? Is there a different function to use for single predictions? Regardless of the input, I am getting the same output array for lists under length 4. I can clarify more and show more test cases if this is not clear.

Any help is appreciated. Thank you

[splevine]

Looked through some other issues, and maybe somebody else may run into the same problem.
When loading the model setting these arguments fixed it
model = ClassificationModel('roberta', model_location, args={'use_cached_eval_features' : False,'reprocess_input_data': True})

[ThilinaRajapakse]

Yeah, this can happen if cached features are used.

Starting to wonder if cache should be turned off by default. I think I kept it on by default before implementing multiprocessing for feature conversion as it could take hours to convert. Maybe caching is more trouble than it is worth now (at least when turned on by default).

[splevine]

Yeah it's hard to know what to set for default because it's enormous benefit to have cached features for training, but then when loading an already trained model and using it for predictions without realizing features were cached - I thought there was something wrong because I kept getting the same predictions. Maybe, some different settings for a "train" mode and "eval" mode like they do in transformers.

Thanks for the quick response and all the hard work your doing. This library is really incredible and I really appreciate it!

[ThilinaRajapakse]

The predict method is never supposed to use cached features actually. That was how I planned it but maybe not how I implemented it. 😅 I'll go through these again soon.

You are welcome and thank you for your kind words!

[namrathaurs]

@ThilinaRajapakse, sorry for the late response as I was busy working on something else over the last few weeks. Here's my response to your reply:

The predict() method is not necessarily meant to be used one example at a time. It's meant to be used with a list of examples, where it is ok for the list to only contain 1 example. The reason is that restricting it to 1 example at a time would slow down predictions if there was a batch of data to be processed.

That makes sense.

The model.eval() method is a Pytorch function that switches a Pytorch model to evaluation mode. The model will perform inference faster when in this mode as opposed to the training mode (which can be activated with model.train()). It is not related to the library's evaluate() method (other than the fact that the model is put into evaluation mode during evaluation). It doesn't really matter whether it is inside the loop or not.

Thank you for the explanation.

The warning is because the model is predicting the same class for all inputs. Does it still happen if you are using the full dataset?

I haven't given this a try yet, but I will soon.

Atm, it is not possible to disable MCC as you can just ignore it. You can add any other metrics if necessary.

Yes. I did notice in the documentation that I can add additional metrics. Was curious to know if there was a way to disable MCC without ignoring it since I was running into an issue during the MCC calculation.

@splevine, did you get all your questions answered on this thread? If you can confirm, I will go ahead and close this issue.

@ThilinaRajapakse, thank you for continuing to answer my questions and making this library accessible. Cheers!

[splevine]

Yes, all set - predict() has been working, I just had to set 'use_cached_eval_features' to False. Thanks!

[namrathaurs]

Closing this issue as all concerns have been addressed!