Simpletransformers: [Advice/Seeking Help] Very Low LRAP Score

[krrishdholakia]

Hi,

I'm currently training the multi-label classifier using XL-Net, to classify questions -> to concepts (for an online-learning platform). There's ~1.7k concepts and 15k+ questions.

here's the model:
model = MultiLabelClassificationModel('xlnet', 'xlnet-base-cased', num_labels=len(lo_id_cols), args={'train_batch_size':8, 'gradient_accumulation_steps':16, 'learning_rate': 3e-5, 'num_train_epochs': 3, 'max_seq_length': 512, "fp16": False, 'overwrite_output_dir': True})

This is my final output:
{'LRAP': 0.004577569729953515, 'eval_loss': 0.27328102769863666}

I'm not sure what i'm doing wrong to have such a low LRAP. Would it be possible to have the model train to improve the LRAP ? any help/advice would be appreciated.

[AbinayaM02]

Hi @krrishdholakia. Could you provide some sample data and the sample predictions?

[krrishdholakia]

@AbinayaM02 the input to the model is something like this:

shortened for brevity but essentially: the text is the question, and the label is the corresponding id

A substance was kept in a far bigger container... | [0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
-- | --

[AbinayaM02]

Is your data skewed? Then you can try using class weights parameter and re-run the experiment to see if your score improves.

[krrishdholakia]

how would i derive a class weight for this ?

data has variance where some questions might have 100 q's tagged to them and some only 3-4.

[AbinayaM02]

Calculate the number of datapoints tagged as 1 for each class individually and calculate the proportion for each class. For example, if there are 4 classes with [100, 500, 1000, 10] datapoints each, then class weights will be [10, 2, 1, 100]. In your case, if a single question is tagged with n classes, it will get accounted in each of the n classes individaully.

[ThilinaRajapakse]

Keep in mind that Transformer models are not good at extreme multiclass/multilabel classification. Even with weights, 1700 classes might just be too many. You can try to model the problem as a hierarchical classification to circumvent this problem. For example, starting with 10 broad categories which further divide into subcategories. This too might not be straightforward in multilabel classification.

There is a new paper by Amazon that seems to tackle the problem but I haven't looked at it properly yet.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.