Sentence-transformers: Loading the saved model for testing, resets some of the layers.

[satya77]

After training the model and saving it, when the model is loaded again for additional evaluation the accuracy is significantly lower than the one that is reported right after training the model. It seems that the model does not load some layers, possibly the softmax layer, or am I loading it wrong?

save_path = "path to the saved model "
reader= "our reader"
model = SentenceTransformer(model_name)

train_loss = losses.SoftmaxLoss(model=model, sentence_embedding_dimension=model.get_sentence_embedding_dimension(), num_labels=train_num_labels)

test_data = SentencesDataset(examples=reader.get_examples('test.tsv'), model=model, shorten=True)
test_dataloader = DataLoader(test_data, shuffle=False, batch_size=batch_size)
evaluator = LabelAccuracyEvaluator(test_dataloader, softmax_model=train_loss)

model.evaluate(evaluator)

[nreimers]

Hi @satya77
That is correct. The layer in Softmax is not stored, as it is only used for training.

If you would be interested in that layer, you would need to store it separately and load it separately.

However, the target of the framework is more for unsupervised tasks where you use cosine similarity or similar. For classification tasks with Softmax, you would not need this framework and could use BERT directly.

Best
Nils Reimers

[satya77]

thank you for the quick response!

[wadhwasahil]

How do I save the Softmax layer and load it while evaluation?

[satya77]

How do I save the Softmax layer and load it while evaluation?

add this to the end of the fit function:
example :
torch.save(SOFTMAX_LAYER,os.path.join(model_save_path,"2_Softmax/pytorch_model.bin"))

and load before the evaluator
train_loss.classifier=torch.load(os.path.join(model_save_path,"2_Softmax/pytorch_model.bin"))

[wadhwasahil]

So I am running this and getting the error

logging.info("Evaluating...")
model = SentenceTransformer(model_save_path)
test_data = SentencesDataset(examples=cr_reader.get_examples(mode="dev"), model=model)
test_dataloader = DataLoader(test_data, shuffle=False, batch_size=batch_size)
train_loss = losses.SoftmaxLoss(model=model,
                                sentence_embedding_dimension=model.get_sentence_embedding_dimension(),
                                num_labels=n_labels)
train_loss.classifier = torch.load(os.path.join(model_save_path,"2_Softmax/pytorch_model.bin"))
evaluator = LabelAccuracyEvaluator(test_dataloader, softmax_model=train_loss)
_, df = model.evaluate(evaluator)
df.to_csv(os.path.join(model_save_path, "results_new.csv"), index=False)

and error is

File "/home/sahilw2/sentence-transformers/sentence_transformers/evaluation/LabelAccuracyEvaluator.py", line 73, in __call__
    reps, prediction = self.softmax_model(features, labels=None)
  File "/home/sahilw2/anaconda3/envs/simpletransformers/lib/python3.8/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/sahilw2/sentence-transformers/sentence_transformers/losses/SoftmaxLoss.py", line 50, in forward
    output = self.classifier(features)
  File "/home/sahilw2/anaconda3/envs/simpletransformers/lib/python3.8/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
TypeError: forward() missing 1 required positional argument: 'labels'

[wadhwasahil]

How do I save the Softmax layer and load it while evaluation?

add this to the end of the fit function:
example :
torch.save(SOFTMAX_LAYER,os.path.join(model_save_path,"2_Softmax/pytorch_model.bin"))

and load before the evaluator
train_loss.classifier=torch.load(os.path.join(model_save_path,"2_Softmax/pytorch_model.bin"))

Actually

train_loss.classifier=torch.load(os.path.join(model_save_path,"2_Softmax/pytorch_model.bin")) doesn't work,
rather this works

train_loss=torch.load(os.path.join(model_save_path,"2_Softmax/pytorch_model.bin"))

[YJiangcm]

Does SOFTMAX_LAYER refer to train_loss in "train_loss = losses.SoftmaxLoss(model=model, sentence_embedding_dimension=model.get_sentence_embedding_dimension(), num_labels=train_num_labels)" ?

[nreimers]

Yes