Sentence-transformers: Fine Tuning Models

[farhaanbukhsh]

We want to fine-tune 'bert-large-nli-stsb-mean-tokens' on multi-label classification task. So that we can use the output model to get embeddings out.
We have a bunch of sentences classified into labels. The prime question that I want clarification on:

1. What are the steps to fine tune the models provided by sentence transformer ?
2. Is there a script or a repo which does this and I can cross reference here ?

The closest we have come to is simpletransformers

But is there a better way to go ahead? Thanks in advance for the help

[farhaanbukhsh]

@nreimers can you please help us out here? Any help will be highly appreciated, Thanks a lot :smile:

[nreimers]

Hi @farhaanbukhsh
I am not sure what you want / what type of data you have?

Do you want multi-label classification for single sentences? Then I can recommend to use transformers package and to fine-tune the model. Sentence-Transformers is not needed for that.

You want to fine-tune sentence embeddings with a training set where you only have labels and single sentences, and then use the sentence embeddings for example with cosine similarity for tasks like clustering? This is a non-trivial case and sadly there is no single "do this" answer. It heavily depends on what your dataset looks like and how you want to use the sentence embeddings afterwards. You can have a look here, how a dataset with (label, single_sentence) can be used with BatchHardTripletLoss: https://github.com/UKPLab/sentence-transformers/blob/master/examples/training_transformers/training_batch_hard_trec_continue_training.py

You have sentence pairs and labels? Then you can maybe use the same setup as for NLI:
https://github.com/UKPLab/sentence-transformers/blob/master/examples/training_transformers/training_nli.py

Best
Nils Reimers

[farhaanbukhsh]

Hi @farhaanbukhsh
I am not sure what you want / what type of data you have?

Do you want multi-label classification for single sentences? Then I can recommend to use transformers package and to fine-tune the model. Sentence-Transformers is not needed for that.

You want to fine-tune sentence embeddings with a training set where you only have labels and single sentences, and then use the sentence embeddings for example with cosine similarity for tasks like clustering?

yes, exactly what we are trying to do.

This is a non-trivial case and sadly there is no single "do this" answer. It heavily depends on what your dataset looks like and how you want to use the sentence embeddings afterwards. You can have a look here, how a dataset with (label, single_sentence) can be used with BatchHardTripletLoss: https://github.com/UKPLab/sentence-transformers/blob/master/examples/training_transformers/training_batch_hard_trec_continue_training.py

You have sentence pairs and labels? Then you can maybe use the same setup as for NLI:
https://github.com/UKPLab/sentence-transformers/blob/master/examples/training_transformers/training_nli.py

Best
Nils Reimers

Thanks a ton @nreimers for replying, means a lot :heart:

What I have is a simple csv file where for a given sentence I have assigned labels, for example:

The heater from amazon was damaged. Labels: ['defective product', 'electric appliance'] so for this do you think https://github.com/UKPLab/sentence-transformers/blob/master/examples/training_transformers/training_batch_hard_trec_continue_training.py

can be used?

[nreimers]

Yes, for this BatchHardTripletLoss seems a good choice.

For triplet loss, have a look at this good blog article:
https://omoindrot.github.io/triplet-loss

With BatchHardTripletLoss, sentences (texts) that have the same labels will become close in vector space, while sentences with a different label will be further away. At the end you will have clusters in your vector space, e.g. all sentences talking about 'defective product' will be in one space, while sentences with label 'great price' while be somewhere else in vector space.

Best
Nils Reimers

[farhaanbukhsh]

Yes, for this BatchHardTripletLoss seems a good choice.

For triplet loss, have a look at this good blog article:
https://omoindrot.github.io/triplet-loss

With BatchHardTripletLoss, sentences (texts) that have the same labels will become close in vector space, while sentences with a different label will be further away. At the end you will have clusters in your vector space, e.g. all sentences talking about 'defective product' will be in one space, while sentences with label 'great price' while be somewhere else in vector space.

Best
Nils Reimers

Thanks a loads @nreimers :100:

[farhaanbukhsh]

Hey @nreimers , we faced a problem the script:

https://github.com/UKPLab/sentence-transformers/blob/master/examples/training_transformers/training_batch_hard_trec_continue_training.py

Doesn't work out of the box, the data source: https://cogcomp.seas.upenn.edu/Data/QA/QC/TREC_10.label

has just one sentence while for

evaluator = TripletEvaluator.from_input_examples(val, name='dev')

to work it needs 3 sentence I guess there is some kind of a confusion to use TripletReader and InputExample being used .

if you could point me the right data source to be used for TripletLoss it would be lot of help. Thanks a lot in advance

[nreimers]

Hi @farhaanbukhsh
I fixed that example, it should be working now.

The triplets for dev & test have to be created before they are bassed to TripletEvaluator

Best
Nils

[farhaanbukhsh]

Hi @farhaanbukhsh
I fixed that example, it should be working now.

The triplets for dev & test have to be created before they are bassed to TripletEvaluator

Best
Nils

Thanks a tone for all the work here :heart:

[farhaanbukhsh]

@nreimers do I need a GPU to run BatchSemiHardTripletLoss.py or I can run without it ?

Getting this error:

AssertionError: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx

coming from /sentence_transformers/losses/BatchSemiHardTripletLoss.py", line 60, in batch_semi_hard_triplet_loss

[nreimers]

Fixed that with the latest push, but you would need to install the framework from sources.

Or you use one of the two other batch hard losses. They work without GPU in the latest release

[farhaanbukhsh]

Got it :) thanks again

[farhaanbukhsh]

Hey @nreimers ,

I just have few follow up questions, I went through the TripletLoss blog that you pointed out, https://omoindrot.github.io/triplet-loss. This did help a lot, so the questions are:

1. In the script https://github.com/UKPLab/sentence-transformers/blob/master/examples/training_transformers/training_batch_hard_trec_continue_training.py, we have randomized forming a Triplet, is this done for demonstration purpose or we can use this technique off the shelf?

2. In case we can't then we need to form our own set of triplets right ? Anyhow is this https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/datasets/wikipedia-sections-triplets.zip dataset related to triplet loss and can it be used?

I also went through https://towardsdatascience.com/image-similarity-using-triplet-loss-3744c0f67973 and here it is mentioned we have to curate the Triplet dataset and supply it to the tunning algorithm. So is human curation required here?

[nreimers]

Hi @farhaanbukhsh
The random triplets are formed only for the dev & test set. For the train set, this is not used.
Random triplet for dev & test set are rather simple to distinguish. Depending on your application, you might need better triplets to fully evaluate the performance of your model.

For training, the application uses what is called "Batch All / Hard / SemiHard Triplets" (also explained in the link). You generate a mini batch with n sentences. It then checks out of the n x n x n possible combinations, which are valid triplets such that anchor and positive have the same label and the negative a different label. It then uses these valid triplets to compute the loss.

The quality (difficulty) of the triplets are quite important to get good results. If the triplets are too easy, the model will not learning something. With the Batch*Triplets approach you already create large combinations of triplets, hence, there designing difficult triplets is not really needed.

Human curation for training is not needed.

Best
Nils Reimers

[ironllamagirl]

Hi!
Just wanted to inquire about this file https://github.com/UKPLab/sentence-transformers/blob/master/examples/training_transformers/training_batch_hard_trec_continue_training.py

Has it been placed somewhere else? can't seem to open the link

[nreimers]

https://github.com/UKPLab/sentence-transformers/blob/master/examples/training/other/training_batch_hard_trec_continue_training.py

[thefirebanks]

Hi @nreimers ! IThank you for the advice written on this thread. It appears that the file https://github.com/UKPLab/sentence-transformers/blob/master/examples/training/other/training_batch_hard_trec_continue_training.py is giving a 404 error.

@farhaanbukhsh We are working on a similar task as you - e.x given a sentence "The tomatoes were rotten" and a set of possible labels ["fruit", "vegetable", "root"] I want to assign one label to the sentence. Did you end up using the triple loss methodology successfully? I was also looking at this thread.

I have 2 questions:

• When we fined tuned a model using the code from https://github.com/UKPLab/sentence-transformers/blob/master/examples/training/nli/training_nli.py, but we found that the resulting model only works in yielding embeddings and not necessarily classifiying text. Is there a way of including the linear layer used for classification in fine tuning, in the saved model? that way when we load the model we can do something like model.predict() instead of just model.encode()?
• If not, should we load the model embeddings/fine tune using huggingface?

Thank you in advance!

[nreimers]

@thefirebanks
It was renamed: https://github.com/UKPLab/sentence-transformers/blob/master/examples/training/other/training_batch_hard_trec.py

The purpose of SentenceTransformer is to create a model that can generate meaningful embeddings for text. If your task is classification, then using sentence embeddings is in most cases the wrong approach. In that case, CrossEncoder work much better: https://www.sbert.net/examples/training/cross-encoder/README.html

Currently there is no implemented way to include that layer in the model. But you can save it with torch.save() and load it with torch.load() and apply it on top of the sentence embeddings.

But as mentioned, if your task is classification, then using CrossEncoders achieve much better performances.