Sentence-transformers: VRAM use grows when using SentenceTransformer.encode + potential fix.

[martinritchie]

Hey,

I have been using this repository to obtain sentence embeddings for a data set I am currently working on. When using SentenceTransformer.encode, I noticed that my VRAM usage grows with time until a CUDA out of memory error is raised. Through my own experiments I have found the following:

1. detaching the embeddings before they are extended to all_embeddings, using: embeddings = embeddings.to("cpu") greatly reduces this growth.
2. Even with the above line added the VRAM use grew, albiet slowy, by adding the line torch.cuda.empty_cache() after the above VRAM usage appears to stop growing over time. The first point makes sense as a fix but I am unsure why this line is necessary?

I am using:
pytorch 1.6.0,
transformers 3.3.1,
sentence_transformers 0.3.7.

Have I missed something in the docs or am doing something daft? I am happy to submit a pull request if needs be?

Thanks,

Martin

[nreimers]

How do you call the encode method? I.e., which parameters do you pass? Especially how to you set convert_to_numpy or convert_to_tensor

[martinritchie]

Thank you for a quick response!

Here is an example:

   model = SentenceTransformer("distilbert-base-nli-stsb-mean-tokens") 
   y = model.encode( text_list, 
                                     batch_size=256,
                                     show_progress_bar=True,
                                     is_pretokenized=False )

So I see that I am using the default values for: convert_to_tensor=False and convert_to_numpy=True, where text_list contain millions of sentences. Shouldn't the embeddings be detached during the for loop define on line 168?

[nreimers]

A quick solution would be to break down text_list to smaller chunks (e.g. only 100k sentences) and to append the embeddings afterwards instead of passing Millions of sentences at once.

I have to think if you can detach at line 168 and what implications this will have if you e.g. want the tensors for some downstream application (e.g. as input to some other pytorch model).

[martinritchie]

I agree that in cases where the embeddings are being immediately used in downstream and GPU based tasks the current approach makes sense, i.e.,. when convert_to_tensor=False. In my use case where I needed to process a large dataset and store the sentence embeddings it was not immediately clear why I encountered cuda OOMs. So, when convert_to_numpy=True one would be safe to detach the tensors _before_ extending them to all_embeddings?

To give you some more context I was using a different repo that makes use of this one and when the original OOM was encountered I was a little perplexed. I think a warning or changing when tenors are detached, as described above, could be a nice way to prevent other users encountering the same problem?

[nreimers]

Agree, when convert_to_numpy=True, I will change the code so that detach and cpu() happens in the loop, not afterwards.

[martinritchie]

Thank you for such a prompt reply and fix, I really appreciate it.

[zeno17]

@nreimers
Do you have ETA on when this will be applied? I am currently having the issue that EmbeddingSimilarityEvaluator consistently runs into "CUDA out of memory" errors because the memory usage keeps increasing untill it hits the GPU limit (16GB) and then crashes.

The model trains succesfully with some batch size but then fails to evaluate it. When monitoring GPU usage it seems that it succesfully encodes batches, but then does not clear it from memory like normally happens during training.

As in pictures, this memory usage will keep increasing untill it runs out of memory and crashes the main training process. I think this is related to this .encode issue

[zeno17]

Full traceback that shows that it breaks in the evaluator, specifically on the self.forward(features)

[martinritchie]

If you install this repo in editable mode pip install -e . and make the corrections I added in the OP you can work around this problem until it is patched.

[nreimers]

In the latest release, I added detach() in the main loop of encode.

Does it work now better?

[zeno17]

Hi @nreimers,

Upon deeper inspection of my code i was selecting a file for the evaluator which was much much bigger than the original one i intended. The resulting problem from this was that because the evaluator (EmbeddingSimilarityEvaluator) stores all the embeddings in memory. This means that while batching limits the amount passed to the GPU (which are cleared properly), if you have a test-file with a large amount of entries youre gonna hit memory limits because it doesnt actually process the embeddings in batches (which i think would save a lot of space) for computing the correlations.

[selfcontrol7]

A quick solution would be to break down text_list to smaller chunks (e.g. only 100k sentences) and to append the embeddings afterwards instead of passing Millions of sentences at once.

I have to think if you can detach at line 168 and what implications this will have if you e.g. want the tensors for some downstream application (e.g. as input to some other pytorch model).

Hi @nreimers Thank you for this trick.

1. Please, I was wondering if it is the optimal solution for OOM error when trying to compute the cosine similarities?
   In my use case, I am trying to compute the cosine similarities of a list containing 50.716 company names and I got an OOM error.

1. Later, I came across the task Paraphrase mining and ran it without any issue.
   I might be wrong but it seems that Paraphrase mining works with "text" instead of embedding as input. Please, is there any differences in the output similarities result when using Paraphrase Mining and Semantic Textual Similarity?

Thank you

[nreimers]

Hi @selfcontrol7

Computing pairwise cosine similarity between all pairs has a quadratic memory requirement. If you run out of memory, you can try to perform it on CPU.

If you still don't have enough memory, it helps to chunk it into smaller parts. This is what the paraphrase mining code does:
Instead of computing a cosine similarity matrix 50k x 50k, it chunks it down and computes e.g. 5 matrices
50k x 10k
50k x 10k
50k x 10k
50k x 10k
50k x 10k

[selfcontrol7]

Hi @selfcontrol7

Computing pairwise cosine similarity between all pairs has a quadratic memory requirement. If you run out of memory, you can try to perform it on CPU.

If you still don't have enough memory, it helps to chunk it into smaller parts. This is what the paraphrase mining code does:
Instead of computing a cosine similarity matrix 50k x 50k, it chunks it down and computes e.g. 5 matrices
50k x 10k
50k x 10k
50k x 10k
50k x 10k
50k x 10k

Hi,

Thank you for your prompt reply and your explanation I appreciate it.

Please, can you guide me on how I can perform the computation on the CPU instead of the GPU?
Also, if I understand well, using paraphrase mining would be the best approach in my use case?
Does the accuracy in the cosine similarity computation would be different if I chunk my list into smaller parts myself and use the semantic textual similarity method instead?

Sorry if my questions looked so basic, I am a novice in sentence similarity.

Thank you again.

[selfcontrol7]

Hello

Hi @selfcontrol7
Computing pairwise cosine similarity between all pairs has a quadratic memory requirement. If you run out of memory, you can try to perform it on CPU.
If you still don't have enough memory, it helps to chunk it into smaller parts. This is what the paraphrase mining code does:
Instead of computing a cosine similarity matrix 50k x 50k, it chunks it down and computes e.g. 5 matrices
50k x 10k
50k x 10k
50k x 10k
50k x 10k
50k x 10k

Hi,

Thank you for your prompt reply and your explanation I appreciate it.

Please, can you guide me on how I can perform the computation on the CPU instead of the GPU?
Also, if I understand well, using paraphrase mining would be the best approach in my use case?
Does the accuracy in the cosine similarity computation would be different if I chunk my list into smaller parts myself and use the semantic textual similarity method instead?

Sorry if my questions looked so basic, I am a novice in sentence similarity.

Thank you again.

Hello,
I dug a bit more and could make it work on the CPU which took a lot of time to compute the similarity.
After a couple of tests with paraphrase and semantic search, I think paraphrase worked better.

However, is there any way to limit the default minimum similarity value when running Semantic Textual Similarity, Paraphrase mining, and Semantic search?
For instance, I would like the output pairs to not consider similarity below 40% when running Semantic search.

Thank you again for your time and your work.

[nreimers]

Hi,
you can sort the results in decreasing order and stop once the score is below your threshold.

[selfcontrol7]

Hi,
you can sort the results in decreasing order and stop once the score is below your threshold.

Ah, Yes simple way to make it.
Thank you.