Sentence-transformers: Noticing significant slow-down with model.encode

[earny-joe]

Hello! First of all, I want to say that this library is fantastic and thanks for putting it together!

However, I noticed while working on a project where I'm currently trying to generate embeddings for ~500k tweets; there seems to be a significant slowdown. I have the progress bar on, and I'm noticing that as more embeddings are generated, the lower the number per iteration becomes.

I did two quick tests with a small subsets of the data (the first 1k & 25k observations) and ran it locally with CPU. The same slowdown pattern occurred in both cases. In other words, the slowdown is proportional to the data size. At around 50% (no matter the size of the data), the iterations per second are half of what it was when the encoding began.

Also, an important note - I first noticed this when encoding on the large data set (~500k tweets) using a GPU. The behavior was, in a sense, replicated on a CPU, albeit with a smaller data set (i.e., the 1k and 25 datasets).

Any idea of why this slowdown might be happening? My initial guess is that the embeddings are using up space in the CPU/GPU, causing it to run slower. However, the pattern staying the same no matter the input data size seems a little odd.

Additionally, I have attached an image of one run-through of the function that shows the decreasing iterations per second.

[nreimers]

Hi,
In order to waste minimal overhead for computation of padding tokens, the passed sentences are sorted by length. Then, sentences of about the same length are encoded for which we only need a minimal number of padding tokens to get them all to the same length.

As a consequence, the shortest sentences are encoded first. With each new batch, the sentences get longer and the computation of the Embedding takes more time. BERT has a quadratic runtime with the input length.

You could try this by passing a long list of always the same sentence. Then you shouldn't see a runtime drop, as all sentences are of the same length.

Best
Nils Reimers

[earny-joe]

Hi Nils, thanks for the quick response. That makes sense. As a quick follow-up, the outputted list of arrays (representing the embeddings), is the order returned to the original input (i.e., sentence are then unsorted back into the original input sequence), or is it still in that sorted order?

Thanks!

[nreimers]

It is returned in the original order.