Simpletransformers: Production - Memory Leak?

[hassant4]

Is anyone serving a Classifier in production? I'm using flask and upon every call to predict I am experiencing an increase in memory usage by the process until OOM error in the pod on Kubernetes, I only invoke this class once and use the same instance. This even happens for batch size 1

class Transformer:
    def __init__(
        self,
        dir = 'model/',
    ):
        self.model = MultiLabelClassificationModel(
            'roberta', dir, use_cuda=False, args={'no_cache': True, 'use_cached_eval_features':False},
        )
        self.mlb=MultiLabelBinarizer()
        self.mlb.fit([['A', 'B', 'C', 'D']])

    def predict(self, batch,):
        predictions, raw_outputs = self.model.predict(batch)
        labels = self.mlb.inverse_transform(np.asarray(predictions))
        return labels

[ThilinaRajapakse]

I have used classifiers with a flask API but I didn't run into this issue. There are also classifiers running in production on EC2 servers without issues. Could this be a Kubernetes specific problem?

[hassant4]

How much memory do you allocate? I think it may be converging on 5GB for me after 400 predicts calls, previously only a max of 2GB was allocated, now it is a max of 16GB. I will update if it looks like it has / has not converged when the API is called further

[ThilinaRajapakse]

Unfortunately, I can't remember how much memory I used with the flask apps. I believe it was around 5 GB. The EC2 servers have GPUs with 16 GB memory I think.

[Shane-Neeley]

classifiers running in production on EC2 servers

Do you have a deployment example for these? I'm trying to deploy simpletransformers with uvicorn, gunicorn, fastapi and docker but predictions are not working. I may switch to Flask.

[ximo1984]

classifiers running in production on EC2 servers

Do you have a deployment example for these? I'm trying to deploy simpletransformers with uvicorn, gunicorn, fastapi and docker but predictions are not working. I may switch to Flask.

I tried with fastapi, gunicorn and uvicorn but It doesn't work me neither. I tried with flask server and it works well but when I try to run my model with gunicorn it fails when it calls predict method. So I think the problem is with gunicorn and predict method of simpletransformers.

Anyone has resolved it?

[ThilinaRajapakse]

Let me see if I can dig up the code.

Bit of a disclaimer though: I am _not_ an expert in server related stuff. :sweat_smile:

[ximo1984]

I have more info. The problem is with multiprocessing.
When I run flask it works perfectly but when I run with gunicorn I have to deactivate 'use_multiprocessing' to work well.

Change this (this works with flask server but not with gunicorn):
model = ClassificationModel('bert', 'outputs', num_labels=11, use_cuda=False, args={'silent': True})

To this (this works for both):
model = ClassificationModel('bert', 'outputs', num_labels=11, use_cuda=False, args={'silent': True, 'use_multiprocessing': False})

[hassant4]

I am running flask without Gunicorn, and it runs fine for me, I know it gives the warning that it isn't production grade, but I'm sure I've processed more that 10k+ requests at this stage while live for at least a month.

And just an update on my memory issues from OP: for batch sizes 1, my instances converged to around 2.9GB. And for batch sizes 300, it did also converge to around 5-6GB so it was ok I guess. My problems were that my initial 2GB instances were too weak.