Detectron2: Memory leak during prediction with 20 images

[memicalem]

I wanted to make batching possible, so I wrote the following code, but it used 10.7 GB RAM and it chrashed. I really don't know why. For one image it uses 1.4 GB RAM. What could be done to fix this issue? I am doing prediction on CPU.

`im = cv2.imread("./input.jpg")
images = [im for _ in range(20)]
images = [{
"image": torch.from_numpy(np.transpose(image, (2, 0, 1)))
} for image in images]

cfg = get_cfg()
cfg.merge_from_file('../detectron2/configs/COCO-Detection/faster_rcnn_R_101_FPN_3x.yaml')
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5 # set threshold for this model

cfg.MODEL.WEIGHTS = "./model_final.pth" #detectron2://COCO-Detection/faster_rcnn_R_101_FPN_3x/137851257/model_final_f6e8b1.pkl"

cfg.MODEL.DEVICE = 'cpu'

model = build_model(cfg)
DetectionCheckpointer(model).load('./model_final.pth')
model.train(False)

print(model(images))`

[ppwwyyxx]

N images use N times more memory than 1 image. This is as expected.
You can predict on N images one by one in a loop instead.

with torch.no_grad() should also be used for inference to reduce memory.

[memicalem]

with torch.no_grad() I did reduce the memory consumption, but I now it uses 9.3 GB of RAM for 25 images. This is really too much. I thought only weights are important here and not the batch size.

[fjourdren]

Same issue on CPU, even with 1 image I use around 4GB RAM (my code is globally the same as memicalem but I use "mask_rcnn_R_101_DC5_3x" model). With the predictor class I use ~1.5GB.

I'll try on next days with GPU to see if there is the same behavior.

Edit: with more images, program get killed becaus it uses too mutch memory.

[memicalem]

export LRU_CACHE_CAPACITY=1 worked for me to reduce memory significantly, but I really don't know why.

[ppwwyyxx]

Google points to https://github.com/pytorch/pytorch/issues/27971 which says the issue will be fixed in pytorch 1.6

[memicalem]

Google points to pytorch/pytorch#27971 which says the issue will be fixed in pytorch 1.6

When can we expect that version 1.6 could be installed with pip or conda?

[dhananjayd232]

export LRU_CACHE_CAPACITY=1 worked for me to reduce memory significantly, but I really don't know why.

Thanks @memicalem , The same thing worked for me as well.

[memicalem]

In the console in your environment. You can also add it to your pycharm as
environment variable.

On Fri, Jul 10, 2020 at 6:04 PM Sam Videlock [e-mail]
wrote:

@dhananjayd232 https://github.com/dhananjayd232 @memicalem
https://github.com/memicalem Where are you writing the line:

export LRU_CACHE_CAPACITY

Thanks!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/facebookresearch/detectron2/issues/1539#issuecomment-656753958,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AOGND2BYAJ2ICCHBYMQCOKLR243XRANCNFSM4NSUZXQQ
.

[svideloc]

In the console in your environment. You can also add it to your pycharm as environment variable.
…
On Fri, Jul 10, 2020 at 6:04 PM Sam Videlock @.*> wrote: @dhananjayd232 https://github.com/dhananjayd232 @memicalem https://github.com/memicalem Where are you writing the line: export LRU_CACHE_CAPACITY Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1539 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOGND2BYAJ2ICCHBYMQCOKLR243XRANCNFSM4NSUZXQQ .

Thanks @memicalem running model now, and no memory issues

[halfdanrump]

yeah thanks @memicalem !