Torch7: Memory leak when creating Tensors inside a for loop ?

[fmassa]

There might be a memory leak in the following (toy) example:

for i=1,10000 do
  local a = torch.Tensor(10000):zero()
end

This allocates around 600MB of memory, and I can't free it afterwards, even after calling collectgarbage() several times. If I run this loop several times, more memory gets used and no way to get it back, unless closing luajit. If I add a collectgarbage() inside the for loop, everything works fine, but the loop takes 20x more time to complete.

I understand that torch Tensors are allocated on C side, and that memory is freed when there is no more reference to the underlying data. I thought it could be a luajit problem, but using jit.off() didn't change anything.

Any thoughts about that ? Or is the only solution to avoid Tensor creation at all price, reusing already created Tensors in situations like that ?

[soumith]

I'll hunt it down.

On Saturday, May 9, 2015, Francisco Massa [email protected] wrote:

There might be a memory leak in the following (toy) example:

for i=1,10000 do
local a = torch.Tensor(10000):zero()end

This allocates around 600MB of memory, and I can't free it afterwards,
even after calling collectgarbage() several times. If I run this loop
several times, more memory gets used and no way to get it back, unless
closing luajit. If I add a collectgarbage() inside the for loop,
everything works fine, but the loop takes 20x more time to complete.

I understand that torch Tensors are allocated on C side, and that memory
is freed when there is no more reference to the underlying data. I thought
it could be a luajit problem, but using jit.off() didn't change anything.

Any thoughts about that ? Or is the only solution to avoid Tensor creation
at all price, reusing already created Tensors in situations like that ?

Reply to this email directly or view it on GitHub
https://github.com/torch/torch7/issues/229.

[soumith]

i'm trying to reproduce your use-case, but I dont see the same behavior.

In the interpreter, I first run:

for i=1,10000 do
  local a = torch.Tensor(10000):zero()
end

Mem usage: 56.9MB

Then I ran

collectgarbage(); collectgarbage();

Mem usage: 33.4MB

I ran the same snippet again, 5 times:
Mem usage: 709.7MB

Then again called collectgarbage,
Mem usage: 27.6MB

Seems like after the initial overhead of 25-30MB or so, it always is consistently freeing memory after collectgarbage calls.

[szagoruyko]

for me os x is fine and always returns to the same mem usage after collectgarbage no matter how many times I run the snippet. Linux however fills up quickly eating gigabytes.

[soumith]

ok I have a repro on Linux. On OSX, it seems okay. Looking into it.

[szagoruyko]

a few more things: if you remove :zero() from the snippet then there is no issue; CudaTensor with zero() call is fine too.
seems that the problem is in TH_TENSOR_APPLY.

[soumith]

Thats interesting. I wonder if its atomic refcounting thats flaky there.
Next to try is OpenMP off.

On Saturday, May 16, 2015, Sergey Zagoruyko [email protected]
wrote:

a few more things: if you remove :zero() from the snippet then there is no
issue; CudaTensor with zero() call is fine too.
seems that the problem is in TH_TENSOR_APPLY.

Reply to this email directly or view it on GitHub
https://github.com/torch/torch7/issues/229#issuecomment-102717569.

[soumith]

If the tensor is contiguous, i think fill never goes into TH_TENSOR_APPLY

On Saturday, May 16, 2015, soumith [email protected] wrote:

Thats interesting. I wonder if its atomic refcounting thats flaky there.
Next to try is OpenMP off.

On Saturday, May 16, 2015, Sergey Zagoruyko <[email protected]

a few more things: if you remove :zero() from the snippet then there is
no issue; CudaTensor with zero() call is fine too.
seems that the problem is in TH_TENSOR_APPLY.

Reply to this email directly or view it on GitHub
https://github.com/torch/torch7/issues/229#issuecomment-102717569.

[szagoruyko]

apparently it goes, I just commented it out from zero and problem gone. disabling openmp and going to older torch (2 months back) didn't help

[fmassa]

@szagoruyko I tested on 3 different machines on ubuntu, and even without the :zero() it also seems to leak memory, but only for "small" Tensor sizes. For instance,

for i=1,10000 do
  local a = torch.Tensor(1000,1000)
end

leaks memory, but

for i=1,10000 do
  local a = torch.Tensor(10000,10000)
end

doesn't...

[soumith]

I found a commit that is good. So I am running a git bisect. A few more minutes and I will find the bad commit.

[soumith]

I didn't find a commit that's good (false-positive),
but, this is very weird.

I verified that all the malloc/realloc pointers are getting freed, by tracking them completely across the program.
However, even after each pointer finishes it's free, the htop usage shows memory usage that looks like they aren't really being freed.
Wonder what I am missing about crazy linux internals.

[szagoruyko]

interesting that only os x with native clang is prone to this. switch to gcc-4.9 - same issue as in linux, and clang 3.6 in ubuntu same issue too.

[soumith]

I found the answer, one of my facebook colleagues enlightened me.
Let me put it verbatim:
"malloc'ed memory is not always released back to the OS, especially for small object sizes. (Larger allocations will use mmap() / munmap() directly.) The memory can be reused in the same process, though." - Tudor

"There are better memory allocators out there, such as our own Jason Evans's jemalloc", "(jemalloc does make a good faith effort to release memory back to the OS if possible; shrinking the data segment is hard because you can't defragment without a GC, so jemalloc uses mmap() rather than sbrk())".

Run this example to understand what that means:

local function foo()
   for i=1,1000000 do
      local a = torch.Tensor(1000,1000)
   end
end

local function foo2()
    foo()
    collectgarbage(); collectgarbage()
    os.execute('ps ax -o rss,user,command | grep luajit | sort -nr')
end

for i=1,10 do
   foo2()
end

luajit -i test.lua

The memory stagnates after a while.

Also run the same with
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so luajit -i test.lua

The memory gets freed nicely.

tl;dr: default linux memory allocator does weird things that I didn't know about till today.

[cdluminate]

So is there a way to fix the memory now? When I'm runing my lua script the memory usage just get linearly increasing, and collectgarbage('collect') doesn't help.

[soumith]

If you give a minimal reproducible use case, i can take a look, but we have heap tracking to call the garbage collector occasionally to make sure this doesn't happen

[cdluminate]

@soumith I can't repro this issue with a small script, but now it seems to be solved by saving cloned model.

originally I was directly saving the model, e.g.

local model = Model()
for i = 1, math.huge do
   forward, backward, update
   torch.save(path, model)
end

and the memory usage keeps increasing even if collectgarbage('collect') ('collect', not 'count') is frequently called, and the script will end up with luajit: not enough memory after several saves. With reference to Andrej's neuraltalk2 code, I now save the cloned model like this

local model = Model()
local thin_model = model:clone()
thin_model.component = model.component:clone()
for i = 1, math.huge do
   forward, backward, update
   torch.save(path, thin_model)
end

and the memory problem seems to be gone. But why?

[staturecrane]

I am also having this memory leak problem, and would be curious if there has been any official resolution.

[soumith]

I'm still not able to reproduce the problem :)
If any of you gives me a working test case, happy to look.

[helson73]

@soumith Similar problem here.
CudaTensors have no such problems.
But tensors with CPU have such problems.

[aschoenauer-sebag]

Hi @soumith,
I am also having this memory leak problem, but whereas it is solved by using libjemalloc as you suggested above on the CPU, it is not on the GPU. Furthermore, both small and big vectors are not freed from the GPU memory. Here is the code with which you can hopefully reproduce the issue (adapted from yours above):

require 'torch'
require 'cutorch'
os.execute('ps ax -o rss,user,command | grep luajit | sort -nr')
local function foo()
   for i=1,1000000 do
      local a = torch.CudaTensor(1000,1000)
   end
end
local function foo2()
    foo()
    print('before')
    os.execute('ps ax -o rss,user,command | grep luajit | sort -nr')
    collectgarbage(); collectgarbage()
    print('after')
    os.execute('ps ax -o rss,user,command | grep luajit | sort -nr')
end
for i=1,10 do
   foo2()
end

I am using Ubuntu 16.04.2 and Nvidia drivers 375.26 on a Titan X (Pascal) GPU.

As a side question, what is the difference between the way "qlua -lenv" and "luajit" deal with the memory? Using your snippet code on the CPU shows different responses of the two...

Thanks!

[fmassa]

@lalil0u this is expected. Since https://github.com/torch/cutorch/pull/648 , the new caching allocator is on by default, so the memory is not immediately freed, so that overall run-time can be improved if you have lots of allocations. If you want to disable the caching allocator, you can set the environment flag THC_CACHING_ALLOCATOR=0 before running th

THC_CACHING_ALLOCATOR=0 th

[aschoenauer-sebag]

That works, thanks @fmassa

[vince62s]

Hello,
I read through this and I have a non-deterministic issue.
With this code here https://github.com/OpenNMT/OpenNMT/pull/196/files#diff-56d1c0c69bc0ce8f30dc5e71d1a67ba9R211
whether I add a second collectgarbage() or not same issue.
On one server, I have almost never the issue
On another one (same config Ubuntu 14.04) most of the time I have the issue where the GPU memory is not freed after the 2 lines (NIL + CG).
of course I use export THC_CACHING_ALLOCATOR=0 before running the lua script.