Julia: Generated code memory leaks

[tmptrash]

Hello guys!

I found a generated code related memory leak issue. I wrote a post about it on julia-lang users group, but it looks like everyone have no time for that. I emailed to @JeffBezanson about this issue and he knows about it. I have no possibility to fix this by myself, because i have no such experience in C. So, i decided to create this issue to track it somehow.

So, here is a problem:

function leak()
    for i=1:100000
        t = Task(eval(:(function() produce() end)))
        consume(t)
        try
          Base.throwto(t, null)
        end
    end
    gc()
end

Every call of leak() eats ~30mb of memory on my PC. This problem exists on both Linux and Windows platforms.

julia> versioninfo()
Julia Version 0.4.2
Commit bb73f34 (2015-12-06 21:47 UTC)
Platform Info:
  System: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Core(TM) i7-4700HQ CPU @ 2.40GHz
  WORD_SIZE: 64
  BLAS: libopenblas (NO_LAPACK NO_LAPACKE DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: liblapack.so.3
  LIBM: libopenlibm
  LLVM: libLLVM-3.3

[ViralBShah]

Thanks for filing - so that this can be tracked.

[yuyichao]

Is this just because we don't free jit code?

[vtjnash]

We don't free the results of eval because the memory cost is lower than the computational cost (in the Julia cost model where functions -- even closures -- are statically defined). Clearly, the eval is trivially unnecessary here, so this won't be fixed unless someone finds a real use case.

[tmptrash]

• I have use case in my app. This example (above) - is simplified version of it. In two words, i have to generate a lot of code from the string, which is modified all the time. So in my case eval(parse("...")) are called all the time. This is something like self modified application. It's used in evolution biology research.
• I also have a case when i modify AST and call eval() after that many times. I was wondering to have this feature (AST modification) in Julia, by the way :) In this case memory leak also appears. So, after 20min of working, my app eats 6Gb of memory :)

Is it possible to solve this?

[JeffBezanson]

Reopened. I think this is a real issue that I'd like to fix eventually.

[tmptrash]

This is great news :) For us this is a real stopper. Thanks Jeff.

[afbarnard]

Is it generated code that is causing the maxrss to monotonically increase with every set of tests in the entire suite? If so, this issue affects my ability to run all the tests on my laptop which has 4GB. Right now, I can only run the tests single-threaded (make testall1) because running the tests multi-threaded (make testall) runs out of memory 2 times. (2 test workers are terminated.) Plus, generated code taking up so much memory is not in concert with my intuition about running tests independently. Perhaps I should open an issue? (Note I have been using the release-0.4 branch, not master. See #13719 for backstory.)

[JeffBezanson]

Yes, that's probably a significant part of the problem. Also ref #14626

[JeffBezanson]

We could also perhaps restart workers more frequently, e.g. every few test files, to use less persistent memory.

[tkelman]

There's an environment variable you can set to do just that. Have to look at runtests to check exactly how it's spelled.

[yuyichao]

We already have JULIA_TEST_MAXRSS_MB

[robertfeldt]

There are also use cases from genetic programming and other code generation/synthesis situations when one wants to compile and run a large number of programs in order to then select some subset of them.

See discussion here:
https://discourse.julialang.org/t/is-mem-of-compiled-evaled-functions-garbage-collected/2231

Also see the (closed) issue here:
https://github.com/JuliaLang/julia/issues/20755#issuecomment-281936581

[Nosferican]

Status of this?

[freddycct]

My issue https://github.com/JuliaLang/julia/issues/37560 was closed. So I am posting my MWE here. I used Flux/Zygote with pmap.

using Distributed
addprocs(4)

@everywhere mutable struct A
    a::Float32
end

@everywhere function genprog(n, p::A)
    map(1:n) do i
        y = rand()
        mdname = gensym()
        expr = :(module $mdname
            f(x) = 2*x + $y + $p.a
            end
        )
        m = eval(expr)
        Base.invokelatest(m.f, p.a)
    end
end

function main()
    i = 0
    x = A(rand())
    while true
        println("epoch $(i)")
        @everywhere GC.gc()

        tasks = rand(1:100, 100)
        _, timeTaken, _, _, _ = @timed let x=x
            pmap(tasks) do n
                genprog(n, x)
            end
        end
        @show timeTaken
        x.a = rand()
        i += 1
    end
end

main()

[aeisman]

In another use case, I have been having the same problem with a combination of Distributed and RCall.jl. It appears that repeated uses of RCall are causing a similar memory leak in my case up to 1TB of combined RAM and VRAM usage.