Rcpp: Memory Management & Garbage Collection

[elbamos]

I've been trying to find documentation drilling-down into how Rcpp handles collecting garbage with used c++ objects, and I haven't been able to find anything.

I'm working on an R package (https://github.com/elbamos/largevis) where memory consumption can be very large. I've been writing the computationally-intensive parts in C++ with Rcpp and RcppArmadillo.

To try to reduce memory usage, I've been trying to do more in C++ and less in R and the "R way." In my initial revisions, for example, my principal data structures were NumericMatrix and NumericVector objects. I've gradually been moving from Numeric* objects to arma:: objects. And, where possible, replacing matrices/vectors as data structures with appropriate objects from the C++ standard library. I've also been reducing returns to R, trying to handle more of the steps inside of C++ code without returning variables to R and then passing them back to Rcpp.

Instead of reducing memory usage, this all seems to be having the opposite effect.

Can you clarify when and how objects created in Rcpp get garbage collected?

For example, let's say that inside a C++ loop, in each iteration, I create a NumericMatrix, and arma::mat, and an std:: structure, each of which should only live during a single pass through the loop. Will these be garbage collected while the loop is executing? Does it matter whether the variable is declared inside or outside the loop?

Similarly as to recursion: If I create objects in a recursive function call, are they garbage collected when the creating-call returns, or must they persist until the Rcpp function returns control to R?

Thank you.

[eddelbuettel]

"It's complicated."

Look at the armadillo documentation. You can clearly create new objects. They will have nothing to do with R. Whereas when we pass objects down from R _we intentionally make them proxy objects_ to not incur copying costs.

Your objectives may be different. At the end of the day you may need to do some profiling at your end. But without actual reproducible code showing anything to go by, there is little we can say now.

Happy to help though.

[elbamos]

Uh-oh :p

Well, I'm happy to share reproducible code, but that may be more detail than you want to get into.

If you want to take a peek, the relevant file is neighbors.cpp in my repo.

Its an algorithm for approximating nearest neighbors in O(N).

In the first stage of the algorithm, we create up to T trees. Each tree is made by randomly sampling two nodes, finding an equidistant hyperplane, dividing the remaining nodes between them, and recursing until we have a leaf with fewer than F nodes. I've generally been storing the assignments in a [F,N] matrix, either a NumericMatrix, or an arma::mat passed by reference when I recurse. Memory consumption should peak at F * N * T.

My current version (not on git) which prompted this issue, I tried to instead to recurse by passing by reference a std::vectorstd::set

The version currently on git returns a merged matrix to R, which then initiates the second phase in another call to Rcpp. (My version not-on-git tries to merge these phases.)

The second phase is neighborhood exploration. Up to M iterations, for each node i, we walk through all of the nodes j identified as neighbors in the first pass, and all of the nodes k identified as neighbors' neighbors. We keep a max heap, implemented as a std::priority_queue. We order in the queue by the distance of (j or k) from i, and pop the queue whenever its size exceeds K.

So, each pass through the loop, we create a new [K,N] matrix, and N priority_queues.
Originally, I was using NumericMatrix objects. More recently I've been trying with arma::mat. Memory consumption should peak at K * N * 3.

Any suggestions?

[eddelbuettel]

On a commuter train home, and slightly fried from the day -- but did you see RcppAnnoy? My binding to to Erik's Annoy library in C++ (but written by him for Python bindings). Maybe you can benchmark some?

Other than that "it's hard" ...

[elbamos]

Yeah I saw that after I'd been working on this for like a week :p I've been
intending to benchmark performance between them.

Sent from Nylas N1, the extensible, open source
mail client.

On May 17 2016, at 9:54 pm, Dirk Eddelbuettel [email protected]
wrote:

On a commuter train home, and slightly fried from the day -- but did you see
RcppAnnoy? My binding to to Erik's Annoy library in C++ (but written by him
for Python bindings). Maybe you can benchmark some?

Other than that "it's hard" ...

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub

[elbamos]

BTW if you decide to take a look, I just pushed the version I think _should_ be memory efficient. In the neighborhood exploration phase, processing a 42k MNIST set (80 MB), memory peaks at ~ 1.2GB. Later, in the SGD phase, memory hits 8+ GB. That phase should actually consume very little memory, since it does all of its work in-place and doesn't need to make copies of anything.

[eddelbuettel]

There is no minimally reproducible issue here.

[elbamos]

Meaning its too detailed, or too nebulous, or that you'd like me to include a test dataset?

[eddelbuettel]

There is really nothing for us to look at. You (quite lovingly) described observations you have about your program. For us to do something here, we need something tangible preferably related to Rcpp. Ideally you'd show an example which needlessly copied, leaked, ...

See e.g. how to create a great reproducible example in R from StackOverflow to give you an idea.

[dcdillon]

The trick here, as far as I can see (and without examining the code) is this:

You say that you loop and "create a NumericMatrix, and arma::mat, and an std:: structure, each of which should only live during a single pass through the loop".

This is fundamentally not correct. When creating the NumericMatrix you tell R to "create a matrix". R will dutifully allocate memory for and create a matrix. When this goes out of scope (i.e. the next iteration of the loop) you lose any C++ reference to it. This is fine, however, you haven't told R that you're done with this Matrix in any meaningful sense. The next time R _might_ free this memory is on the next call to an R function (which necessarily occurs after you exit your full Rcpp context).

If you're not going to return something to R, there is no reason to create an Rcpp object to contain it.

[elbamos]

@dcdillon Thanks, that's exactly the kind of thing that's the reason I'm asking.

So what's the memory-optimal thing to do?

I need to create objects, in a loop and recursively, which are rows of matrices. Some of these objects can be pointers to subsets of the actual matrix data. For other objects, I could re-use the same space each pass through the loop (except that each OpenMP thread will need its own).

I tried transposing the matrices and making them arma:: objects, because arma::Col objects are pointers. But then the whole dataset gets duplicated, and memory consumption actually seemed to go up.

What would you recommend?

[eddelbuettel]

I like Armadillo _a lot_. Well documented, reliable, widely used. Disconnected from R objects (unless you objects in from R, they remains proxies -- important point) so that you can use OpenMP or TBB.

[elbamos]

@eddelbuettel Thanks!

When does arma collect trash though?

consider:

void function(NumericMatrix x, NumericVector y) {

     const arma::vec y2 = as<arma::vec>(y);
     for (int i = 0; i < 10; i++) {
          const arma::vec x2 = x.row(i);
         // ignore issues of matrix dimensions for this example
          const double d = sqrt(sum(pow(y2 - x2, 2)));
         // /do something with d
     }
}

Will memory be allocated for 10 arma::vec's for y2? Will the call to x.row(i) create an incidental NumericVector? If memory would be allocated here, could I solve the issue with:

void function(NumericMatrix x, NumericVector y) {

     const arma::vec y2 = as<arma::vec>(y);
     arma::vec x2;
     #pragma omp parallel for private(x2) 
     for (int i = 0; i < 10; i++) {
          for (j = 0; j < y.ncol(); j++) x2[j] = y[i,j];
         // ignore issues of matrix dimensions for this example
          const double d = sqrt(sum(pow(y2 - x2, 2)));
         // /do something with d
     }
}

[dcdillon]

There's no concept of arma trash collection. When the object goes out of scope it will be destructed.

[eddelbuettel]

Just like @dcdillon said: C++ has well defined scope; Armadillo gets you that. With R objects (which are managed in a pool that gets occassionally garbage collected) you can be less sure.

Both side are reasonably well understood and documented, respectively. What messes with us here is that we are at the intersection. But if you choose carefully you get to pick the behaviour you want...

[elbamos]

This is really helpful guys.

So in the first example, I would not end up with memory for 10 rows being allocated, because each row would get destructed at the end of that iteration? And I don't need to resort to the second method?

If so, can I control when the object is destructed by inserting otherwise extraneous {}'s in the middle of the function?

Also, do calls to a NumericMatrix's .row() method allocate RAM?

[eddelbuettel]

Note that you use the proxy constructors. See the Armadillo documentation and look closely at 'advanced constructors'. By default Rcpp et al do re-use because it is the right thing to do in most case _but clearly not when you want OpenMP_. Consider extra copies (via the corresponding arguments to the constructors).

[eddelbuettel]

I am going to close this here as there never was an actual bug or issue.

For general discussion how the state of the world and how to do this, the rcpp-devel list is a better place.

[elbamos]

thanks so much guys -- it looks like you gave me enough I was able to solve it