Plumber: future + promises within Docker is slow

[C24IO]

System details

Output of sessioninfo::session_info()():

─ Session info ───────────────────────────────────────────────────────────────────────────────
 setting  value                       
 version  R version 4.0.4 (2021-02-15)
 os       macOS Catalina 10.15.7      
 system   x86_64, darwin17.0          
 ui       AQUA                        
 language (EN)                        
 collate  en_US.UTF-8                 
 ctype    en_US.UTF-8                 
 tz       America/Indiana/Indianapolis
 date     2021-05-05                  

─ Packages ───────────────────────────────────────────────────────────────────────────────────
 ! package     * version    date       lib source                       
   assertthat    0.2.1      2019-03-21 [1] CRAN (R 4.0.2)               
   cachem        1.0.4      2021-02-13 [1] CRAN (R 4.0.2)               
   callr         3.5.1      2020-10-13 [1] CRAN (R 4.0.2)               
   cli           2.3.1      2021-02-23 [1] CRAN (R 4.0.2)               
   codetools     0.2-18     2020-11-04 [1] CRAN (R 4.0.4)               
   colorspace    2.0-0      2020-11-11 [1] CRAN (R 4.0.2)               
   commonmark    1.7        2018-12-01 [1] CRAN (R 4.0.2)               
   crayon        1.4.1      2021-02-08 [1] CRAN (R 4.0.2)               
   data.table    1.14.0     2021-02-21 [1] CRAN (R 4.0.2)               
   DBI           1.1.1      2021-01-15 [1] CRAN (R 4.0.2)               
   desc          1.2.0      2018-05-01 [1] CRAN (R 4.0.2)               
   devtools      2.3.2      2020-09-18 [1] CRAN (R 4.0.2)               
   digest        0.6.27     2020-10-24 [1] CRAN (R 4.0.2)               
   dplyr         1.0.5      2021-03-01 [1] Github (hadley/dplyr@7a96866)
   ellipsis      0.3.1      2020-05-15 [1] CRAN (R 4.0.2)               
   fansi         0.4.2      2021-01-15 [1] CRAN (R 4.0.2)               
   fastmap       1.1.0      2021-01-25 [1] CRAN (R 4.0.2)               
   fs            1.5.0      2020-07-31 [1] CRAN (R 4.0.2)               
   functools     0.2.0      2015-09-02 [1] CRAN (R 4.0.2)               
   future      * 1.21.0     2020-12-10 [1] CRAN (R 4.0.2)               
   generics      0.1.0      2020-10-31 [1] CRAN (R 4.0.2)               
   ggplot2     * 3.3.3      2020-12-30 [1] CRAN (R 4.0.2)               
   globals       0.14.0     2020-11-22 [1] CRAN (R 4.0.2)               
   glue          1.4.2      2020-08-27 [1] CRAN (R 4.0.2)               
   gtable        0.3.0      2019-03-25 [1] CRAN (R 4.0.2)               
   htmltools     0.5.1.1    2021-01-22 [1] CRAN (R 4.0.2)               
   htmlwidgets * 1.5.3      2020-12-10 [1] CRAN (R 4.0.2)               
   httpuv        1.5.5      2021-01-13 [1] CRAN (R 4.0.2)               
   httr          1.4.2      2020-07-20 [1] CRAN (R 4.0.2)               
   jsonlite      1.7.2      2020-12-09 [1] CRAN (R 4.0.2)               
   knitr         1.31       2021-01-27 [1] CRAN (R 4.0.2)               
   later         1.1.0.1    2020-06-05 [1] CRAN (R 4.0.2)               
   lattice       0.20-41    2020-04-02 [1] CRAN (R 4.0.4)               
   lazyeval      0.2.2      2019-03-15 [1] CRAN (R 4.0.2)               
   lifecycle     1.0.0      2021-02-15 [1] CRAN (R 4.0.2)               
   listenv       0.8.0      2019-12-05 [1] CRAN (R 4.0.2)               
   magrittr    * 2.0.1      2020-11-17 [1] CRAN (R 4.0.2)               
 P marsbackend * 0.0.0.9000 2021-04-12 [?] local                        
   Matrix        1.3-2      2021-01-06 [1] CRAN (R 4.0.4)               
   memoise       2.0.0      2021-01-26 [1] CRAN (R 4.0.2)               
   munsell       0.5.0      2018-06-12 [1] CRAN (R 4.0.2)               
   parallelly    1.24.0     2021-03-14 [1] CRAN (R 4.0.2)               
   paws        * 0.1.11     2021-03-13 [1] CRAN (R 4.0.2)               
   pillar        1.5.0      2021-02-22 [1] CRAN (R 4.0.2)               
   pkgbuild      1.2.0      2020-12-15 [1] CRAN (R 4.0.2)               
   pkgconfig     2.0.3      2019-09-22 [1] CRAN (R 4.0.2)               
   pkgload       1.2.0      2021-02-23 [1] CRAN (R 4.0.2)               
   plotly      * 4.9.3      2021-01-10 [1] CRAN (R 4.0.2)               
   plumber     * 1.1.0      2021-03-24 [1] CRAN (R 4.0.2)               
   prettyunits   1.1.1      2020-01-24 [1] CRAN (R 4.0.2)               
   processx      3.4.5      2020-11-30 [1] CRAN (R 4.0.2)               
   promises    * 1.2.0.1    2021-02-11 [1] CRAN (R 4.0.2)               
   ps            1.6.0      2021-02-28 [1] CRAN (R 4.0.4)               
   purrr       * 0.3.4      2020-04-17 [1] CRAN (R 4.0.2)               
   R6            2.5.0      2020-10-28 [1] CRAN (R 4.0.2)               
   Rcpp          1.0.6      2021-01-15 [1] CRAN (R 4.0.2)               
   remotes       2.3.0      2021-04-01 [1] CRAN (R 4.0.2)               
   rlang         0.4.10     2020-12-30 [1] CRAN (R 4.0.2)               
   roxygen2      7.1.1      2020-06-27 [1] CRAN (R 4.0.2)               
   rprojroot     2.0.2      2020-11-15 [1] CRAN (R 4.0.2)               
   rstudioapi    0.13       2020-11-12 [1] CRAN (R 4.0.2)               
   scales        1.1.1      2020-05-11 [1] CRAN (R 4.0.2)               
   sessioninfo   1.1.1      2018-11-05 [1] CRAN (R 4.0.2)               
   stringi       1.5.3      2020-09-09 [1] CRAN (R 4.0.2)               
   stringr     * 1.4.0      2019-02-10 [1] CRAN (R 4.0.2)               
   swagger       3.33.1     2020-10-02 [1] CRAN (R 4.0.2)               
   testthat      3.0.2      2021-02-14 [1] CRAN (R 4.0.2)               
   tibble        3.1.0      2021-02-25 [1] CRAN (R 4.0.2)               
   tidyr         1.1.2      2020-08-27 [1] CRAN (R 4.0.2)               
   tidyselect    1.1.0      2020-05-11 [1] CRAN (R 4.0.2)               
   usethis       2.0.1      2021-02-10 [1] CRAN (R 4.0.2)               
   utf8          1.1.4      2018-05-24 [1] CRAN (R 4.0.2)               
   vctrs         0.3.6      2020-12-17 [1] CRAN (R 4.0.2)               
   viridisLite   0.3.0      2018-02-01 [1] CRAN (R 4.0.1)               
   webutils      1.1        2020-04-28 [1] CRAN (R 4.0.2)               
   withr         2.4.1      2021-01-26 [1] CRAN (R 4.0.2)               
   xfun          0.21       2021-02-10 [1] CRAN (R 4.0.2)               
   xgboost       1.3.2.1    2021-01-18 [1] CRAN (R 4.0.2)               
   xml2          1.3.2      2020-04-23 [1] CRAN (R 4.0.2)               

[1] /Library/Frameworks/R.framework/Versions/4.0/Resources/library

 P ── Loaded and on-disk path mismatch.

Example application or steps to reproduce the problem

Implement future + promises within an R script, call the script from within a Docker container

Describe the problem in detail

When we utilize future_promise + plumber in a docker container, large data structures are being loaded at each function call instead of being included in the process fork.

This causes a constant high latency.

[schloerke]

Do you have a small example file that shows this off? There are many ways to implement future_promise + plumber()

[C24IO]

Hey Barret,

Thanks for the reply

I am getting a colleague to confirm too but something like this -

library(future)
library(promise)
future::plan(multicore)

#' Ping to show server is there
#' @get /ping
function() {
  return('')}


#' Parse input and return prediction from model
#' @param req The http request sent
#' @post /invocations
function(req) {
  future_promise({
  # Setup locations
  prefix <- '/opt/ml'
  model_path <- paste(prefix, 'model', sep='/')

  # Bring in model file and factor levels
  load(paste(model_path, 'mars_model.RData', sep='/'))

  # Read in data
  conn <- textConnection(gsub('\\\\n', '\n', req$postBody))
  data <- read.csv(conn)
  close(conn)

  # Convert input to model matrix
  scoring_X <- model.matrix(~., data, xlev=factor_levels)

  # Return prediction
  return(paste(predict(mars_model, scoring_X, row.names=FALSE), collapse=','))}
})

[meztez]

Do you have cpu limits on your container?

[C24IO]

No

[meztez]

I had the same behaviour and was able to track it down to omp lib overhead. I'm curious to see if what you are experiencing is similar.

[C24IO]

Can you please elaborate about omp lib, sorry I am a bit rusty when it comes to R.

[schloerke]

Your original code is not leveraging the benefits of forking. Each route is reading your data each time it is executed.

From https://github.com/HenrikBengtsson/future#multicore-futures

Forking an R process can be faster than working with a separate R session running in the background. One reason is that the overhead of exporting large globals to the background session can be greater than when forking, and therefore shared memory, is used.

So the earlier we can get it into memory, the better.

Suggested changes:

• Move all code that can be executed once, outside of the route functions. No need to do this many times.
• Do not use textConnections(). I have found them to be slow.
• When reading a csv, use readr::read_csv() with a cols = readr::col_type(....) definition. It is faster than read.csv(). Using cols provides clear and consistent to parse the column types.
  
  
  
  • If you do not need all of the columns, I suggest using vroom::vroom() and its col_select parameter which will only read selected columns into memory. VERY handy for non-trivial data.
  
  
  • It seems like you are trying to parse a csv file as input, I suggest using the csv body parser and using the parsed csv data from req$body. Ex:
  
  

#' Parse input and return prediction from model
#' @param req The http request sent
#' @post /invocations
#' @parser csv
function(req) {
  ....
  scoring_X <- model.matrix( ~., req$body, xlev = factor_levels)
  ....
}

• I don't fully understand your output type. I would either leverage the json type or change it to a text type by adding the plumber line

#' @serializer text

Final suggested code:

library(future)
library(promise)
future::plan(multicore)

#' Ping to show server is there
#' @get /ping
function() {
  return('')
}


## Read static data once on server start
# Setup locations
prefix <- '/opt/ml'
model_path <- paste(prefix, 'model', sep='/')

# Bring in model file and factor levels
load(paste(model_path, 'mars_model.RData', sep='/'))


#' Parse input and return prediction from model
#' @param req The http request sent
#' @post /invocations
#' @parser csv
#' @serializer text
function(req) {
  future_promise({
    # Convert input to model matrix
    scoring_X <- model.matrix( ~ ., req$body, xlev = factor_levels)

    # Return prediction
    return(paste(predict(mars_model, scoring_X, row.names=FALSE), collapse=','))
  })
}

[meztez]

OMP observation : https://github.com/dmlc/xgboost/issues/6732

We decided to do without futures and promises for time sensitive (<50ms) deployment as forking is a non-zero cost operation (more memory use + delays).

[ecrowell4]

For concreteness, the code is along these lines:

library(future)
library(promise)
library(my_package)
future::plan(multicore)

exported_functions <- getNamespaceExports("my_package")
#' Ping to show server is there
#' @get /ping
function() {
  return('')}


#' Parse input and return prediction from model
#' @param req The http request sent
#' @post /invocations
function(req) {
  future_promise({
    input <- as.list(req$body)
    f <- input$pkgfunction
    input$pkgfunction <- NULL
    args <- input
    req$format <- "json"

    if(f %in% exported_functions) {
      out <- do.call(f, args)
      attr(out, "serialize_format") <- req$format
      return(out)
    } else {
      out <- NULL
      return (out)
    }
  })
}

The package my_package is what contains the heavy data structures which some of the functions in my_package use. Would this not be loading them in before the fork?

[meztez]

maybe not because of lazydata loading in R

[schloerke]

So here is a scary message: https://github.com/HenrikBengtsson/future/issues/59#issuecomment-397141732

Also, some people argue that forked processes (used by multicore futures, mclapply(), ... - so not Windows) may consume less memory because of the "shared memory" property of process forking. However, it has been shown/mentioned several that R's garbage collector can really mess this up - if the garbage collector starts running in one of the forked child processes, or the master process, then that originally shared memory can no longer be shared and the operating system starts copying memory blocks into each child process. Since the garbage collector runs whenever it wants to, there is no simple way to avoid this.

Maybe you could use cluster which always exists, rather than forking the process or starting a new process for each request?

https://future.futureverse.org/reference/cluster.html

library(future)
plan(cluster, workers = 2)

....

Worth a try

[ecrowell4]

Using cluster is worth a shot. I'll try it out.

Regarding

maybe not because of lazydata loading in R

How awful would it be to set LazyData: false in my_package's DESCRIPTION? I'm rather new to R so I'm not familiar with best practices...

[schloerke]

maybe not because of lazydata loading in R

Great idea, @meztez!

https://r-pkgs.org/data.html#data-data

I recommend that you always include LazyData: true in your DESCRIPTION. usethis::create_package() does this for you.

In your use-case, you probably want LazyData: false in your DESCRIPTION file. Then it might work as you'd expect.

[schloerke]

Haha. Great timing! I'd try the lazydata first. Adjusting the DESCRIPTION file is much less headache than starting and stopping clusters.

[ecrowell4]

After setting LazyData: false, it appears that the data files in my_package/data are not available. The general file structure of the package is

  - R
    - file1.R
    - file2.R
  - data
    - data_table_1.Rda
    - data_table_2.Rda

If I invoke a function that uses the global data_table_1, I get an error "... object 'data_table_1' not found>"

[schloerke]

Maybe set LazyData: true, but call the data in the script?

# plumber.R

mypackage::data_table_1
mypackage::data_table_1

library(future)
....

[ecrowell4]

I will give it a try and let you know.

Something we haven't discussed yet: do we have any idea why this only happens when the package is placed in a docker container and I run the docker process? If I host the plumber script locally, there is no issue with performance.

[ecrowell4]

Regarding setting LazyData: true and calling the data in the script, there are upwards of 75 data tables in /data, so I don't think calling each one in the script is feasible in our particular situation. Maybe there's a slick way of doing it I don't know about?

Regarding the earlier suggestion of moving away from multicore, I tried implementing multisession as in the following

library(future)
library(promise)
library(my_package)
future::plan(multisession)

exported_functions <- getNamespaceExports("my_package")
#' Ping to show server is there
#' @get /ping
function() {
  return('')}


#' Parse input and return prediction from model
#' @param req The http request sent
#' @post /invocations
function(req) {
    input <- as.list(req$body)
    f <- input$pkgfunction
    input$pkgfunction <- NULL
    args <- input
    req$format <- "json"

    if(f %in% exported_functions) {
      out <- future_promise({do.call(f, args)})
      attr(out, "serialize_format") <- req$format
      return(out)
    } else {
      out <- NULL
      return (out)
    }
}

(Note: I'm now only wrapping the do.call in future_promise({}), otherwise none of the functions in my_package are recognized). However, the namespace for my_package is getting truncated in the parallel R sessions. For example, if I run the snippet

ns <- ls(getNamespace('my_package'))
print(ns)

in the main R session, I get a list of functions f1, f2, f3,..fn. However, when I call it within future_promise({}), the result is a strict subset of the result from the main R session, e.g. f3, f6, ....

[ecrowell4]

I see the same behavior for

future::plan(cluster, workers = 2)

[schloerke]

Maybe there's a slick way of doing it I don't know about?

Maybe?: (untested)

data(list("name1", "name2", ....), package = "mypackage")

---

You're trying to add a "serialize_format" attribute to the final value. This step should be added to the result of the future_promise.

So replace this code:

      out <- future_promise({do.call(f, args)})
      attr(out, "serialize_format") <- req$format
      return(out)

with

      prom <- future_promise({do.call(f, args)}) %...>% {
        out <- .
        attr(out, "serialize_format") <- req$format
        out
      }
      return(prom)

The plumber route async execution will continue to evaluate the promise object until it returns a non-promising object. This final value will be passed on to the serializer. So, we need to add the attr to the realized object with a followup promise (via %...>%).

---

My gut reaction (to avoid constantly loading large data) would be to not use future::plan(multisession) as it launches a new R session for every future execution.

[ecrowell4]

I went back to multicore and loaded the data in the script. That seems to have done the trick! My script looks like the following:

library(future)
library(promise)
future::plan(multicore)

library(my_package)
d<-as.list((data(package = 'my_package')$results[,'Item']))
data(list=d, package='my_package')

exported_functions <- getNamespaceExports("my_package")
#' Ping to show server is there
#' @get /ping
function() {
  return('')}


#' Parse input and return prediction from model
#' @param req The http request sent
#' @post /invocations
function(req) {
  future_promise({
    input <- as.list(req$body)
    f <- input$pkgfunction
    input$pkgfunction <- NULL
    args <- input
    req$format <- "json"

    if(f %in% exported_functions) {
      out <- do.call(f, args)
      attr(out, "serialize_format") <- req$format
      return(out)
    } else {
      out <- NULL
      return (out)
    }
  })
}

Right now I still have LazyData: false in the DESCRIPTION file, which is likely redundant. I will set it to true and see if the behavior is the same.

[ecrowell4]

@schloerke @meztez thank you!

[ecrowell4]

It seems the code as I have it is unstable. I often come across the error

<UnexpectedFutureResultError: Unexpected result (of class ‘NULL’ != ‘FutureResult’) retrieved for MulticoreFuture future (label = ‘<none>’, expression = ‘{; body-of-future-promise; }; }’): >
DEBUG: BEGIN TROUBLESHOOTING HELP
Future involved:
MulticoreFuture:
label: '<none>'
Expression:
body-of-future-promise
Lazy evaluation: FALSE
Asynchronous evaluation: TRUE
Local evaluateion: TRUE
Environment: string
Capture standard output: TRUE
Capture condition classes: 'condition'
Globals: <none>
Packages: 1 packages ('my_package')
L'Ecuyer-CMRG RNG seed: c(10407, -904526415, -1004183404, 739845927, 1610186213, 1565061197, 1754220748)
Resolved: TRUE
Value: <not collected>
Conditions captured: <none>
Early signaling: FALSE
Owner process: string
Class: ‘MulticoreFuture’, ‘MultiprocessFuture’, ‘Future’, ‘environment’
DEBUG: END TROUBLESHOOTING HELP

Is this a common problem?

[ecrowell4]

I found this here https://github.com/HenrikBengtsson/future/issues/198#issuecomment-408170152:

Getting Unexpected result (of class ‘NULL’ != ‘FutureResult’) ... errors for multicore futures, means that the forked child process terminated - that's what "If none of the specified children are still running, it returns 'NULL'" means in ?parallel::mccollect. AFAIK, termination of a forked process prior to the value has been collected sounds like it has crashed.

So maybe the child process is getting terminated. Any thoughts?

[schloerke]

It is probably not an R error as those should be caught and returned.

You said it was a Docker container, could it be that the process ran out of memory? That would cause the process to unexpected terminate / disappear. https://docs.docker.com/config/containers/resource_constraints/

[ecrowell4]

I know the Docker container has enough memory to run at least one of the R processes. However, if memory starts getting copied, then that is no longer the case. Could it be that the garbage collector is kicking in and causing memory to be copied, like you mentioned earlier?

If I were to use cluster, then I have a fixed number of R processes running constantly, is that right? Of course I would have to make sure the container has enough memory allocated to it, but I would at least have a concrete value I know I would have to allocate.

[jeffkeller87]

What is the scale of the latency you are experiencing? This TCP behavior on Linux could also be contributing if you expect very low latency (<100ms): https://github.com/HenrikBengtsson/future/issues/437#issuecomment-720844779.

I have had the most success using a container start script that spawns multiple plumber processes and then uses nginx to round-robin requests to those processes. Of course, this suffers from the duplicated-objects-in-memory issue that some of the other options discussed in this thread also suffer.

[C24IO]

Hey @jeffkeller87 would you mind pointing me to a sample implementation of the method you mention? I would like to try it out in parallel.

[jeffkeller87]

@C24IO If you ~where~ were to start 4 plumber services in the container the nginx config would be something like this.

[C24IO]

Thanks a lot @jeffkeller87 . I did not have a chance to try this yet, but will do soon.

[C24IO]

Since we got the answer to our original question and in one way or other addressed it I am resolving this issue. If anyone disagrees, please reopen with a comment.