Plumber: Error in temporary files

[katiamar63]

Hi,
Does anyone knows what can be the reason for errors as presented below? Generally, Plumber works OK (I get plots, results of linear regression, pdfs etc.). Every single API endpoint has been tested and worked well, but from time to time on 'production' R server I receive errors on R console. As a result the main script does not work anymore and I have to restart server and starting script of Plumber from scratch. What can be the reason for that? Do I need to make special settings on R server?
It happens both in Windows 10 with Rx64 3.4.4 and Ubuntu 16.04 with the same version of R server.
Best
Andrzej

Here is code of main function

library(plumber)
library(jsonlite)

lmJSON <- function(){
    function(val, req, res, errorHandler){
        tryCatch({
            # json <- jsonlite::toJSON(val,auto_unbox=TRUE) not working with lm
            json <- serializeJSON(val,digits=8,pretty=FALSE)
            res$setHeader("Content-Type", "application/json")
            res$body <- json

            return(res$toResponse())
        }, error=function(e){
            errorHandler(req, res, e)
        })
    }
}

tryCatch({
addSerializer("lmJSON",lmJSON)},
 error=function(e){
})
pr <-plumber :: plumb("spinescripts.r") #here are my API Endpoints

pr$registerHook("exit", function(){
    print("Bye bye!")
})
pr$run(port=8000)

Errors:

1: In file(open = "w+b", encoding = "UTF-8") :

 cannot open file '/tmp/Rtmp0WrQg2/Rf30be2240828b': No such file or directory

2: In file(open = "w+b", encoding = "UTF-8") :

 cannot open file '/tmp/Rtmp0WrQg2/Rf30be62c4551d': No such file or directory

3: In file(open = "w+b", encoding = "UTF-8") :

 cannot open file '/tmp/Rtmp0WrQg2/Rf30be48d6d67c': No such file or directory

4: In file(open = "w+b", encoding = "UTF-8") :

 cannot open file '/tmp/Rtmp0WrQg2/Rf30be1ea5f8d0': No such file or directory

[schloerke]

I'm trying to think of situations outside of plumber as it works under normal situations.

When the cannot open file errors appear, is the machine running out of disk space? If temp files can't be opened, that's my initial reaction.

How long is plumber running until the errors appear?

[katiamar63]

The plumber works for a long time, and definitely there is a space on disk. Probably I should use a pool of plumber processes with load balancer? Maybe this is related to the fact that I am not clearing environment after handling request?

BTW, I am using evaluate function to get output and messages after evaluating a script (as you suggested - thank you again for that). Do you know how could I use evaluate to get results of plots as well? When I serialize evaluate response to JSON in Plumber and send it back to front end, I can see in Chrome inspector, that the plot is also handled by evaluate and produces the list element with big amount of encoded binary data (of type "raw"). This looks similar to png data encoded with base64; but with exception that it does not contain iVBOR prefix) and when added to it does not work. Do I have to overwrite output handler for that?

Certainly, when I use Plumber endpoint with @png serializer for plots, then sending plots works perfect. But I would like to use only evaluate response to process full script and send back all outputs, warning and binary image data in one JSON with character data and binary data (encoded with base64).

[schloerke]

Maybe I'm not clearing the environment after handling the request?

The route handling function should be handled by regular gc. So no need to do that there.

I'd say the root of the problem is still unsolved.

evaluate handle plots?

If you have a reprex::reprex({...}) of the plots, I could look into it.

I'm wondering if reformatting the code as an rmarkdown input and processing it with rmarkdown with yaml header option self_contained: TRUE, might be easier in the end. It'd be formatted and not in parts, but it might work for your needs.

[katiamar63]

Thanks again,
Sorry, I tried with reprex, but I did not know how to deal with it. My plumber endpoint is just as simple as

#' @post /executescript
#' @serializer lmJSON
function(txtScript,data){
library(evaluate)
evaluate<-evaluate(txtScript)
}

and the exemplary script I am sending to it:

"x<-c(1,2,3,4,5,9,11,12,90);plot(x)"

When I get a response I parse returned JSON. Every element of the list is parsed and all characters are concatenated and displayed in my visual component. The problem is that for 'plot' call I have 'raw' variable containing this:

 {
                    "type": "raw",
                    "attributes": {
                        "pkgName": {
                            "type": "character",
                            "attributes": {},
                            "value": [
                                "graphics"
                            ]
                        }
                    },
                    "value": "AAAAAAAAAAAAAAAAAADgPwEAAAD/////bwAAA .... and many many more, mostly A values, with '\n' characters (all about 90KB)
}

and I don't know how to deal with it. Value property does not look like base64. I studied your code in Plumber for @png serializer and datacamp project as well, but these are slightly different scenarios.

Probably I will have to use temporary file and capture all graphics to pdf and I will be getting this pdf in second request.

[schloerke]

@katiamar63 Not saying it'll work for everything, but this should get you farther...

update_images_to_png <- function(eval_list, height = 480, width = 480) {
  lapply(eval_list, function(val) {
    if (!inherits(val, "recordedplot")) {
      return(val)
    }
    tmp <- tempfile(fileext = ".png")
    tryCatch({
      png(tmp, height = height, width = width, units = "px")
      # replay plot inside a png device
      replayPlot(val, reloadPkgs = TRUE)
    }, finally = {
      dev.off()
    })
    # read the binary plot and convert to a base64 image tag
    img <- readBin(tmp, "raw", file.info(tmp)$size)
    htmltools::tags$img(
      style = paste0("height: ", height, "px; width: ", width, "px;"),
      src = paste0("data:image/png;base64,", base64enc::base64encode(img))
    )
  })
}

lmJSON <- function(){
  function(val, req, res, errorHandler){
    tryCatch({
        val <- update_images_to_png(val)
        json <- serializeJSON(val,digits=8,pretty=FALSE)
        res$setHeader("Content-Type", "application/json")
        res$body <- json

        return(res$toResponse())
    }, error=function(e){
        errorHandler(req, res, e)
    })
  }
}

[katiamar63]

Again, great thanks that you are willing to help me with that. I will try it.

[katiamar63]

So far works perfectly, I can even serialize many files in one response,
Many thanks again.
Andrzej