Prophet: Can't fit.prophet() when updated to R 3.6.0 in macOS: storage error

[laresbernardo]

I've used prophet a thousand times and updated to R 3.6.0 in mi macOS and now I'm getting this error when I try to fit a model:

m <- fit.prophet(m, prop)
Error in get("storage", envir = as.environment(x)) : 
  object 'storage' not found

I uninstalled and installed from scratch, and when loaded for the first time, this is what I get:

library(prophet)
Loading required package: Rcpp
Loading required package: rlang
Registered S3 methods overwritten by 'ggplot2':
  method         from 
  [.quosures     rlang
  c.quosures     rlang
  print.quosures rlang

Here is my session info if needed:

R version 3.6.0 (2019-04-26)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Mojave 10.14.5

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] rstan_2.18.2       StanHeaders_2.18.1 ggplot2_3.1.1      prophet_0.5        rlang_0.3.4       
[6] Rcpp_1.0.1        

loaded via a namespace (and not attached):
 [1] pillar_1.4.1       compiler_3.6.0     plyr_1.8.4         prettyunits_1.0.2  tools_3.6.0       
 [6] pkgbuild_1.0.3     tibble_2.1.3       gtable_0.3.0       pkgconfig_2.0.2    cli_1.1.0         
[11] rstudioapi_0.10    parallel_3.6.0     loo_2.1.0          gridExtra_2.3      withr_2.1.2       
[16] dplyr_0.8.1        stats4_3.6.0       grid_3.6.0         tidyselect_0.2.5   glue_1.3.1        
[21] inline_0.3.15      R6_2.4.0           processx_3.3.1     tidyr_0.8.3        callr_3.2.0       
[26] purrr_0.3.2        magrittr_1.5       scales_1.0.0       ps_1.3.0           matrixStats_0.54.0
[31] assertthat_0.2.1   colorspace_1.4-1   stringi_1.4.3      lazyeval_0.2.2     munsell_0.5.0     
[36] crayon_1.3.4      

Any idea on why is this happening, and most important of all, how to fix it? Thanks 👍

[bletham]

I don't think that code get("storage", envir = as.environment(x)) is istelf in prophet, so I'm wondering if the error might be coming from a dependency. Could you run traceback() right after the error to see where it's coming from?

It looks like R 3.6 hasn't made it into the package manager repository for me yet so I haven't been able to try and reproduce this.

[laresbernardo]

I had to reduce the amount of data so the traceback() results can be shown. This is what I get:

11: get("storage", envir = as.environment(x))
10: new("Module", .xData = <environment>)$stan_fit4model15cd647f12245_prophet
9: new("Module", .xData = <environment>)$stan_fit4model15cd647f12245_prophet
8: eval(call("$", mod, paste("stan_fit4", model_cppname, sep = "")))
7: eval(call("$", mod, paste("stan_fit4", model_cppname, sep = "")))
6: object@mk_cppmodule(object)
5: .local(object, ...)
4: (new("nonstandardGenericFunction", .Data = function (object, 
       ...) 
   {
       if (is.sparc()) {
           msg <- "optimizing() will likely crash on SPARC."
           if (interactive()) 
               stop(msg, " Run in batch mode to test.")
           else message(msg)
       }
       standardGeneric("optimizing")
   }, generic = "optimizing", package = "rstan", group = list(), 
       valueClass = character(0), signature = "object", default = NULL, 
       skeleton = (function (object, ...) 
       stop("invalid call in method dispatch to 'optimizing' (no default method)", 
           domain = NA))(object, ...)))(object = new("stanmodel", 
       model_name = "prophet", model_code = "functions {\n  matrix get_changepoint_matrix(vector t, vector t_change, int T, int S) {\n    // Assumes t and t_change are sorted.\n    matrix[T, S] A;\n    row_vector[S] a_row;\n    int cp_idx;\n\n    // Start with an empty matrix.\n    A = rep_matrix(0, T, S);\n    a_row = rep_row_vector(0, S);\n    cp_idx = 1;\n\n    // Fill in each row of A.\n    for (i in 1:T) {\n      while ((cp_idx <= S) && (t[i] >= t_change[cp_idx])) {\n        a_row[cp_idx] = 1;\n        cp_idx = cp_idx + 1;\n      }\n      A[i] = a_row;\n    }\n    return A;\n  }
# (...)

# (...)
current_statement_begin__ = 107;\n            lp_accum__.add(normal_log<propto__>(sigma_obs, 0, 0.5));\n            current_statement_begin__ = 108;\n            lp_accum__.add(normal_log<propto__>(beta, 0, sigmas));\n            current_statement_begin__ = 111;\n            if (as_bool(logical_eq(trend_indicator,0))) {\n\n                current_statement_begin__ = 112;\n                lp_accum__.add(normal_log<propto__>(y, add(elt_multiply(linear_trend(k,m,delta,t,A,t_change, pstream__),add(1,multiply(X,elt_multiply(beta,s_m)))),multiply(X,elt_multiply(beta,s_a))), sigma_obs));\n            } else if (as_bool(logical_eq(trend_indicator,1))) {\n\n                current_statement_begin__ = 119;\n                lp_accum__.add(normal_log<propto__>(y, add(elt_multiply(logistic_trend(k,m,delta,t,cap,A,t_change,S, pstream__),add(1,multiply(X,elt_multiply(beta,s_m)))),multiply(X,elt_multiply(beta,s_a))), sigma_obs));\n            }\n\n        } catch (const std::exception& e) {\n            stan::lang::rethrow_located(e, current_statement_begin__, prog_reader__());\n            // Next line prevents compiler griping about no return\n            throw std::runtime_error(\"*** IF YOU SEE THIS, PLEASE REPORT A BUG ***\");\n        }\n\n        lp_accum__.add(lp__);\n        return lp_accum__.sum();\n\n    } // log_prob()\n\n    template <bool propto, bool jacobian, typename T_>\n    T_ log_prob(Eigen::Matrix<T_,Eigen::Dynamic,1>& params_r,\n               std::ostream* pstream = 0) const {\n      std::vector<T_> vec_params_r;\n      vec_params_r.reserve(params_r.size());\n      for (int i = 0; i < params_r.size(); ++i)\n        vec_params_r.push_back(params_r(i));\n      std::vector<int> vec_params_i;\n      return log_prob<propto,jacobian,T_>(vec_params_r, vec_params_i, pstream);\n    }\n\n\n    void get_param_names(std::vector<std::string>& names__) const {\n        names__.resize(0);\n        names__.push_back(\"k\");\n        names__.push_back(\"m\");\n        names__.push_back(\"delta\");\n        names__.push_back(\"sigma_obs\");\n        names__.push_back(\"beta\");\n    }\n\n\n    void get_dims(std::vector<std::vector<size_t> >& dimss__) const {\n        dimss__.resize(0);\n        std::vector<size_t> dims__;\n        dims__.resize(0);\n        dimss__.push_back(dims__);\n        dims__.resize(0);\n        dimss__.push_back(dims__);\n        dims__.resize(0);\n        dims__.push_back(S);\n        dimss__.push_back(dims__);\n        dims__.resize(0);\n        dimss__.push_back(dims__);\n        dims__.resize(0);\n        dims__.push_back(K);\n        dimss__.push_back(dims__);\n    }\n\n    template <typename RNG>\n    void write_array(RNG& base_rng__,\n                     std::vector<double>& params_r__,\n                     std::vector<int>& params_i__,\n                     std::vector<double>& vars__,\n                     bool include_tparams__ = true,\n                     bool include_gqs__ = true,\n                     std::ostream* pstream__ = 0) const {\n        typedef double local_scalar_t__;\n\n        vars__.resize(0);\n        stan::io::reader<local_scalar_t__> in__(params_r__,params_i__);\n        static const char* function__ = \"model15cd647f12245_prophet_namespace::write_array\";\n        (void) function__;  // dummy to suppress unused var warning\n        // read-transform, write parameters\n        double k = in__.scalar_constrain();\n        double m = in__.scalar_constrain();\n        vector_d delta = in__.vector_constrain(S);\n        double sigma_obs = in__.scalar_lb_constrain(0);\n        vector_d beta = in__.vector_constrain(K);\n        vars__.push_back(k);\n        vars__.push_back(m);\n            for (int k_0__ = 0; k_0__ < S; ++k_0__) {\n            vars__.push_back(delta[k_0__]);\n            }\n        vars__.push_back(sigma_obs);\n            for (int k_0__ = 0; k_0__ < K; ++k_0__) {\n            vars__.push_back(beta[k_0__]);\n            }\n\n        // declare and define transformed parameters\n        double lp__ = 0.0;\n        (void) lp__;  // dummy to suppress unused var warning\n        stan::math::accumulator<double> lp_accum__;\n\n        local_scalar_t__ DUMMY_VAR__(std::numeric_limits<double>::quiet_NaN());\n        (void) DUMMY_VAR__;  // suppress unused var warning\n\n        try {\n\n\n\n            // validate transformed parameters\n\n            // write transformed parameters\n            if (include_tparams__) {\n            }\n            if (!include_gqs__) return;\n            // declare and define generated quantities\n\n\n\n            // validate generated quantities\n\n            // write generated quantities\n        } catch (const std::exception& e) {\n            stan::lang::rethrow_located(e, current_statement_begin__, prog_reader__());\n            // Next line prevents compiler griping about no return\n            throw std::runtime_error(\"*** IF YOU SEE THIS, PLEASE REPORT A BUG ***\");\n        }\n    }\n\n    template <typename RNG>\n    void write_array(RNG& base_rng,\n                     Eigen::Matrix<double,Eigen::Dynamic,1>& params_r,\n                     Eigen::Matrix<double,Eigen::Dynamic,1>& vars,\n                     bool include_tparams = true,\n                     bool include_gqs = true,\n                     std::ostream* pstream = 0) const {\n      std::vector<double> params_r_vec(params_r.size());\n      for (int i = 0; i < params_r.size(); ++i)\n        params_r_vec[i] = params_r(i);\n      std::vector<double> vars_vec;\n      std::vector<int> params_i_vec;\n      write_array(base_rng,params_r_vec,params_i_vec,vars_vec,include_tparams,include_gqs,pstream);\n      vars.resize(vars_vec.size());\n      for (int i = 0; i < vars.size(); ++i)\n        vars(i) = vars_vec[i];\n    }\n\n    static std::string model_name() {\n        return \"model15cd647f12245_prophet\";\n    }\n\n\n    void constrained_param_names(std::vector<std::string>& param_names__,\n                                 bool include_tparams__ = true,\n                                 bool include_gqs__ = true) const {\n        std::stringstream param_name_stream__;\n        param_name_stream__.str(std::string());\n        param_name_stream__ << \"k\";\n        param_names__.push_back(param_name_stream__.str());\n        param_name_stream__.str(std::string());\n        param_name_stream__ << \"m\";\n        param_names__.push_back(param_name_stream__.str());\n        for (int k_0__ = 1; k_0__ <= S; ++k_0__) {\n            param_name_stream__.str(std::string());\n            param_name_stream__ << \"delta\" << '.' << k_0__;\n            param_names__.push_back(param_name_stream__.str());\n        }\n        param_name_stream__.str(std::string());\n        param_name_stream__ << \"sigma_obs\";\n        param_names__.push_back(param_name_stream__.str());\n        for (int k_0__ = 1; k_0__ <= K; ++k_0__) {\n            param_name_stream__.str(std::string());\n            param_name_stream__ << \"beta\" << '.' << k_0__;\n            param_names__.push_back(param_name_stream__.str());\n        }\n\n        if (!include_gqs__ && !include_tparams__) return;\n\n        if (include_tparams__) {\n        }\n\n\n        if (!include_gqs__) return;\n    }\n\n\n    void unconstrained_param_names(std::vector<std::string>& param_names__,\n                                   bool include_tparams__ = true,\n                                   bool include_gqs__ = true) const {\n        std::stringstream param_name_stream__;\n        param_name_stream__.str(std::string());\n        param_name_stream__ << \"k\";\n        param_names__.push_back(param_name_stream__.str());\n        param_name_stream__.str(std::string());\n        param_name_stream__ << \"m\";\n        param_names__.push_back(param_name_stream__.str());\n        for (int k_0__ = 1; k_0__ <= S; ++k_0__) {\n            param_name_stream__.str(std::string());\n            param_name_stream__ << \"delta\" << '.' << k_0__;\n            param_names__.push_back(param_name_stream__.str());\n        }\n        param_name_stream__.str(std::string());\n        param_name_stream__ << \"sigma_obs\";\n        param_names__.push_back(param_name_stream__.str());\n        for (int k_0__ = 1; k_0__ <= K; ++k_0__) {\n            param_name_stream__.str(std::string());\n            param_name_stream__ << \"beta\" << '.' << k_0__;\n            param_names__.push_back(param_name_stream__.str());\n        }\n\n        if (!include_gqs__ && !include_tparams__) return;\n\n        if (include_tparams__) {\n        }\n\n\n        if (!include_gqs__) return;\n    }\n\n}; // model\n\n}\n\ntypedef model15cd647f12245_prophet_namespace::model15cd647f12245_prophet stan_model;\n\n"), 
       mk_cppmodule = function (object) 
       {
           prep_call_sampler(object)
           model_cppname <- object@model_cpp$model_cppname
           mod <- get("module", envir = object@[email protected], 
               inherits = FALSE)
           eval(call("$", mod, paste("stan_fit4", model_cppname, 
               sep = "")))
       }, dso = new("cxxdso", sig = list(file15cd618f2e0aa = character(0)), 
           dso_saved = TRUE, dso_filename = "file15cd618f2e0aa", 
           modulename = "stan_fit4model15cd647f12245_prophet_mod", 
           system = "x86_64, darwin15.6.0", cxxflags = "CXXFLAGS = -Wall -g -O2 $(LTO)", 
           .CXXDSOMISC = <environment>)), data = list(T = 50L, K = 26L, 
       S = 25L, y = c(0.274753391875146, 0.265794018096522, 0.338336557352391, 
# (...)

And after loads of numbers...

# (...)
       0.623489801858333, -0.900968867901821, -0.222520933955954, 
       1, -0.222520933952989, -0.900968867903141, 0.6234898018588, 
       0.623489801857301, -0.900968867900816, -0.222520933958213, 
       1, -0.222520933954277, -0.900968867902567, 0.623489801859833, 
       0.623489801861956, -0.900968867901389, -0.222520933956925, 
       1, -0.222520933955565, -0.900968867903573, 0.623489801858021, 
       0.623489801860923, -0.900968867901962, -0.222520933955637, 
       1, -0.222520933953306, -0.900968867902999, 0.623489801859054, 
       0.62348980185989, -0.900968867902535, -0.222520933957896, 
       1, -0.222520933954594, -0.900968867902426, 0.623489801860087, 
       0.623489801861702, -0.90096886790153, -0.222520933956608, 
       1, -0.222520933955882, -0.900968867901853, 0.623489801855432, 
       0.623489801860669, -0.974927912180744, 0.781831482470042, 
       -0.433883739114633, -2.44362079523015e-12, 0.433883739119036, 
       -0.781831482468552, 0.974927912183451, -0.974927912181994, 
       0.781831482464473, -0.433883739113141, 3.17603745655276e-12, 
       0.433883739120529, -0.781831482465048, 0.9749279121822, -0.974927912181626, 
       0.781831482467976, -0.433883739118204, -5.75621952003119e-12, 
       0.433883739115466, -0.781831482470618, 0.974927912182569, 
       -0.974927912181257, 0.781831482466944, -0.433883739110156, 
       -1.36561268248286e-13, 0.433883739123513, -0.781831482467114, 
       0.974927912181318, -0.974927912180889, 0.781831482470447, 
       -0.433883739115219, -1.79286063064881e-12, 0.43388373911845, 
       -0.781831482468146, 0.974927912183306, -0.974927912182139, 
       0.781831482464878, -0.433883739120283, -3.44915999304933e-12, 
       0.433883739119942, -0.781831482469179, 0.974927912182055, 
       -0.974927912180152, 0.781831482468382, -0.433883739112235, 
       -5.10545935544985e-12, 0.433883739114879, -0.781831482470212, 
       0.974927912180805, -0.974927912181402, -0.222520933961045, 
       0.623489801856211, -0.900968867903828, 1, -0.900968867901707, 
       0.623489801858078, -0.222520933949187, -0.222520933955566, 
       0.623489801863194, -0.900968867904546, 1, -0.900968867900989, 
       0.623489801862472, -0.222520933954665, -0.222520933957181, 
       0.623489801858801, -0.900968867902108, 1, -0.900968867903427, 
       0.623489801855488, -0.222520933953051, -0.222520933958796, 
       0.623489801860096, -0.900968867905984, 1, -0.900968867899551, 
       0.623489801859882, -0.222520933958529, -0.22252093396041, 
       0.623489801855702, -0.900968867903545, 1, -0.90096886790199, 
       0.623489801858587, -0.222520933949821, -0.222520933954932, 
       0.623489801862686, -0.900968867901107, 1, -0.900968867901271, 
       0.623489801857292, -0.2225209339553, -0.22252093396364, 0.623489801858292, 
       -0.900968867904983, 1, -0.900968867903709, 0.623489801855997, 
       -0.222520933960779, -0.222520933958161), sigmas = c(10, 10, 
       10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 
       10, 10, 10, 10, 10, 10, 10, 10, 10), tau = 0.05, trend_indicator = 0, 
       s_a = c(1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
       1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L), s_m = c(0, 
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 
       0, 0, 0, 0, 0, 0), cap = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 
       0, 0)), init = function () 
   {
       list(k = kinit[1], m = kinit[2], delta = array(rep(0, length(m$changepoints.t))), 
           beta = array(rep(0, ncol(seasonal.features))), sigma_obs = 1)
   }, algorithm = "Newton", iter = 10000, as_vector = FALSE)
2: do.call(rstan::optimizing, args)
1: fit.prophet(m, prop[1:50, ])

[laresbernardo]

I also get this error the first time I run after reseting my R session: Error: vector memory exhausted (limit reached?)
And when traceback() is run:

12: Module(module, mustStart = TRUE)
11: .getModulePointer(x)
10: new("Module", .xData = <environment>)$stan_fit4model1600c2a310a8b_prophet
9: new("Module", .xData = <environment>)$stan_fit4model1600c2a310a8b_prophet
8: eval(call("$", mod, paste("stan_fit4", model_cppname, sep = "")))
7: eval(call("$", mod, paste("stan_fit4", model_cppname, sep = "")))
6: object@mk_cppmodule(object)
5: .local(object, ...)
4: (new("nonstandardGenericFunction", .Data = function (object, 
       ...) 
   {
       if (is.sparc()) {
           msg <- "optimizing() will likely crash on SPARC."
           if (interactive()) 
               stop(msg, " Run in batch mode to test.")
           else message(msg)
       }
       standardGeneric("optimizing")
   }, generic = "optimizing", package = "rstan", group = list(), 
       valueClass = character(0), signature = "object", default = NULL, 
       skeleton = (function (object, ...) 
       stop("invalid call in method dispatch to 'optimizing' (no default method)", 
           domain = NA))(object, ...)))(object = new("stanmodel", 
       model_name = "prophet", model_code = "functions {\n  matrix get_changepoint_matrix(vector t, vector t_change, int T, int S) {\n    // Assumes t and t_change are sorted.\n    matrix[T, S] A;\n    row_vector[S] a_row;\n    int cp_idx;\n\n    // Start with an empty matrix.
# (...)

The second time I try to run exactly the same line, I get the previous error results and traceback! 👎

[bletham]

OK so the error is being thrown in rstan. The "vector memory exhausted" seems to be related to R having a maximum memory allocation limit (https://stackoverflow.com/questions/51248293/error-vector-memory-exhausted-limit-reached-r-3-5-0-macos/52612848). Maybe that changed in R3.6? Could you see what it is?

Stan does require a surprising amount of RAM to compile and fit models. Could you see if this code runs? That would help to identify if it is just the usual rstan RAM usage causing the issue, or if it may be actually something in how it is being called by prophet:

library(rstan)
model_code <- 'parameters {real y;} model {y ~ normal(0,1);}'
fit <- stan(model_code=model_code)  # this will take a minute
mean(extract(fit)$y)  # should be close to 0

[laresbernardo]

This is what I get when I run your code (tried twice, same result):

> fit <- stan(model_code=model_code)  # this will take a minute
Error: vector memory exhausted (limit reached?)
> mean(extract(fit)$y)  # should be close to 0
Error in extract(fit) : object 'fit' not found

And about the memory limit, it doesn't make much sense. In this test I used a 50 row dataset to forecast, and I'm on a 16GB RAM laptop, with a fresh RStudio session (nothing in the Global environment other than the data frame).

Thanks for your help @bletham Really appreciate it!

[bletham]

I'm not really sure what's happening here. There are a lot of stackoverflow posts addressing the "Error: vector memory exhausted (limit reached?)" error where people suggest various things, which seem to work for some but not others. It looks like there are some additional considerations for RStudio.

rstan does need about 4GB of RAM to compile the model, regardless of the size of the data. A 16GB laptop would certainly have that, but it does leave me wondering if there is something set in R that is limiting it.

Besides trying the tips scattered around stackoverflow, since this is an issue generally with rstan I'd suggest asking in the rstan issues and they might have better experience with / knowledge of this issue.

[djacobs7]

I had a similar issue. I had to modify my ~.R/Makevars file to use clang, like so:

~.R/Makevars

CC=gcc
CXX=g++
CXX11=g++
CXX14=g++
cxx18=g++
cxx1X=g++
LDFLAGS=-L/usr/lib 

Then I did install.packages('prophet') and the problem went away.