Cmdstanr: Possible source of elusive Mac OS segfaults

[wlandau]

Bug and reprex

As I explain below, I suspect this issue is not actually a bug in cmdstanr or CmdStan, and it could come from how processx behaves on Mac OS. The reason I am posting here is to follow up on #295, https://github.com/stan-dev/cmdstanr/pull/254#issuecomment-665249546, and https://github.com/stan-dev/cmdstanr/pull/238#issuecomment-653581383, and because I can only reproduce it with cmdstanr.

The following reprex frequently segfaults on my local Macbook, and I am experiencing similar segfaults on Mac OS GitHub Actions jobs.

library(cmdstanr)
code <- "data {
  int <lower = 1> n;
  vector[n] x;
  vector[n] y;
}
parameters {
  real beta;
}
model {
  y ~ normal(x * beta, 1);
  beta ~ normal(0, 1);
}
generated quantities {
  real y_rep[n] = normal_rng(x * beta, 1);
}"
file <- write_stan_file(code)
n <- 1e3
data <- list(n = n, x = rnorm(n), y = rnorm(n))
model <- cmdstan_model(file)
for (i in seq_len(100)) {
  fit <- model$sample(
    data = data,
    iter_sampling = 1e3,
    refresh = 0
  )
}

Output:

 *** caught segfault ***
address 0x7fd5458c5fff, cause 'memory not mapped'

Traceback:
 1: .Call(list(name = "processx_connection_read_lines", address = <pointer: 0x7fd54751c220>,     dll = list(name = "processx", path = "/Library/Frameworks/R.framework/Versions/4.0/Resources/library/processx/libs/processx.so",         dynamicLookup = FALSE, handle = <pointer: 0x7fd54751a5c0>,         info = <pointer: 0x7fd54946f6c0>), numParameters = 2L),     <pointer: 0x7fd52e01ca50>, -1)
 2: do.call(".Call", list(.NAME, ...))
 3: withCallingHandlers(do.call(".Call", list(.NAME, ...)), error = function(e) {    e$`_nframe` <- nframe    e$call <- call    if (inherits(e, "simpleError")) {        class(e) <- c("c_error", "rlib_error", "error", "condition")    }    e$`_ignore` <- list(c(nframe + 1L, sys.nframe() + 1L))    throw(e)})
 4: rethrow_call(c_processx_connection_read_lines, con, n)
 5: process_read_output_lines(self, private, n)
 6: procs$get_proc(chain_id)$read_output_lines()
 7: private$run_sample_()
 8: runset$run_cmdstan()
 9: model$sample(data = data, iter_sampling = 1000, refresh = 0)
10: eval(expr, envir, enclos)
11: eval(expr, envir, enclos)
12: withVisible(eval(expr, envir, enclos))
13: withCallingHandlers(withVisible(eval(expr, envir, enclos)), warning = wHandler,     error = eHandler, message = mHandler)
14: handle(ev <- withCallingHandlers(withVisible(eval(expr, envir,     enclos)), warning = wHandler, error = eHandler, message = mHandler))
15: timing_fn(handle(ev <- withCallingHandlers(withVisible(eval(expr,     envir, enclos)), warning = wHandler, error = eHandler, message = mHandler)))
16: evaluate_call(expr, parsed$src[[i]], envir = envir, enclos = enclos,     debug = debug, last = i == length(out), use_try = stop_on_error !=         2L, keep_warning = keep_warning, keep_message = keep_message,     output_handler = output_handler, include_timing = include_timing)
17: evaluate::evaluate(...)
18: evaluate(code, envir = env, new_device = FALSE, keep_warning = !isFALSE(options$warning),     keep_message = !isFALSE(options$message), stop_on_error = if (options$error &&         options$include) 0L else 2L, output_handler = knit_handlers(options$render,         options))
19: in_dir(input_dir(), evaluate(code, envir = env, new_device = FALSE,     keep_warning = !isFALSE(options$warning), keep_message = !isFALSE(options$message),     stop_on_error = if (options$error && options$include) 0L else 2L,     output_handler = knit_handlers(options$render, options)))
20: block_exec(params)
21: call_block(x)
22: process_group.block(group)
23: process_group(group)
24: withCallingHandlers(if (tangle) process_tangle(group) else process_group(group),     error = function(e) {        setwd(wd)        cat(res, sep = "\n", file = output %n% "")        message("Quitting from lines ", paste(current_lines(i),             collapse = "-"), " (", knit_concord$get("infile"),             ") ")    })
25: process_file(text, output)
26: knitr::knit(knit_input, knit_output, envir = envir, quiet = quiet)
27: rmarkdown::render("rpx.Rmd")
An irrecoverable exception occurred. R is aborting now ...
Segmentation fault: 11

The segfaults go away when I change this line:

https://github.com/stan-dev/cmdstanr/blob/0fee21fb1bb31ac43f78badc17172b5e86925804/R/run.R#L277

to this:

output <- ""

@gaborcsardi, do you have any suggestions for troubleshooting? I would love to be more helpful, but I am not sure how to enable core dumps on Mac OS.

Expected behavior

No segfault.

Operating system

Mac OS 10.15.7

CmdStanR version number

0.2.1

Additional context

> cmdstan_version()
[1] "2.25.0"
> sessionInfo()
R version 4.0.3 (2020-10-10)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Catalina 10.15.7

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] cmdstanr_0.2.1

loaded via a namespace (and not attached):
 [1] prettyunits_1.1.1 ps_1.4.0          fansi_0.4.1       digest_0.6.27    
 [5] withr_2.3.0       rprojroot_2.0.2   crayon_1.3.4      assertthat_0.2.1 
 [9] R6_2.5.0          evaluate_0.14     rlang_0.4.9       cli_2.2.0        
[13] curl_4.3          remotes_2.2.0     rstudioapi_0.13   callr_3.5.1      
[17] rmarkdown_2.5     tools_4.0.3       glue_1.4.2        xfun_0.19        
[21] yaml_2.2.1        compiler_4.0.3    processx_3.4.4    pkgbuild_1.1.0   
[25] htmltools_0.5.0   knitr_1.30       

[rok-cesnovar]

Thank you so much! This has been bugging me for a long time, originally I thought it was a vroom or a Github Actions thing because there were no reports of anyone experiencing this locally. The code you posted will help me track this down.

I think it's a problem with processx yeah. Possibly trying to read output lines when the process finished or something segfaults. Hopefully, I can reproduce this constantly on GA with your example and that will help me fix this.

[gaborcsardi]

I can reproduce this, it seems like a processx bug indeed. See https://github.com/r-lib/processx/issues/281

[rok-cesnovar]

Thanks to both! I will kindly ask you to test a patch once I have it ready. Will try to circumvent the processxbug until they sort it out.

[gaborcsardi]

Oh, I think I found the bug and will fix it in a minute.

[gaborcsardi]

Fixed in https://github.com/r-lib/processx/pull/282 Will have a release soon.

[rok-cesnovar]

Thank you so much @gaborcsardi !!

[gaborcsardi]

Let's wait a day to confirm that it is indeed fixed, although I am fairly sure that it is. Then I can do a processx release tomorrow, and you can depend on the new version.

[rok-cesnovar]

I am going to have our Github Actions job depend on the Github version of processx and then try to re-run it as often as possible. It typically failed 1/5 at least.

[wlandau]

Fixed in r-lib/processx#282 Will have a release soon.

Thank you @gaborcsardi! That patch makes such a difference.

Let's wait a day to confirm that it is indeed fixed, although I am fairly sure that it is. Then I can do a processx release tomorrow, and you can depend on the new version.

I am currently farming out a bunch of processes to test everything that segfaulted for me locally in the past. No problems so far.

[wlandau]

My local Macbook spent the last half hour looping over 3 different situations that previously gave me segfaults, and there were no errors.

[rok-cesnovar]

Also passed 6 runs on master, but will run at least a dozen more. I would conaider this issue to be resolved, but will close the issue once the new processx version is released.

Thanks again @wlandau @gaborcsardi !

[jgabry]

Thanks again @wlandau @gaborcsardi !

Indeed, thank you! This had definitely been bugging us (and confusing us) for a while.