Cmdstanr: Don't rely on rstan::read_stan_csv and rstan::stan_rdump

[jgabry]

As a placeholder while working on the rest of the package rstan::read_stan_csv() is currently used to read in the output files from MCMC (optimize and variational are implemented in cmdstanr) and rstan::stan_rdump() is used to write a data file readable by CmdStan (for all methods). Eventually these should be removed to get rid of the dependency on RStan.

[rok-cesnovar]

Update: We are only relying on read_stan_csv now.

[rok-cesnovar]

@jgabry now that the main parts of the parallel this is the next issue I would like to tackle. A few questions before I start, so you wont have as much work fixing up my code later :) :

• do we just want the extract and names functionality we use here? https://github.com/stan-dev/cmdstanr/blob/85c39cbb2db2922879fb35af30b0c299ff44e985/R/fit.R#L94
  Or do we want a similar stanfit object as the rstan function returns?

• should I just implement this as utils functions or make a class CmdstanCSV? I really dont have a preference here.

[mitzimorris]

in CmdStanPy these are utils functions. hope we can get CmdStan's I/O better, because the stan_csv files differ depending on the method in a bad way.

[jgabry]

@rok-cesnovar Cool, it would be great to get rid of this dependency! To answer your questions:

I think utils is fine for now and also sounds like that's what @mitzimorris is doing. once this exists we can change easily if we want.

In terms of what we need to implement, thankfully I don't think we need anything like a stanfit object from rstan. We can keep it much simpler than that. I think a new function read_sample_csv() could do something like this:

1) read output in efficiently
2) create three separate objects (warmup draws, post-warmup draws, sampler param draws). For now I think each of these objects can be a 3-D array (iters x chains x variables) unless there's an obviously better way to store them in terms of memory/speed.
3) return a list of these three objects

So a signature and return type like this maybe:

read_sample_csv <- function(output_files) {
  # read in output and create the 3-D arrays, then return:
  list(
    post_warmup = post_warmup_draws_array,
    warmup = warmup_draws_array, # if warmup wasn't saved this can be NULL
    sampler = sampler_param_draws_array # contains things like divergent__, treedepth__, etc. 
  )
}

And then we can replace these lines:

stanfit <- rstan::read_stan_csv(self$output_files())
private$draws_ <- rstan::extract(stanfit, permuted = FALSE, inc_warmup = FALSE) 
private$sampler_params_ <- rstan::get_sampler_params(stanfit, inc_warmup = FALSE)

with something like this:

all_draws <- read_sample_csv(self$output_files()) 
private$draws_ <- all_draws$post_warmup
private$warmup_draws_ <- all_draws$warmup
private$sampler_params_ <- all_draws$sampler

What do you think?

If possible (although this could be a separate issue/PR), I think it would be nice to also grab things like the stepsize and inverse mass matrix that CmdStan writes to the csv after adaptation. We could then provide a method for the user to access these in case they want to pass them in to $sample() at some point.