Cmdstanr: MPI execution

[rok-cesnovar]

Another one of @avehtari great idea, supporting MPI execution. Basically, support the equivalent of

mpiexec -n 4 --tag-output ./radon sample data file=radon.data.R

modelc1 = cmdstan_model("bernoulli_logit_glm_rhs.stan", quiet = FALSE, mpi = TRUE)
fit1 = modelc1$sample(data = data, mpi = TRUE, n = 4 )`

The API is very much up for debate.

[rok-cesnovar]

Also see https://discourse.mc-stan.org/t/cross-chain-warmup-adaptation-using-mpi/12912/15

[wds15]

Maybe it's worth to think for a moment of how we specify MPI processes and threads to be used. I don't really have a good idea atm.

[wds15]

One more thought: Traditionally MPI is used in clusters and you do send your program into a queue with a little script which fires off everything. These scripts typically are very simple shell scripts. From that angle it would not make a lot of sense to include MPI into CmdStanR as it does not really make sense to call out to R just to start the MPI program.

Should we recommend to users to run MPI on their desktops, then it does again make sense, of course - but for that purpose we have threads. So for me a R user will only ever use the CPUs on his own machine in which case you do not need MPI convenience features in CmdStanR.

This obviously changes if we are going to have MPI only features.

[rok-cesnovar]

I am seeing this primarily for:

• developer and early-adopter user oriented MPI testing (new MPI features that we need to test locally); basically what the reason leading to this issue
• users that want to test if everything is running with MPI as they thought it should before sending it to the cluster

I doubt we will have MPI-only features but that could be a use case also.

[wds15]

As a dev oriented feature it does make sense, indeed.

[yizhang-yiz]

I can confirm the following works:

> library("cmdstanr")
> set_cmdstan_path("/Users/yiz/Work/temp/cmdstan")
cmdstan_make_local(cpp_options=list("CXX"="mpicxx", "CC"="mpicxx", "STAN_MPI"="1","STANC2"="true","TBB_CXX_TYPE"="clang"))
[1] "STAN_MPI=1"         "STANC2=true"        "TBB_CXX_TYPE=clang"
[4] "CXX=mpicxx"         "CC=mpicxx"          "STAN_MPI=1"        
[7] "STANC2=true"        "TBB_CXX_TYPE=clang"
> rebuild_cmdstan(cores=4, quiet=FALSE)
...
--- CmdStan v2.23.0 built ---
> file <- file.path("~/Work/cmdstanr", "cmdstan_map_rect_tutorial", "logistic1.stan")
> mod <- cmdstan_model(file, quiet=FALSE)
...
--- Compiling, linking C++ code ---
mpicxx -std=c++1y -D_REENTRANT     -Wno-delete-non-virtual-dtor  -I stan/lib/stan_math/lib/tbb_2019_U8/include -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.72.0 -I stan/lib/stan_math/lib/sundials_5.2.0/include    -DBOOST_DISABLE_ASSERTS   -DSTAN_MPI   -c  -x c++ -o /var/folders/5d/_pkf036n7rzcbc2b5yl_zqsh0000gn/T/RtmpZXqLuc/model-134bf2838c7b5.o /var/folders/5d/_pkf036n7rzcbc2b5yl_zqsh0000gn/T/RtmpZXqLuc/model-134bf2838c7b5.hpp
mpicxx -std=c++1y -D_REENTRANT     -Wno-delete-non-virtual-dtor  -I stan/lib/stan_math/lib/tbb_2019_U8/include -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.3 -I stan/lib/stan_math/lib/boost_1.72.0 -I stan/lib/stan_math/lib/sundials_5.2.0/include    -DBOOST_DISABLE_ASSERTS   -DSTAN_MPI        -Wl,-L,"/Users/yiz/Work/temp/cmdstan/stan/lib/stan_math/lib/boost_1.72.0/stage/lib" -Wl,-rpath,"/Users/yiz/Work/temp/cmdstan/stan/lib/stan_math/lib/boost_1.72.0/stage/lib" -Wl,-L,"/Users/yiz/Work/temp/cmdstan/stan/lib/stan_math/lib/tbb" -Wl,-rpath,"/Users/yiz/Work/temp/cmdstan/stan/lib/stan_math/lib/tbb"  /var/folders/5d/_pkf036n7rzcbc2b5yl_zqsh0000gn/T/RtmpZXqLuc/model-134bf2838c7b5.o src/cmdstan/main.o         stan/lib/stan_math/lib/sundials_5.2.0/lib/libsundials_nvecserial.a stan/lib/stan_math/lib/sundials_5.2.0/lib/libsundials_cvodes.a stan/lib/stan_math/lib/sundials_5.2.0/lib/libsundials_idas.a stan/lib/stan_math/lib/sundials_5.2.0/lib/libsundials_kinsol.a stan/lib/stan_math/lib/boost_1.72.0/stage/lib/libboost_serialization.dylib stan/lib/stan_math/lib/boost_1.72.0/stage/lib/libboost_mpi.dylib stan/lib/stan_math/stan/math/prim/functor/mpi_cluster_inst.o stan/lib/stan_math/lib/tbb/libtbb.dylib stan/lib/stan_math/lib/tbb/libtbbmalloc.dylib stan/lib/stan_math/lib/tbb/libtbbmalloc_proxy.dylib -o /var/folders/5d/_pkf036n7rzcbc2b5yl_zqsh0000gn/T/RtmpZXqLuc/model-134bf2838c7b5
> 
> processx::run("mpiexec", c("-n", "4", "/var/folders/5d/_pkf036n7rzcbc2b5yl_zqsh0000gn/T/RtmpZXqLuc/model-134bf2838c7b5","sample", "data", "file=cmdstan_map_rect_tutorial/redcard_input.R","random", "seed=123"),echo=TRUE)
...
  Elapsed Time: 36.2808 seconds (Warm-up)
                37.3004 seconds (Sampling)
                73.5812 seconds (Total)

Sequential run:

> fit <- mod$sample(data="cmdstan_map_rect_tutorial/redcard_input.R",seed = 123, chains=1)

Chain 1 finished in 152.4 seconds.

[jgabry]

Nice, that's great!

We should be able to wrap this

processx::run("mpiexec", c("-n", "4", "/var/folders/5d/_pkf036n7rzcbc2b5yl_zqsh0000gn/T/RtmpZXqLuc/model-134bf2838c7b5","sample", "data", "file=cmdstan_map_rect_tutorial/redcard_input.R","random", "seed=123"),echo=TRUE)

up into a method (e.g. mod$mpi_sample() or mod$sample_mpi()) so that it's easy to run MPI from CmdStanR.

[rok-cesnovar]

Thanks @yizhang-yiz for the example code! Should make this easy to add as Jonah says.

If anyone has a preference on the name do let us know. Will try to add this this week.

So the plan is to add another method (name TBD) here https://github.com/stan-dev/cmdstanr/blob/master/R/model.R#L521 that mostly does the same thing but uses CmdstanMCMCProcsMPI or something like that. CmdstanMCMCProcsMPI will just inherit everything from CmdstanMCMCProcs except for the new_proc() function https://github.com/stan-dev/cmdstanr/blob/master/R/run.R#L390

I think that should work.

[rok-cesnovar]

I marked this for beta, as I think this won't be much work and we have covered most of the remaining beta issues. Planning to work on this next week. The biggest thing will probably be setting up a Github Actions agent to test MPI.

[jgabry]

@rok-cesnovar Ok cool. If you want to do this for the beta release that's great but it's also fine if this doesn't end up getting done for the beta release (so if you end up running into any issues we don't need to delay the beta release).

[jgabry]

@rok-cesnovar Should we push this off for 1.0? I'm ok either way, just don't feel any pressure to do it for the beta release. It's ok if you don't have time yet!

[rok-cesnovar]

Yes, other things were given priority on my TODO so this will not happen before mid-August I guess.