Cmdstanr: Detect the use of the M1 ARM-based CPU and suggest adding CXX

[rok-cesnovar]

A user in https://github.com/stan-dev/math/pull/2208 reported that cmdstanr works fine on the new M1 macs, but required adding a CXX makefile flag.

cmdstan_model('test.stan', cpp_options=list(CXX="arch -arch arm64e clang++"))

We could probably detect the M1 CPU and could add the flag automatically on install_cmdstan() or rebuild_cmdstan(). Should we rather instead just suggest the user to add these to make/local?

This won't be required for 2.26 but will be for earlier versions.

[rok-cesnovar]

We won't be able to detect the CPU architecture in R as R will run under emulation of x86 until there is a native R version (the target for that is April 2021). We may be able to run a command with processx that would help us determine that.

[nicokist]

This might be a candidate

$ /usr/sbin/sysctl -n hw.machine
arm64

[rok-cesnovar]

Thanks Nico! @jgabry I am guessing that command returns x86_64 for an Intel Mac?

@nicokist do the models runs lower/faster compared to a machine you used previously?

[nicokist]

Interestingly that reports x86_64 even on an Arm Mac when running Rosetta, which would be the case if it were called from R!

It might be better to use something like the below, which returns hello on an Arm Mac and presumably gives an error on a Big Sur Intel Mac.

arch -arch arm64e echo 'hello'

Regarding speed, I'm just about to start running my models using cmdstanr, I've always used rstan previously. Is there a favored benchmark?

[rok-cesnovar]

So

processx::run("/usr/sbin/sysctl", args = c("-n", "hw.machine"))

returns x86_64? That is unfortunate.

Is there a favored benchmark?

I am guessing any model you have available that runs for at least a minute.

[nicokist]

processx::run("/usr/sbin/sysctl", args = c("-n", "sysctl.proc_translated"))

Might be an alternative, it returns 1 in Rosetta 2 and 0 otherwise.

One of my gnarlier and unfinished models takes ~125 seconds on my new M1 MacBook Pro and 489.6 seconds on my Mid 2013 1.3 GHz MacBook Air, about a 4x speedup!

I'll do more testing this week.

[rok-cesnovar]

Might be an alternative, it returns 1 in Rosetta 2 and 0 otherwise.

Nice!

about a 4x speedup!

Even nicer! Thanks again for the feedback and all the information.

[nicokist]

And about 2x as fast as under Rosetta 2:
mid-2013 1.3Ghz Intel MacBook Air: ~490 seconds
M1 MacBook Pro under Rosetta 2: ~240 seconds
M1 MacBook Pro native Arm: ~125 seconds.

[rok-cesnovar]

@nicokist

when you have a few moments can you first try

remotes::install_github("stan-dev/cmdstanr@m1cxx")
and then try install_cmdstan() to install a fresh cmdstan.

If that does not work try remotes::install_github("stan-dev/cmdstanr@m1")

Thanks!

If possible also please post the entire output of processx::run("/usr/sbin/sysctl", args = c("-n", "sysctl.proc_translated"))

@jgabry can you run processx::run("/usr/sbin/sysctl", args = c("-n", "sysctl.proc_translated")) as well so I can compare?
I am not exactly sure if 0/1 is printed to stdout or if its a return code.

[jgabry]

@jgabry can you run processx::run("/usr/sbin/sysctl", args = c("-n", "sysctl.proc_translated")) as well so I can compare?
I am not exactly sure if 0/1 is printed to stdout or if its a return code.

I get an error:

> processx::run("/usr/sbin/sysctl", args = c("-n", "sysctl.proc_translated"))
Error in processx::run("/usr/sbin/sysctl", args = c("-n", "sysctl.proc_translated")) : 
  System command 'sysctl' failed, exit status: 1, stderr:
E> sysctl: unknown oid 'sysctl.proc_translated'

If I force it to not error then I get a return code of 1:

> x <- processx::run("/usr/sbin/sysctl", args = c("-n", "sysctl.proc_translated"), error_on_status = FALSE)
> x$status
[1] 1

[nicokist]

Thanks @rok-cesnovar! I think I was able to make it work, please see https://github.com/stan-dev/cmdstanr/pull/372

[mitzimorris]

is this relevant for CmdStan and its Makefile as well?

[nicokist]

Most changes in this patch are only necessary to compile CmdStan from a Rosetta 2 environment. I think the normal flow for CmdStan would be to compile it in Terminal.app running natively on arm64e, which already works if you use this version of stan-math or later (https://github.com/stan-dev/math/commit/92fce0218c9fb15fd405ef031f488cad05c5546b).

All of this is because the Rosetta 2 emulation is inherited by child processes; using arch -arch arm64e make overrides that.

[rok-cesnovar]

So for cmdstan the suggestion is either:

• use current develop
  or
• replace the TBB makefile with this: https://github.com/stan-dev/math/blob/develop/lib/tbb_2019_U8/build/macos.inc

We will have to do the additional arch -arch arm64e patch that @nicokist suggested for R for at least until there is a native ARM R build (I think the latest ETA is March or April). And even after that, there will be users running rosetta2 until CRAN builds native R packages I guess. So this gets a nice speedup for M1 users until all that is sorted out. So it's worth it.

I am guessing Python does not run via rosetta2 so cmdstanpy won't have that issue (apart from having to use develop not 2.25)? Though I have seen people struggling to get Tensorflow running on M1 chips so who knows.