Julia: excising Rmath

[StefanKarpinski]

I think we'd like to get rid of Rmath as something we ship with Julia. I know we depend on it in various places for random number generation and some statistical distribution functions but I have a hard time figuring out where, how and what the status of weaning ourselves off of it is. Let's make this a central point for tracking how this is going.

[johnmyleswhite]

For Distributions, @simonbyrne knows best.
For HypothesisTests, @simonster knows best.

[andreasnoack]

This one is relevant https://github.com/JuliaStats/Distributions.jl/pull/138

[simonster]

In HypothesisTests the only thing we use Rmath for is the Wilcoxon tests (Mann-Whitney U and signed rank). The relevant bug is JuliaStats/HypothesisTests.jl#7

[ViralBShah]

The plan is to do so in 0.4.

[JeffBezanson]

Bump.

[simonbyrne]

Rmath certainly won't be done with it in the immediate future. I wouldn't be opposed to making it a binary package if that would make building easier?

[ViralBShah]

@lindahua has also asked elsewhere (can't recollect) to hold on the Rmath removal from Base. We can move it to BinDeps, and although the experience is not the best, perhaps we have to bite the bullet.

[tkelman]

Was there a decision on whether or not to keep patching rmath to use dsfmt?

[ViralBShah]

I believe the state is no longer shared. I should revisit this issue in light of the recent RNG work.

[ViralBShah]

Actually, that statement was not true. The RNGs are still synced. Of course, since Rmath-julia is its own repo, we can easily still excise it from Base, so long as we can make it effortless to get. We may also want to hack Rmath-julia further so that it can support some more of our features, like passing state to the underlying generators, etc.

julia> srand(1);

julia> [rand(), rand()]
2-element Array{Float64,1}:
 0.236033
 0.346517

julia> srand(1);

julia> using Distributions

julia> [rand(Uniform()), rand(Uniform())]
2-element Array{Float64,1}:
 0.236033
 0.346517

Cc: @rfourquet @dmbates also

[ViralBShah]

@simonbyrne already has https://github.com/JuliaStats/RmathDist.jl

What we need is BinDeps support and binaries for Windows, Mac, Ubuntu/Debian, and Fedora/RHEL/CentOS.

Cc: @staticfloat @nalimilan @sebastien-villemot

[tkelman]

https://sourceforge.net/projects/juliadeps-win/files/libRmath-julia.7z

[tkelman]

Bumping this to 0.5 since we probably won't be backporting any removals, and there's a workaround for now of USE_GPL_LIBS=0 on release-0.4.

[ViralBShah]

Yup. But we should get this done early in the 0.5 cycle now.

[Keno]

@simonbyrne do you think we can get this done in 0.5?

[simonbyrne]

I think so, though I'll need some help getting bindeps set up

[staticfloat]

reporting for duty. If you can give me a TODO list, I'll see what I can do.

[tkelman]

The above Windows binary file is also reposted at https://bintray.com/tkelman/generic/download_file?file_path=libRmath-julia.7z for a "not sourceforge" option. If it needs rebuilding for any reason, do ping me.

[staticfloat]

@simonbyrne should we move forward with creating a BinDeps package for Rmath?

[simonbyrne]

I would like to fix https://github.com/JuliaStats/Distributions.jl/issues/294 first, as then we don't need to worry about samplers hooking into the Julia RNG. I have some code, I'll try to get it cleaned up next week.

[tkelman]

let's get a build scaffold for rmath up and running in a package first, if we're going to need that no matter what we do about which RNG it uses

[simonbyrne]

@tkelman What's the best way to do this? Is it sufficient to just dump all of https://github.com/JuliaLang/Rmath-julia in the deps directory, and have Pkg.build call make? Presumably that will require a C compiler, no?

[tkelman]

Easiest to have BinDeps download a source tarball from a tag of the existing repo. On Linux we generally do just build from source, or try to use a system package manager if the unmodified librmath from R ends up okay to use here. There are ways to build generic Linux binaries that I can show you how to do, but I'd start with the source build as a fallback and set up binaries for Windows and OSX first where people are less likely to have compilers installed. The simplest example packages to look at and copy how they use bindeps are probably JuliaOpt/SCS.jl, JuliaOpt/ECOS.jl, or JuliaWeb/MbedTLS.jl.

edit: staticfloat/Nettle.jl is also a good one to look at. Can technically just extract our patched rmath from 0.4 binaries if you'd like, that's not the best for easy reproducibility and making future modifications though.

[ViralBShah]

@simonbyrne If you can remove the dependence on Rmath RNGs, that will be best. How close are we?

[tkelman]

I'll do the BinDeps piece here as the example I use in my JuliaCon workshop, Simon we can talk on the details of RNG's and which sources to use.

[tkelman]

Is it worth conserving a little bit of METADATA namespace and un-deprecating (really just replacing) https://github.com/dmbates/Rmath.jl with the new version of this? https://github.com/JuliaStats/RmathDist.jl looks a little newer, but isn't registered?

[simonbyrne]

Calling it RMath.jl is fine (it can be a binary-only package).

RmathDist.jl was only used by myself, and could be safely deleted.

By the way, the instructions for building Rmath from upstream are here:
https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Standalone-Mathlib

[tkelman]

They allow you to build the math lib by itself, but they don't allow you to configure it by itself. And they don't use configure for their Windows build, you have to do some fiddly manual makefile modification. So I think the simple makefiles we already have in Rmath-julia are worth keeping, since configuring all of R requires a bunch of other libraries to have devel packages installed which may often be missing. I'll proceed with using the rmath-julia source but not going out of our way to link to dsfmt for now.

[simonbyrne]

That seems like a good idea