Reticulate: py_to_r does not convert scipy csr matrix

[scottgigante]

py_to_r works fine on a csc matrix, but not on a csr matrix.

> N <- 1000
> x <- sparseMatrix(
+     i = sample(N, N),
+     j = sample(N, N),
+     x = runif(N),
+     dims = c(N, N))
> result <- r_to_py(x)
> all(py_to_r(result) == x)
[1] TRUE
> all(py_to_r(result$tocsr()) == x)
Error: not-yet-implemented method for ==(<scipy.sparse.csr.csr_matrix>, <dgCMatrix>).
 ->>  Ask the package authors to implement the missing feature.

Package reticulate version 1.10.0.9003, scipy 1.1.0, R 3.5.1, Miniconda python 3.6.5

[scottgigante]

Thoughts on how to implement this: either an analogue to https://github.com/rstudio/reticulate/blob/4fa33ffff29c5a3c3147a65292797b2cf838feb8/R/conversion.R#L415 for csrmatrix and dgRMatrix, or simply call csrmatrix.tocsc() and then convert to dgCmatrix from there.

From the dgRMatrix documentation:

the RsparseMatrix class, the virtual class of all sparse compressed row-oriented matrices, with its methods. The dgCMatrix class (column compressed sparse) is really preferred.

[kevinushey]

The conversion of the sparse matrix to Python doesn't work for me at all:

library(Matrix)
library(reticulate)

N <- 1000
x <- sparseMatrix(
  i = sample(N, N),
  j = sample(N, N),
  x = runif(N),
  dims = c(N, N)
)

result <- r_to_py(x)
#> Error in py_call_impl(callable, dots$args, dots$keywords): TypeError: could not interpret data type
#> 
#> Detailed traceback: 
#>   File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/sparse/compressed.py", line 52, in __init__
#>     self.data = np.array(data, copy=copy, dtype=getdtype(dtype, data))
#>   File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/sparse/sputils.py", line 101, in getdtype
#>     raise TypeError("could not interpret data type")

^{Created on 2018-11-26 by the reprex package (v0.2.1)}

Not yet sure why that might be.

[terrytangyuan]

@scottgigante Thanks for bringing this up! CSC is more widely used so I implemented it first but supporting CSR is definitely of interest. A pull request for this feature is welcomed if you are interested!

@kevinushey Hmm it should work. Could you post your scipy + numpy versions?

[kevinushey]

Sorry, it looks like my numpy + scipy were quite out of date (I had inadvertently been using the system Python installation in this case)

library(Matrix)
library(reticulate)

N <- 1000
x <- sparseMatrix(
  i = sample(N, N),
  j = sample(N, N),
  x = runif(N),
  dims = c(N, N)
)

result <- r_to_py(x)
#> Error in py_call_impl(callable, dots$args, dots$keywords): TypeError: could not interpret data type
#> 
#> Detailed traceback: 
#>   File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/sparse/compressed.py", line 52, in __init__
#>     self.data = np.array(data, copy=copy, dtype=getdtype(dtype, data))
#>   File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/sparse/sputils.py", line 101, in getdtype
#>     raise TypeError("could not interpret data type")
py_config()
#> python:         /usr/bin/python
#> libpython:      /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config/libpython2.7.dylib
#> pythonhome:     /System/Library/Frameworks/Python.framework/Versions/2.7:/System/Library/Frameworks/Python.framework/Versions/2.7
#> version:        2.7.10 (default, Aug 17 2018, 17:41:52)  [GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)]
#> numpy:          /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy
#> numpy_version:  1.8.0
#> 
#> python versions found: 
#>  /usr/bin/python
#>  /usr/local/bin/python3
#>  /Users/kevin/.virtualenvs/test/bin/python
scipy <- import("scipy")
numpy <- import("numpy")
scipy[["__version__"]]
#> [1] "0.13.0b1"
numpy[["__version__"]]
#> [1] "1.8.0rc1"

^{Created on 2018-11-27 by the reprex package (v0.2.1)}

Upgrading to the latest numpy + scipy fixes this for me.

[terrytangyuan]

@kevinushey Great! We could probably identify the minimum working version of those and construct an informative message to the user that encounters this issue.

[scottgigante]

@terrytangyuan I'm putting together a PR to convert dgRMatrix and dgTMatrix. Is it desirable to have a generic conversation for base.spmatrix which converts via csc if any given sparse format doesn't have a dedicated conversion function?

Re: versions, I've had similar pathological experiences with old versions of sklearn. May be worth opening a separate issue for this.

[eddelbuettel]

Just FYI we went from "not a lot" to "quite a bit" in RcppArmadillo thanks the Google Summer of Code by @binxiangni -- there are plenty of tests, and a vignette. Plus, if memory, optional comparison with SciPy. And all of that used Matrix as the base case as one probably should. Maybe you can use that as a baseline.

[terrytangyuan]

Thanks @eddelbuettel! I did some Google search and found these vignettes (not personally familiar with RcppArmadillo yet):

http://gallery.rcpp.org/articles/armadillo-sparse-matrix-performance/
http://gallery.rcpp.org/articles/armadillo-sparse-matrix/

Indeed they could serve as good baselines. @scottgigante Also yes IMO regarding generic conversion for base.spmatrix for better user experience.

[eddelbuettel]

What I had in mind were

• this vignette
• these tests

[binxiangni]

Another test for sparse matrix under RcppArmadillo.