Umap: TypeError when fitting on >= 4096 samples

[cupidio]

Hi, I systematically get a TypeError when I run the fit() method on a dataset that is >=4096 samples. E.g.:

reducer = umap.UMAP() 
np.random.seed(0)
reducer.fit(np.random.rand(4096,16))

produces the error, while

reducer = umap.UMAP() 
np.random.seed(0)
reducer.fit(np.random.rand(4095,16))

runs fine.

I have installed umap with:

pip3 install umap-learn[plot]

This is the traceback of the error:

TypeError Traceback (most recent call last)
in
1 reducer = umap.UMAP(random_state=41,init='random', force_approximation_algorithm=True)
----> 2 reducer.fit(np.random.rand(4096,16))

~/anaconda2/envs/ecpackage3/lib/python3.8/site-packages/umap/umap_.py in fit(self, X, y)
1831 self._search_graph.data = _data
1832 self._search_graph = self._search_graph.maximum(
-> 1833 self._search_graph.transpose()
1834 ).tocsr()
1835

~/anaconda2/envs/ecpackage3/lib/python3.8/site-packages/scipy/sparse/lil.py in transpose(self, axes, copy)
435
436 def transpose(self, axes=None, copy=False):
--> 437 return self.tocsr(copy=copy).transpose(axes=axes, copy=False).tolil(copy=False)
438
439 transpose.__doc__ = spmatrix.transpose.__doc__

~/anaconda2/envs/ecpackage3/lib/python3.8/site-packages/scipy/sparse/lil.py in tocsr(self, copy)
460 indptr = np.empty(M + 1, dtype=idx_dtype)
461 indptr[0] = 0
--> 462 _csparsetools.lil_get_lengths(self.rows, indptr[1:])
463 np.cumsum(indptr, out=indptr)
464 nnz = indptr[-1]

_csparsetools.pyx in scipy.sparse._csparsetools.lil_get_lengths()

~/anaconda2/envs/ecpackage3/lib/python3.8/site-packages/scipy/sparse/_csparsetools.cpython-38-darwin.so in View.MemoryView.memoryview_cwrapper()

~/anaconda2/envs/ecpackage3/lib/python3.8/site-packages/scipy/sparse/_csparsetools.cpython-38-darwin.so in View.MemoryView.memoryview.__cinit__()

TypeError: a bytes-like object is required, not 'list'

[lmcinnes]

This could be due to changes in SciPy sparse? I'll try to look at it soon.

[cupidio]

Update: after installing pynndescent the error disappeared. Not sure if it is the expected behavior though...

[shivamsaboo17]

faced the same issue and installing pynndescent fixes it

[AlexandreAbreu8]

A similar problem occurs when running transform() method.
The error occurs if we fit a UMAP model on data with more than 4095 samples and then use the transform() method on different data.

How to reproduce the error:

import numpy as np
import umap

reducer = umap.UMAP()
reducer.fit(np.random.rand(4096, 4000))
reducer.transform(np.random.rand(500, 4000))

Meanwhile, if we fit the model with 4095 samples or less everything works just fine

import numpy as np
import umap

reducer = umap.UMAP()
reducer.fit(np.random.rand(4095, 4000))
reducer.transform(np.random.rand(500, 4000))

Note: I have pynndescent==0.4.7 installed and umap-learn==0.4.4.

### Traceback

ValueError Traceback (most recent call last)
in
----> 1 transformed_data = reducer.transform(np.random.rand(500, 4000))

~/.local/share/virtualenvs/Test-yifRmGUs/lib/python3.7/site-packages/umap/umap_.py in transform(self, X)
2067 dists = submatrix(dmat_shortened, indices_sorted, self._n_neighbors)
2068 elif _HAVE_PYNNDESCENT:
-> 2069 indices, dists = self._rp_forest.query(X, self.n_neighbors)
2070 elif self._sparse_data:
2071 if not scipy.sparse.issparse(X):

~/.local/share/virtualenvs/Test-yifRmGUs/lib/python3.7/site-packages/pynndescent/pynndescent_.py in query(self, query_data, k, epsilon)
1216 # query_data = check_array(query_data, dtype=np.float64, order='C')
1217 query_data = np.asarray(query_data).astype(np.float32, order="C")
-> 1218 self._init_search_graph()
1219 result = search(
1220 query_data,

~/.local/share/virtualenvs/Test-yifRmGUs/lib/python3.7/site-packages/pynndescent/pynndescent_.py in _init_search_graph(self)
1065
1066 # Get rid of any -1 index entries
-> 1067 self._search_graph = self._search_graph.tocsr()
1068 self._search_graph.data[self._search_graph.indices == -1] = 0.0
1069 self._search_graph.eliminate_zeros()

~/.local/share/virtualenvs/Test-yifRmGUs/lib/python3.7/site-packages/scipy/sparse/lil.py in tocsr(self, copy)
460 indptr = np.empty(M + 1, dtype=idx_dtype)
461 indptr[0] = 0
--> 462 _csparsetools.lil_get_lengths(self.rows, indptr[1:])
463 np.cumsum(indptr, out=indptr)
464 nnz = indptr[-1]

_csparsetools.pyx in scipy.sparse._csparsetools.lil_get_lengths()

ValueError: Buffer has wrong number of dimensions (expected 1, got 2)

I don't know if this might be relevent but if I run the transform() method again I get a different error:

### Traceback

AttributeError Traceback (most recent call last)
in
----> 1 transformed_data = reducer.transform(np.random.rand(500, 4000))

~/.local/share/virtualenvs/Test-yifRmGUs/lib/python3.7/site-packages/umap/umap_.py in transform(self, X)
2067 dists = submatrix(dmat_shortened, indices_sorted, self._n_neighbors)
2068 elif _HAVE_PYNNDESCENT:
-> 2069 indices, dists = self._rp_forest.query(X, self.n_neighbors)
2070 elif self._sparse_data:
2071 if not scipy.sparse.issparse(X):

~/.local/share/virtualenvs/Test-yifRmGUs/lib/python3.7/site-packages/pynndescent/pynndescent_.py in query(self, query_data, k, epsilon)
1221 k,
1222 self._raw_data,
-> 1223 self._search_forest,
1224 self._search_graph.indptr,
1225 self._search_graph.indices,

AttributeError: 'NNDescent' object has no attribute '_search_forest'

[lmcinnes]

Should be fixed in master; I'll try to roll up any other fixes/patches and make a release soon.

[michaeljtobias]

I ran into what looked to be the same issue, when running transform with a previously fit and saved model, so I installed from master as described in the readme, recreated the new fit model with the new installation, but now can't seem to be able to load it with joblib just before running transform.

Traceback (most recent call last):
...
File "/home/ubuntu/clustering/predict.py", line 42, in load_saved
mapper_umap = joblib.load(saved_umap_model_filepath)
File "/home/ubuntu/clustering/.venv/lib/python3.6/site-packages/joblib/numpy_pickle.py", line 585, in load
obj = _unpickle(fobj, filename, mmap_mode)
File "/home/ubuntu/clustering/.venv/lib/python3.6/site-packages/joblib/numpy_pickle.py", line 504, in _unpickle
obj = unpickler.load()
File "/home/ubuntu/anaconda3/lib/python3.6/pickle.py", line 1050, in load
dispatchkey[0]
File "/home/ubuntu/clustering/.venv/lib/python3.6/site-packages/joblib/numpy_pickle.py", line 329, in load_build
Unpickler.load_build(self)
File "/home/ubuntu/anaconda3/lib/python3.6/pickle.py", line 1507, in load_build
setstate(state)
File "/home/ubuntu/clustering/.venv/lib/python3.6/site-packages/pynndescent/pynndescent_.py", line 1028, in __setstate__
self._rp_forest = tuple([renumbaify_tree(tree) for tree in d["_rp_forest"]])
File "/home/ubuntu/clustering/.venv/lib/python3.6/site-packages/pynndescent/pynndescent_.py", line 1028, in
self._rp_forest = tuple([renumbaify_tree(tree) for tree in d["_rp_forest"]])
File "/home/ubuntu/clustering/.venv/lib/python3.6/site-packages/pynndescent/rp_trees.py", line 1178, in renumbaify_tree
hyperplanes.extend(tree.hyperplanes)
File "/home/ubuntu/clustering/.venv/lib/python3.6/site-packages/numba/typed/typedlist.py", line 366, in extend
return _extend(self, iterable)
File "/home/ubuntu/clustering/.venv/lib/python3.6/site-packages/numba/core/dispatcher.py", line 415, in _compile_for_args
error_rewrite(e, 'typing')
File "/home/ubuntu/clustering/.venv/lib/python3.6/site-packages/numba/core/dispatcher.py", line 358, in error_rewrite
reraise(type(e), e, None)
File "/home/ubuntu/clustering/.venv/lib/python3.6/site-packages/numba/core/utils.py", line 80, in reraise
raise value.with_traceback(tb)
numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)

• Resolution failure for literal arguments:
  No implementation of function Function() found for signature:

impl_extend(ListType[array(float64, 2d, C)], reflected list(array(float32, 1d, C)))

There are 2 candidate implementations:

• Of which 2 did not match due to:
  Overload in function 'impl_extend': File: numba/typed/listobject.py: Line 1027.
  With argument(s): '(ListType[array(float64, 2d, C)], reflected list(array(float32, 1d, C)))':
  Rejected as the implementation raised a specific error:
  TypingError: Failed in nopython mode pipeline (step: nopython frontend)
  
  
  
  • Resolution failure for literal arguments:
    
    No implementation of function Function() found for signature:
  
  

>>> impl_append(ListType[array(float64, 2d, C)], array(float32, 1d, C))

There are 2 candidate implementations:
- Of which 2 did not match due to:
Overload in function 'impl_append': File: numba/typed/listobject.py: Line 589.
With argument(s): '(ListType[array(float64, 2d, C)], array(float32, 1d, C))':
Rejected as the implementation raised a specific error:
LoweringError: Failed in nopython mode pipeline (step: nopython mode backend)

  File ".venv/lib/python3.6/site-packages/numba/typed/listobject.py", line 597:
      def impl(l, item):
          casteditem = _cast(item, itemty)
          ^

  During: lowering "$0.4 = call $0.1(item, $0.3, func=$0.1, args=[Var(item, listobject.py:597), Var($0.3, listobject.py:597)], kws=(), vararg=None)" at /home/ubuntu/clustering/.venv/lib/python3.6/site-packages/numba/typed/listobject.py (597)
 raised from /home/ubuntu/clustering/.venv/lib/python3.6/site-packages/numba/core/utils.py:81

• Resolution failure for non-literal arguments:
  None

  During: resolving callee type: BoundFunction((, 'append') for ListType[array(float64, 2d, C)])
  During: typing of call at /home/ubuntu/clustering/.venv/lib/python3.6/site-packages/numba/typed/listobject.py (1051)

  File ".venv/lib/python3.6/site-packages/numba/typed/listobject.py", line 1051:
  def impl(l, iterable):
  
  for i in iterable:
  l.append(i)
  ^

  raised from /home/ubuntu/clustering/.venv/lib/python3.6/site-packages/numba/core/typeinfer.py:994

  • Resolution failure for non-literal arguments:
    None

During: resolving callee type: BoundFunction((, 'extend') for ListType[array(float64, 2d, C)])
During: typing of call at /home/ubuntu/clustering/.venv/lib/python3.6/site-packages/numba/typed/typedlist.py (101)

File ".venv/lib/python3.6/site-packages/numba/typed/typedlist.py", line 101:
def _extend(l, iterable):
return l.extend(iterable)

[AuSpotter]

I had the same error, both <>4095 - fixed by installing pynndescent