Umap: Trouble loading UMAP model done with version 0.4.2 - Error coming from pynndescent

Created on 7 May 2020 · 11Comments · Source: lmcinnes/umap

Some tips given already in #178, although it does not solve the problem here.

I'm having troubles when loading a previously saved UMAP model.
I used umap version 0.4.2
The model I construct calls umap as:
mapper = umap.UMAP(n_neighbors=150, n_components=2, metric='cosine', random_state=69, transform_queue_size=8.0, min_dist=0.0, target_weight=1, target_n_neighbors=20, verbose=True).fit(nzDense_norm,y=labels_train)

nzDense_norm is 290000x4096 numpy float32 matrix
labels_train is 290000 numpy int8 vector

The model is computed correctly and I save it using joblib:
outputFile = open(filename, 'wb')
joblib.dump(mapper, outputFile, compress=True)
outputFile.close()

Then, when I tried to load it using:
mapper = joblib.load(filename)

I get the error below, (which has been reported in #179 but once the model has been loaded)
Any clues or help?
In fact I performed this operation a lot of times with umap-learn 0.3.10 without any trouble. Afterwards, to get 0.4.2 running I needed to install pynndescent (which I did no thave when using 0.3.10) because I was experiencing RAM problems. But now the models are correctly done, but there is this problem of loading it once saved into disk.

Traceback (most recent call last):
File "", line 1, in
File "/home/peppone/Feina/INLOC/Projectes/WorkDir/Trainer/lib/python3.6/site-packages/joblib/numpy_pickle.py", line 605, in load
obj = _unpickle(fobj, filename, mmap_mode)
File "/home/peppone/Feina/INLOC/Projectes/WorkDir/Trainer/lib/python3.6/site-packages/joblib/numpy_pickle.py", line 529, in unpickle
obj = unpickler.load()
File "/usr/lib/python3.6/pickle.py", line 1050, in load
dispatchkey[0]
File "/home/peppone/Feina/INLOC/Projectes/WorkDir/Trainer/lib/python3.6/site-packages/joblib/numpy_pickle.py", line 342, in load_build
Unpickler.load_build(self)
File "/usr/lib/python3.6/pickle.py", line 1507, in load_build
setstate(state)
File "/home/peppone/Feina/INLOC/Projectes/WorkDir/Trainer/lib/python3.6/site-packages/pynndescent/pynndescent.py", line 1026, in setstate
self._rp_forest = tuple([renumbaify_tree(tree) for tree in d["rp_forest"]])
File "/home/peppone/Feina/INLOC/Projectes/WorkDir/Trainer/lib/python3.6/site-packages/pynndescent/pynndescent.py", line 1026, in
self._rp_forest = tuple([renumbaify_tree(tree) for tree in d["_rp_forest"]])
File "/home/peppone/Feina/INLOC/Projectes/WorkDir/Trainer/lib/python3.6/site-packages/pynndescent/rp_trees.py", line 1178, in renumbaify_tree
hyperplanes.extend(tree.hyperplanes)
File "/home/peppone/Feina/INLOC/Projectes/WorkDir/Trainer/lib/python3.6/site-packages/numba/typed/typedlist.py", line 302, in extend
return _extend(self, iterable)
File "/home/peppone/Feina/INLOC/Projectes/WorkDir/Trainer/lib/python3.6/site-packages/numba/dispatcher.py", line 401, in _compile_for_args
error_rewrite(e, 'typing')
File "/home/peppone/Feina/INLOC/Projectes/WorkDir/Trainer/lib/python3.6/site-packages/numba/dispatcher.py", line 344, in error_rewrite
reraise(type(e), e, None)
File "/home/peppone/Feina/INLOC/Projectes/WorkDir/Trainer/lib/python3.6/site-packages/numba/six.py", line 668, in reraise
raise value.with_traceback(tb)
numba.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Invalid use of Function() with argument(s) of type(s): (ListType[array(float64, 2d, C)], reflected list(array(float32, 1d, C)))

parameterized
In definition 0:
TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Invalid use of Function(<function impl_append at 0x7f97dcd350d0>) with argument(s) of type(s): (ListType[array(float64, 2d, C)], array(float32, 1d, C))
parameterized
In definition 0:
LoweringError: Failed in nopython mode pipeline (step: nopython mode backend)

File "../../Trainer/lib/python3.6/site-packages/numba/listobject.py", line 487:
def impl(l, item):
casteditem = _cast(item, itemty)
^

[1] During: lowering "$0.4 = call $0.1(item, $0.3, func=$0.1, args=[Var(item, listobject.py:487), Var($0.3, listobject.py:487)], kws=(), vararg=None)" at /home/peppone/Feina/INLOC/Projectes/WorkDir/Trainer/lib/python3.6/site-packages/numba/listobject.py (487)
raised from /home/peppone/Feina/INLOC/Projectes/WorkDir/Trainer/lib/python3.6/site-packages/numba/six.py:669
This error is usually caused by passing an argument of a type that is unsupported by the named function.
[1] During: resolving callee type: BoundFunction((, 'append') for ListType[array(float64, 2d, C)])
[2] During: typing of call at /home/peppone/Feina/INLOC/Projectes/WorkDir/Trainer/lib/python3.6/site-packages/numba/listobject.py (895)

File "../../Trainer/lib/python3.6/site-packages/numba/listobject.py", line 895:
def impl(l, iterable):

for i in iterable:
l.append(i)
^

raised from /home/peppone/Feina/INLOC/Projectes/WorkDir/Trainer/lib/python3.6/site-packages/numba/typeinfer.py:985

In definition 1:
TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Invalid use of Function() with argument(s) of type(s): (ListType[array(float64, 2d, C)], array(float32, 1d, C))

parameterized
In definition 0:
LoweringError: Failed in nopython mode pipeline (step: nopython mode backend)

File "../../Trainer/lib/python3.6/site-packages/numba/listobject.py", line 487:
def impl(l, item):
casteditem = _cast(item, itemty)
^

File "../../Trainer/lib/python3.6/site-packages/numba/listobject.py", line 895:
def impl(l, iterable):

for i in iterable:
l.append(i)
^

raised from /home/peppone/Feina/INLOC/Projectes/WorkDir/Trainer/lib/python3.6/site-packages/numba/typeinfer.py:985

This error is usually caused by passing an argument of a type that is unsupported by the named function.
[1] During: resolving callee type: BoundFunction((, 'extend') for ListType[array(float64, 2d, C)])
[2] During: typing of call at /home/peppone/Feina/INLOC/Projectes/WorkDir/Trainer/lib/python3.6/site-packages/numba/typed/typedlist.py (82)

File "../../Trainer/lib/python3.6/site-packages/numba/typed/typedlist.py", line 82:
def _extend(l, iterable):
return l.extend(iterable)
^

Source

jmirats

Most helpful comment

Ok, I think I found the cause of that error in my first coment:

In rp_trees.py lines 32-33 there are two possibilities for the hyperplanes definition:

dense_hyperplane_type = numba.float32[::1]
sparse_hyperplane_type = numba.float64[:, ::1]

Then in the code, depending on whether the input data is sparse or not make_sparse_tree() or make_dense_tree() functions are called

The fact is that once a model (umap) is obtained and saved to disk, the _rp_forest object is included. But when we load that model using joblib, we go to rp_trees.py renumbaify_tree() function which has a definition like:

hyperplanes = numba.typed.List.empty_list(sparse_hyperplane_type)

And this is providing the error type when loading if the data for the umap model was not sparse (as it is my case)

I simply tried to change this line of the function to:

hyperplanes = numba.typed.List.empty_list(dense_hyperplane_type)

and I could properly load the model saved (although now, a warning from numba is obtained as detailed below)

I assume a conditional check for the sparsity of data should be added in renumbaify_tree() function. I'm sorry I can not do it myself as I'm not fully aware of where else will this imply changes, but hope that this analysis helps to solve the issue.

The new warning:
/home/peppone/Feina/INLOC/Projectes/WorkDir/Trainer/lib/python3.6/site-packages/numba/typed/typedlist.py:82: NumbaPendingDeprecationWarning:
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'iterable' of function 'impl_extend..select_impl..impl'.

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../Trainer/lib/python3.6/site-packages/numba/listobject.py", line 893:
else:
def impl(l, iterable):
^

return l.extend(iterable)
/home/peppone/Feina/INLOC/Projectes/WorkDir/Trainer/lib/python3.6/site-packages/numba/ir_utils.py:2041: NumbaPendingDeprecationWarning:
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'iterable' of function '_extend'.

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../Trainer/lib/python3.6/site-packages/numba/typed/typedlist.py", line 81:
@njit
def _extend(l, iterable):
^

warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/home/peppone/Feina/INLOC/Projectes/WorkDir/Trainer/lib/python3.6/site-packages/numba/typed/typedlist.py:82: NumbaPendingDeprecationWarning:
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'iterable' of function 'impl_extend..select_impl..impl'.

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../Trainer/lib/python3.6/site-packages/numba/listobject.py", line 893:
else:
def impl(l, iterable):
^

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../Trainer/lib/python3.6/site-packages/numba/typed/typedlist.py", line 81:
@njit
def _extend(l, iterable):
^

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../Trainer/lib/python3.6/site-packages/numba/listobject.py", line 893:
else:
def impl(l, iterable):
^

return l.extend(iterable)
/home/peppone/Feina/INLOC/Projectes/WorkDir/Trainer/lib/python3.6/site-packages/numba/typed/typedlist.py:82: NumbaTypeSafetyWarning: unsafe cast from UniTuple(int64 x 2) to UniTuple(int32 x 2). Precision may be lost.
return l.extend(iterable)
/home/peppone/Feina/INLOC/Projectes/WorkDir/Trainer/lib/python3.6/site-packages/numba/ir_utils.py:2041: NumbaPendingDeprecationWarning:
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'iterable' of function '_extend'.

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../Trainer/lib/python3.6/site-packages/numba/typed/typedlist.py", line 81:
@njit
def _extend(l, iterable):
^

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../Trainer/lib/python3.6/site-packages/numba/listobject.py", line 893:
else:
def impl(l, iterable):
^

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../Trainer/lib/python3.6/site-packages/numba/typed/typedlist.py", line 81:
@njit
def _extend(l, iterable):
^

warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))

jmirats on 8 May 2020

👍3

All 11 comments

Hi again, I've testing different options. By now I realized this error only affects data sets bigger than 4096 data points, so the issue is within the "# Standard case" solving in the code

jmirats on 8 May 2020

Hello, looking a little bit more into the code, I see the difference in the umap object on those two situations (dataset<4096 or dataset>4096) are the elements '_knn_dists', '_knn_indices' and '_rp_forest', so that could be aclue of where the issue is.

I suspect the error comes from pynndescent with the '_rp_forest' object. I see that within rp_trees.py file there are two definitions as
dense_hyperplane_type = numba.float32[::1]
sparse_hyperplane_type = numba.float64[:, ::1]

which could be related to that error when trying to load the umap generated model
But I do not know how to fix or overcome this

jmirats on 8 May 2020

Ok, I think I found the cause of that error in my first coment:

In rp_trees.py lines 32-33 there are two possibilities for the hyperplanes definition:

dense_hyperplane_type = numba.float32[::1]
sparse_hyperplane_type = numba.float64[:, ::1]

Then in the code, depending on whether the input data is sparse or not make_sparse_tree() or make_dense_tree() functions are called

hyperplanes = numba.typed.List.empty_list(sparse_hyperplane_type)

And this is providing the error type when loading if the data for the umap model was not sparse (as it is my case)

I simply tried to change this line of the function to:

hyperplanes = numba.typed.List.empty_list(dense_hyperplane_type)

and I could properly load the model saved (although now, a warning from numba is obtained as detailed below)

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../Trainer/lib/python3.6/site-packages/numba/listobject.py", line 893:
else:
def impl(l, iterable):
^

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../Trainer/lib/python3.6/site-packages/numba/typed/typedlist.py", line 81:
@njit
def _extend(l, iterable):
^

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../Trainer/lib/python3.6/site-packages/numba/listobject.py", line 893:
else:
def impl(l, iterable):
^

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../Trainer/lib/python3.6/site-packages/numba/typed/typedlist.py", line 81:
@njit
def _extend(l, iterable):
^

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../Trainer/lib/python3.6/site-packages/numba/listobject.py", line 893:
else:
def impl(l, iterable):
^

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../Trainer/lib/python3.6/site-packages/numba/typed/typedlist.py", line 81:
@njit
def _extend(l, iterable):
^

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../Trainer/lib/python3.6/site-packages/numba/listobject.py", line 893:
else:
def impl(l, iterable):
^

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types

File "../../Trainer/lib/python3.6/site-packages/numba/typed/typedlist.py", line 81:
@njit
def _extend(l, iterable):
^

warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))

jmirats on 8 May 2020

👍3

Just ran into this also on 0.4.4. Same error with loading after saving w/ pickle or joblib. Works fine on datasets < 4096

everyoneishappy on 15 Jun 2020

I'll have to try and look at this soon. Things involving integrating with numba and pickle are always rather hard to get working just so.

lmcinnes on 16 Jun 2020

👍1

Thanks so much. Such an amazing package!

everyoneishappy on 16 Jun 2020

Hi,
Just ran into this also on umap=0.4.5, pynndescent=0.4.8; Works fine on smaller datasets, but not on large ones. Looking forward to a fix. Thanks for all your work!

jonlehrer on 17 Jul 2020

In case it helps anybody: I've been able to "bypass" the issue for now by uninstalling pynndescent. UMAP still works without it, albeit slower, and this issue doesn't occur for me anymore.

jonlehrer on 22 Jul 2020

The problem with training with pynndescent is that for large number of samples, you'll have memory explosion issues mentioned in #462
If the model is already trained using pynndescent and try to load it without it installed, it will lead to import errors.

wise-east on 12 Aug 2020

Also having trouble with joblib. When dumping large file, the kernel crashes after the file* has been dumped. Then when loading file I am getting EOFerror. Also installed pynndescent as I was getting memory explosion before.

*File is sklearn pipeline object