hdbscan.prediction.all_points_membership_vectors datatype error

[JMistral]

Hi I have run the following code for one of my project:

ms = 14
mc = 10
clusterer = hdbscan.HDBSCAN(min_samples = ms, min_cluster_size=mc,prediction_data=True).fit(X)
X_memberships = hdbscan.prediction.all_points_membership_vectors(clusterer)

And got the following error message:

/.local/lib/python2.7/site-packages/hdbscan/prediction.pyc in all_points_membership_vectors(clusterer)
534 all_points,
535 clusterer.prediction_data_.exemplars,
--> 536 clusterer.prediction_data_.dist_metric)
537 outlier_vecs = all_points_outlier_membership_vector(
538 clusters,

hdbscan/_prediction_utils.pyx in hdbscan._prediction_utils.all_points_dist_membership_vector()

ValueError: Buffer dtype mismatch, expected 'float64_t' but got 'float'

[cvanoort]

I also encountered this error.
It seems to work properly if the data is explicitly cast to the numpy.float64 type, i.e.

clusterer = hdbscan.HDBSCAN().fit(X.astype(numpy.float64))

The error comes from type checking in the Cython function indicated in the error trace, hdbscan._prediction_utils.all_points_dist_membership_vector, where the first argument expects a numpy.float64 type.
Working backwards through the error trace:

• all_points, the argument that causes the error here, is created on line 527 of hdbscan.prediction using the _raw_data attribute of a PredictionData object
• PredictionData._raw_data is created on line 101 of hdbscan.prediction in the __init__ method and is simply a stored argument.
• When hdbscan.HDBSCAN is called with prediction_data=True, it calls generate_prediction_data, which creates a PredictionData object using the self._raw_data attribute of the HDBSCAN object.
• HDBSCAN._raw_data is created in HDBSCAN.__inii__ and then updated in HDBSCAN.fit

HDBSCAN._raw_data is validated by sklearn.utils.check_array on line 833 of hdbscan.hdbscan_, so it seems like the issue here may be that check_array is not performing a strict enough type validation.

[lmcinnes]

Thanks for tracking this down. I'll see if I can get a proper fix in place
soon.

On Sun, Sep 2, 2018 at 1:28 PM Colin Van Oort [e-mail]
wrote:

I also encountered this error.
It seems to work properly if the data is explicitly cast to the
numpy.float64 type, i.e.

clusterer = hdbscan.HDBSCAN().fit(X.astype(numpy.float64))

The error comes from type checking in the Cython function indicated in the
error trace, hdbscan._prediction_utils.all_points_dist_membership_vector
https://github.com/scikit-learn-contrib/hdbscan/blob/80dd6d21830b9e2f703eb3862b28fd2f5b67476b/hdbscan/_prediction_utils.pyx#L98,
where the first argument expects a numpy.float64 type.
Working backwards through the error trace:

• all_points, the argument that causes the error here, is created on line
  527 of hdbscan.prediction
  https://github.com/scikit-learn-contrib/hdbscan/blob/80dd6d21830b9e2f703eb3862b28fd2f5b67476b/hdbscan/prediction.py#L527
  using the _raw_data attribute of a PredictionData
  https://github.com/scikit-learn-contrib/hdbscan/blob/80dd6d21830b9e2f703eb3862b28fd2f5b67476b/hdbscan/prediction.py#L21
  object
• PredictionData._raw_data is created on line 101 of hdbscan.prediction
  https://github.com/scikit-learn-contrib/hdbscan/blob/80dd6d21830b9e2f703eb3862b28fd2f5b67476b/hdbscan/prediction.py#L101
  in the __init__ method and is simply a stored argument.
• When hdbscan.HDBSCAN is called with prediction_data=True, it calls
  generate_prediction_data
  https://github.com/scikit-learn-contrib/hdbscan/blob/80dd6d21830b9e2f703eb3862b28fd2f5b67476b/hdbscan/hdbscan_.py#L879,
  which creates a PredictionData object
  https://github.com/scikit-learn-contrib/hdbscan/blob/80dd6d21830b9e2f703eb3862b28fd2f5b67476b/hdbscan/hdbscan_.py#L896
  using the self._raw_data attribute of the HDBSCAN object.
• HDBSCAN._raw_data is created in HDBSCAN.__inii__
  https://github.com/scikit-learn-contrib/hdbscan/blob/80dd6d21830b9e2f703eb3862b28fd2f5b67476b/hdbscan/hdbscan_.py#L813
  and then updated in HDBSCAN.fit
  https://github.com/scikit-learn-contrib/hdbscan/blob/80dd6d21830b9e2f703eb3862b28fd2f5b67476b/hdbscan/hdbscan_.py#L834

HDBSCAN._raw_data is validated by sklearn.utils.check_array
http://scikit-learn.org/stable/modules/generated/sklearn.utils.check_array.html
on line 833 of hdbscan.hdbscan_
https://github.com/scikit-learn-contrib/hdbscan/blob/80dd6d21830b9e2f703eb3862b28fd2f5b67476b/hdbscan/hdbscan_.py#L833,
so it seems like the issue here may be that check_array is not performing a
strict enough type validation.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/scikit-learn-contrib/hdbscan/issues/231#issuecomment-417946161,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ALaKBQ3vxZkkZT07aZR6lwffZ7NSXwJdks5uXBUsgaJpZM4WIMgp
.