Umap: Systematically determine `min_dist` and `n_neighbors`

[NTNguyen13]

I tested multiple combination of min_dist and n_neighbor on my data, I found that a suitable combination of those hyperparameters can separate the all the cluster(according to labels). I wonder how to determine those hyperparameters systematically, not by visualization of the data after embedding.

[lmcinnes]

If you have specific labels to measure against you could cluster the embedding (with, say, hdbscan) and look at the adjusted Rand score (adjusted_rand_score in sklearn.metrics). of the clustering against the labels. Maximising that score would be giving an embedding that most closely matches your labels.

Personally I would be a little wary of such hyper-parameter searching, but a lot depends on your experiment and the objectives you have in mind. Best of luck.

[NTNguyen13]

yes, I'm actually doing the Hyperparameter Optimization with ARI metric for the particular problem. But I would like to know if there is a more general approach, such as in problems without the labels.

[lmcinnes]

If there are no labels then there is not "truth", so no, there isn't a general method that works without labels.

[MattWenham]

This has got me thinking, and that can be a dangerous or wonderful thing depending on circumstances.

I could envisage a scheme which had min_dist, n_neighbor and n_components as variable hyperparameters, with n_components maybe being optional. You'd then feed the output from UMAP into a clustering method which did not require specification of the number of clusters up front.

This would result in a clustering of the UMAP into a number of classes, with the classes themselves and the number of classes being output from the clustering method. This output could then be fed into a figure-of-merit calculation such as Calinski–Harabasz, which could then be optimised by tuning the hyperparameters.

[NTNguyen13]

yes, I was thinking the same. As far as I know, HDBSCAN/DBSCAN employ the n_sample parameter, I just don't know how closely related theirs are to UMAP n_sample

@lmcinnes could you please clarify the relationship between n_sample of HDBSCAN and UMAP? Can they be passed down between 2 algorithm?

[lmcinnes]

The n_neighbors parameter in UMAP and the n_samples parameter in DBSCAN/HDBSCAN mean different things. They are, at least, measured in the same units. There is some reason to believe one could construct a clustering algorithm that would make use of the UMAP n_neighbors in a similar way to DBSCAN/HDBSCAN. As it stands however, I wouldn't say they should be tied or linked. I would say, however, that they should probably be on roughly the same scale (I would expect them to be the same order of magnitude in general, for instance).

[MattWenham]

If you're interested, I can point you towards a clustering algorithm based on the Self-Organising Map (SOM) which essentially has zero hyper parameters of its own and does not require prior definition of the number of clusters / classes. I have developed a prototype implementation in Python.

I say 'essentially' as the original researchers did not address the size of the SOM which should be used for the clustering, which form a pair of hyperparameters. I have an idea as to how to address this issue, but have not yet had the bandwidth to test it. Happy to discuss.

[bzip2]

@MattWenham: please do tell.

[MattWenham]

Start here for the best description of the algorithm: https://hal.archives-ouvertes.fr/hal-01461451

My implementation use somoclu as the SOM engine. It's trapped in a Jupyter notebook at the moment, but I will try to get that shared at some point soon.