Umap: Create a benchmark to track speed improvements / regressions

[sleighsoft]

Considering a lot of time goes into optimizing the speed of UMAP it seems necessary to have some kind of bechmark/suite to evaluate performance.

EDIT:
A collection of useful links to get started:
Tracking System Capabilities: https://github.com/giampaolo/psutil
Datasets: https://www.tensorflow.org/datasets
Benchmarking: https://pypi.org/project/pyperf/

[lmcinnes]

I would love to have that -- but I'm not sure of the best way to actually do that. I have very little experience/expertise in CI infrastructure, so yes, definitely help wanted.

[sleighsoft]

First things first we would have to gather some datasets on which benchmarking makes sense.

[lmcinnes]

I believe fetch_openml in sklearn can get us a bunch easily. At the least we can get MNIST and Fashion-MNIST with that. It would also be beneficial to have a word-embedding dataset on hand. For a sparse dataset I think a CountVectorized 20newsgroups is easy enough to get and should be somewhat representative. Ideally we would have a biology dataset as well, but I don't know where to get those.

[sleighsoft]

I have code to easily use https://www.tensorflow.org/datasets/catalog

But that would require tensorflow as a dependency for benchmarks.

[lmcinnes]

For a test suite to run in CI only for certain configurations (i.e. not by default) that's probably fine.

[raghavmecheri]

@lmcinnes Are you still looking to get this done? I've been working on a lightweight side-project that we could plug and play UMAP constructor calls into for performance metrics, and then integrate the same with the project's Azure pipelines

[lmcinnes]

I would love to have some benchmarks like that available, so yes, if you have the opportunity please let me know if there's anythign I can do from this end to help.

[raghavmecheri]

I would love to have some benchmarks like that available, so yes, if you have the opportunity please let me know if there's anythign I can do from this end to help.

Sounds good, will work on the same. Please feel free to assign this issue to me, and will get to it over the weekend/next week. I'm a big fan of this library, and I'm glad to help out. Feel free to tag me on other issues that you may need a hand with :)