Umap: Custom embedding initialization

[kylemcdonald]

Looking at the embedding initialization options, I see 'random' and 'spectral'. Would it be possible to initialize with a custom embedding? And if so, would this embedding be at all preserved?

In trying to compare the effect of different parameter changes, it could be helpful to use the embedding of a previous run as the initialization to a new UMAP instance with slightly different parameters. For example, these two min_dist values result in embeddings with different global orientations but similar local relationships.

[lmcinnes]

That's a great idea that I hadn't considered. It is something along these lines that I was hoping to use to make an "update" procedure, but what you are proposing here is the easy concrete way to move toward that.

As to whether the embedding would be preserved at all -- as long as it is "close" in objective function space to the final embedding then it will be preserved. That's really what the spectral embedding is doing: it provides a good starting point and ensures a degree of consistency over multiple runs (up to rotation/reflection).

[kylemcdonald]

Awesome! For these tests I will use the spectral initialization to encourage consistency between runs. But I was also considering what it would look like to initialize with a t-SNE embedding, and then it would be much easier to see how the embedding changes after running UMAP.

[lmcinnes]

That's a great idea for making the comparison clear. It certainly raises the priority on getting the custom initialisation done!

[lmcinnes]

Provisional support turned out to be straightforward. Of course it may be a little glitchy if I didn't catch all the ways it can go wrong, but you should now be able to pass a numpy array of initial positions to the init parameter and have it work from there (in current master).

[kylemcdonald]

great! it works!

the only hurdle i had to get across: t-SNE tends towards larger embeddings than umap. for example, t-SNE with default parameters embeds MNIST into a +/-35 range while UMAP is closer to +/-15. so initializing with the raw output of t-SNE produces some minor artifacts. once i scale the t-SNE output closer to the UMAP output it works better.

some gifs for your trouble:

https://www.dropbox.com/s/wbv73swh6qlrexg/mnist-all-init-0.8.gif?dl=0

this one is UMAP with min_dist=0.8 and initialized with a t-SNE embedding scaled by 0.4.

https://www.dropbox.com/s/cufrbjsbm79a3kh/mnist-all-init.gif?dl=0

this one is UMAP with default parameters also initialized with a t-SNE embedding scaled by 0.4.

the final embeddings of both are normalized uniformly across both axes to fill the screen with a little padding.