Umap: Combining multiple UMAP for different embedding models

[ajkl]

https://umap-learn.readthedocs.io/en/latest/composing_models.html
This tutorial shows how 2 views of same data can be combined. I was wondering if this would work on 2 different embeddings on the same data?
Usecase -- We often cluster using tsne or umap using representation embeddings from some ML model. And often times we have 2 different embeddings capturing different features of the same data. One way to combine them for similarity is
w1*sim1 + w2*sim2 and use that as a combined similarity score that capture both emb1 and emb2 feature similarity together.
Made me wonder if combining through umap would have an advantage to cluster data which will capture features from both embedding models.

[lmcinnes]

A lot will depend on what you are trying to combine. The composition operators will work as long as you are dealing with UMAP models; how you create those models -- with different views of the same data; with different parameter choices for UMAP applied to the data; different processing (ML models) for the same basic data; etc. -- doesn't really matter as long as we can line the data up on a row by row basis. What the output will be ... will ultimately depend on how well the relative structures of the different input data and parameters happen to align. At the very least it ought to work, and it may provide a useful combination of things.

please note that I think I am understanding your question, but I am not 100% sure, so please correct me if this isn't quite answering what you want to know.

[ajkl]

I think you got the question but just in case, here is an example --
Lets say I have 2 models for fashion/clothing dataset. One model is optimized to capture the colors of the clothing images and other one captures more semantics like style and category of the clothing images. So if I have to cluster the images now by combining both semantics and color similarity of images I will do something like w1sim1 + w2sim2 and use that as a similarity metric for clustering.
That made me wonder if I train 2 UMAP models on those 2 embeddings and then use composition operators, it might give some interesting results. I will try this out anyway and report back if I get any interesting results. The motivation behind creating this issue was just to check if you have already tried this or if you have any thoughts on it.

[lmcinnes]

I haven't personally tried it, but I do believe it can work -- and by "can" I don't mean "there is some possibility that it might", but rather something more like "if there are clusters in the dataset then this will likely help find them", but just as standard UMAP doesn't magically work on all data, it may not work depending on the dataset(s). I think it is the "right" thing to do to attempt clustering that respects the two different models.

In terms of why I think this might be a useful approach for this problem: I would assume that part of the difficulty in combining similarity measurements between the models as w1 * sim1 + w2 * sim2 is in finding the right values of w1 and w2. In effect the UMAP composition will be using exactly this combined similarity, but will choosing different relative scaling weights locally for each sample -- so in some regions of the space one scaling may be appropriate, but in another region a different scaling may make sense. These local scaling weights will be data driven, based on the local distribution of data (and similarities) in each model. Presumably this can offer some significant advantages.

Where this approach won't work is where some smarter or different weighting scheme is required. The combined model with its local relative weighting is good because it removes the burden of choices, but if the automatically selected option isn't right for your data then there is little leeway to fix it.

I would certainly be interested in any results you get -- and I think others may be interested as well, so please post them to this issue if you can. In fact, presuming the data is shareable and the example works out well, a PR from you adding it to the documentation (properly credited to you for the example of course) would be very welcome.

[ajkl]

Thank you @lmcinnes for umap work and sharing your thoughts!
I will keep you posted on the results, a PR to the docs if everything works well sounds good :)