Umap: Transform shifting clusters on repeated calls

[Kenneth-Schroeder]

Hi @lmcinnes,
I am trying to use UMAP to show the increasing ability to seperate different classes over the training process of a neural network, similar to https://medium.com/@rakesh.chada/understanding-neural-networks-by-embedding-hidden-representations-f256842ebf3a

My idea was to select an epoch e, a layer l, as well as an input i.
I would then fit a UMAP instance on the high dimensional representations of i at layer l for that epoch e.
Afterwards, I want to use the learned transformation on other epochs for a fixed i and l.

However, the transformations for the same e, l and i seem to change after several runs on other e's. I don't mean some noisy variations, that I would expect to derive from the non-deterministic nature of the algorithm, but all the clusters are changing their shape and position within the embedding.

On the images you can see results of alternating transforms on epoch e=83 / 82 for fixed l and i.

Do you know, why the clusters are shifting so much and how I could possibly prevent this?
Thank you very much for your time.

[lmcinnes]

It should not change things like this -so there is a bug somewhere, possibly accidentally introduced when fixing some other issues. I'm pretty busy at the moment, but if you could ping me again in a week or two I will try to look at this properly.

[Kenneth-Schroeder]

Thanks for the fast response. There's no hurry and I will be happy to remind you :-)

[Kenneth-Schroeder]

Just a short update:
After a little more research I believe that the reason for this issue might be that UMAP is transductive, as you stated here.
However, this confuses me since in your talk you say that new unseen points can be embedded using UMAP.

To clarify: In my experiment above I only fit the UMAP instance on the embedding of layer l of epoch 83. The embedding of layer l of epoch 82 might contain new (unseen) points for the UMAP reducer, which I just passed into the transform function.

[lmcinnes]

It should work as you are expecting, but for some reason something is
astray. It is essentially transductive, but there are some ways we can deal
with new data (even though we don't get an explicit map).

On Tue, Jun 4, 2019 at 8:46 AM Kenneth-Schroeder [e-mail]
wrote:

Just a short update:
After a little more research I believe that the reason for this issue
might be that UMAP is transductive, as you stated here
https://github.com/lmcinnes/umap/issues/55#issuecomment-371170888.
However, this confuses me since in your talk
https://youtu.be/nq6iPZVUxZU?t=1205 you say that new unseen points can
be embedded using UMAP.

To clarify: In my experiment above I only fit the UMAP instance on the
embedding of layer l of epoch 83. The embedding of layer l of epoch 82
might contain new (unseen) points for the UMAP reducer, which I just passed
into the transform function.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/lmcinnes/umap/issues/247?email_source=notifications&email_token=AC3IUBLMUXIZBUCLVWW37TDPYZP4FA5CNFSM4HSHOUYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODW4OJDI#issuecomment-498656397,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AC3IUBLKKIQSMLAORJ34IILPYZP4FANCNFSM4HSHOUYA
.

[Kenneth-Schroeder]

Here's a friendly reminder, just as you wished :-)

[lmcinnes]

Thanks for the reminder. I did some digging and it seems like the way this might occur is if the number of points you are transforming is exactly equal to the number of points in the initial embedding -- there is a check in the code to see whether you are trying to transform new points (and hence holds reference points fixed) but that only checks if the numbers of points are the same (because that was cheap and easy). Is it possible that this is the problem? If so I can try to make a still cheap but more robust check. Otherwise it might be time to dig deeper and figure out what is going on.

[Kenneth-Schroeder]

That sounds exactly right! I am using 5000 points for each class each time (35000 points in total) because the counts would be badly distributed otherwise. Nice catch :-)

[lmcinnes]

Ah, I see. I'll try to make the test a little more robust against this, ideally without causing too much overhead.