Umap: TypeError: Cannot use scipy.linalg.eigh for sparse A with k >= N. Use scipy.linalg.eigh(A.toarray()) or reduce k.

[Cristianasp]

Hello, I am using umap on Kaggle competition´s dataset:
https://www.kaggle.com/c/dont-overfit-ii

I have scaled the dataset and ran umap using different values for n_neighbors and n_components.
for n_components in (2,3,4,5,6,7,9,10,15, 20, 30, 37, 40, 50, 75, 100, 250)
for n_neighbors in (2,3,5,7,9,13,15, 16, 19, 20, 35, 30, 35, 40, 50 , 60, 100)

For ALL variations of n-neighbors and n-component I got a warning message:
C:\ProgramData\Anaconda3\envs\NOVO\lib\site-packages\sklearn\metrics\pairwise.py:258: RuntimeWarning: invalid value encountered in sqrt
return distances if squared else np.sqrt(distances, out=distances)

And specifically when running for 250 components and 60 neighbors, I got the following warnings and an exception:
C:\ProgramData\Anaconda3\envs\NOVO\lib\site-packages\sklearn\metrics\pairwise.py:258: RuntimeWarning: invalid value encountered in sqrt
return distances if squared else np.sqrt(distances, out=distances)
C:\ProgramData\Anaconda3\envs\NOVO\lib\site-packages\scipy\sparse\linalg\eigen\arpack\arpack.py:1556: RuntimeWarning: k >= N for N * N square matrix. Attempting to use scipy.linalg.eigh instead.

RuntimeWarning)

TypeError Traceback (most recent call last)
in ()
2 for n in (60, 100):
3 print(c, n)
----> 4 embedding = umap.UMAP(n_neighbors=n, n_components=c).fit(X)
5 pd.DataFrame(embedding.transform(X)).to_csv("processadas/train_umap_n"+str(n)+"_c"+str(c)+".csv")
6 pd.DataFrame(embedding.transform(final)).to_csv("processadas/test_umap_n"+str(n)+"_c"+str(c)+".csv")

C:\ProgramData\Anaconda3\envs\NOVO\lib\site-packagesumapumap_.py in fit(self, X, y)
1536 self.metric,
1537 self._metric_kwds,
-> 1538 self.verbose,
1539 )
1540

C:\ProgramData\Anaconda3\envs\NOVO\lib\site-packagesumapumap_.py in simplicial_set_embedding(data, graph, n_components, initial_alpha, a, b, gamma, negative_sample_rate, n_epochs, init, random_state, metric, metric_kwds, verbose)
941 random_state,
942 metric=metric,
--> 943 metric_kwds=metric_kwds,
944 )
945 expansion = 10.0 / initialisation.max()

C:\ProgramData\Anaconda3\envs\NOVO\lib\site-packagesumap\spectral.py in spectral_layout(data, graph, dim, random_state, metric, metric_kwds)
263 tol=1e-4,
264 v0=np.ones(L.shape[0]),
--> 265 maxiter=graph.shape[0] * 5,
266 )
267 else:

C:\ProgramData\Anaconda3\envs\NOVO\lib\site-packages\scipy\sparse\linalg\eigen\arpack\arpack.py in eigsh(A, k, M, sigma, which, v0, ncv, maxiter, tol, return_eigenvectors, Minv, OPinv, mode)
1557
1558 if issparse(A):
-> 1559 raise TypeError("Cannot use scipy.linalg.eigh for sparse A with "
1560 "k >= N. Use scipy.linalg.eigh(A.toarray()) or"
1561 " reduce k.")

TypeError: Cannot use scipy.linalg.eigh for sparse A with k >= N. Use scipy.linalg.eigh(A.toarray()) or reduce k.

sklearn - 0.20.0
scipy - 1.1.0

[lmcinnes]

The warnings are to do with how sklearn processes pairwise distances, and should be able to be safely ignored. The error, on the other hand, is due to a restriction on how ARPACK computes eigenvectors when using the spectral embedding as initialisation. The short answer is that in general you can't use spectral initialisation (init='spectral', the default) when n_components is greater or equal to the number of samples. Since your dataset has 250 samples this is the issue. You can either use 249 instead of 250, or use init='random' for the 250 dimensional case as a workaround.

[Cristianasp]

Got it! Thanks a lot.

[physics-programmer]

This small test suggests that "n_samples <= n_components +1" should force init to be 'random'. Otherwise it generates an error.

import numpy as np import pandas as pd import umap for n_samples in [3,4,5,6]: for n_components in [3, 4 ,5]: print(f"njobs = {n_samples}") print(f"n_components = {n_components}") #n_jobs = 3 #n_components = 4 samples = ['job'+str(i+1) for i in range(n_samples)] mat = pd.DataFrame(np.random.rand(n_samples, 10), index = samples) print(mat.shape) init = 'spectral' if mat.shape[0] <= n_components + 1: init = 'random' print(init) try: coord = umap.UMAP(init = init, n_components=n_components).fit_transform(mat) except: print('NOT POSSIBLE') finally: print('----------------------')

[DoDzilla-ai]

TL.DR:Accoding to my tests, n_components should be 2 less than the number of samples, not 1. So, it should be 248, not 249 (as suggested by the @lmcinnes)

PS: Isn't this weird?

So, I've downloaded the train.csv from the given kaggle dataset URL. I've tried to apply UMAP on the dataset, which has 250 rows. I didn't specify the first row as header so the dataset has 250 samples. I set the n_components as 249 and still got the same error. Only 248 worked without a problem. I encountered the same thing with a different dataset. Below you can find the minimal working example:

import umap
import pandas as pd

df = pd.read_csv("./train.csv")
print(f"Shape of dataframe is {df.shape}")
for n_components in [248, 249, 250]:
    try:
        reduced_df = umap.UMAP(n_components=n_components, random_state=42).fit_transform(df)
    except TypeError:
        print(f"Error with n_components:{n_components}")
    else:
        print(f"Succesfull with n_components:{n_components}")

Test environment:
python 3.7
scipy==1.1.0
umap-learn==0.5.0
scikit-learn==0.22