Scanpy: Regressing out, potential boundary problem

[jayypaul]

>>> sc.pp.regress_out(adata, ['n_counts', 'percent_mito', 'S_score', 'G2M_score'], n_jobs = 1)
regressing out ['n_counts', 'percent_mito', 'S_score', 'G2M_score']
    sparse input is densified and may lead to high memory use
... storing 'phase' as categorical

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/scanpy/preprocessing/simple.py", line 783, in regress_out
    res = list(map(_regress_out_chunk, tasks))
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/scanpy/preprocessing/simple.py", line 809, in _regress_out_chunk
    result = sm.GLM(data_chunk[:, col_index], regres, family=sm.families.Gaussian()).fit()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/statsmodels/genmod/generalized_linear_model.py", line 1012, in fit
    cov_kwds=cov_kwds, use_t=use_t, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/statsmodels/genmod/generalized_linear_model.py", line 1109, in _fit_irls
    raise ValueError("The first guess on the deviance function "
ValueError: The first guess on the deviance function returned a nan.  This could be a boundary  problem and should be reported.

Hello, after going through the data once and evaluating which clusters are potential unwanted cell types, I grabbed the barcodes (cell sample names or observation indices) and removed them from the dataset after freshly reloading the dataset so that I can do the preprocessing steps without those cells. When I go to regress out, this occurs, but it didn't before I removed those cell types.

Here's an example of how I removed those cell types:

keep_cells = [i for i in adata.obs.index if i not in e13_blood2.obs.index]
adata = adata[keep_cells, :]
adata

Any help appreciated.

[jayypaul]

Note also, when I follow the same protocol for a similar dataset (different timepoint for sequencing), regressing out does not cause this problem.

[fidelram]

can you check that adata.X does not contain columns or rows with only zeros?

print(np.any(adata.X.sum(axis=0) == 0))
print(np.any(adata.X.sum(axis=1) == 0))

[jayypaul]

Hello Fidelram,

Here's the output. Looks like a I have column(s) with zeros. Any suggestions for a remedy and/or possible explanation for why this occurs after removing certain indices from the anndata structure? I'm almost done processing each of my datasets for each timepoint and it hasn't been a problem except for one.

>>> print(np.any(adata.X.sum(axis=0) == 0))
True
>>> print(np.any(adata.X.sum(axis=1) == 0))
False

[fidelram]

You can do:

adata = adata[:,adata.X.sum(axis=0) > 0]

To remove the problematic genes.

Probably after removing some cells, some genes no longer had any value in
the matrix.

Let me know if this helps with your problem.

On Fri, Aug 10, 2018 at 2:33 AM jayypaul [e-mail] wrote:

Hello Fidelram,

Here's the output. Looks like a I have column(s) with zeros. Any
suggestions for a remedy and/or possible explanation for why this occurs
after removing certain indices from the anndata structure? I'm almost done
processing each of my datasets for each timepoint and it hasn't been a
problem except for one.

print(np.any(adata.X.sum(axis=0) == 0))
True
print(np.any(adata.X.sum(axis=1) == 0))
False

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/theislab/scanpy/issues/230#issuecomment-411939560,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEu_1fnAqNW3l-t4w865sLwW6-_2zPU4ks5uPNTDgaJpZM4V0Faw
.

[jayypaul]

I've tried the command ...

adata = adata[:,adata.X.sum(axis=0) > 0]

.. and it kicks back:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/anndata/base.py", line 1205, in __getitem__
    return self._getitem_view(index)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/anndata/base.py", line 1209, in _getitem_view
    return AnnData(self, oidx=oidx, vidx=vidx, asview=True)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/anndata/base.py", line 635, in __init__
    self._init_as_view(X, oidx, vidx)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/anndata/base.py", line 661, in _init_as_view
    var_sub = adata_ref.var.iloc[vidx_normalized]
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas/core/indexing.py", line 1478, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas/core/indexing.py", line 2087, in _getitem_axis
    return self._getbool_axis(key, axis=axis)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas/core/indexing.py", line 1494, in _getbool_axis
    inds, = key.nonzero()
ValueError: too many values to unpack (expected 1)

I've tried several variations of this yet I don't see why your command wouldn't work, it seems like it should do what you intend ..

Note however, I ran:

print(np.any(adata.X.sum(axis=0) == 0)) # True
print(np.any(adata.X.sum(axis=1) == 0))  # False

right after loading the dataset and it still shows True and False, yet if I were to regress out WITHOUT removing cell types via:

keep_cells = [i for i in adata.obs.index if i not in adata_blood.obs.index]
adata = adata[keep_cells, :]

or

adata = adata[adata.obs['blood'] < 0.25, :] # classification score threshold for blood cells 

... the regression will work. However, once I remove, it won't. I could try to remove the 0 columns with R, unless you have another suggestion?

Thank you for any feedback.

[fidelram]

actually, I had the same problem today and solved the problem by removing
zero value columns.

Maybe you want to try:

sc.pp.filter_genes(adata, min_counts=3)

On Fri, Aug 10, 2018 at 4:25 PM jayypaul [e-mail] wrote:

I've tried the command ...

adata = adata[:,adata.X.sum(axis=0) > 0]

.. and it kicks back:

Traceback (most recent call last):
File "", line 1, in
File
"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/anndata/base.py",
line 1205, in getitem
return self._getitem_view(index)
File
"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/anndata/base.py",
line 1209, in _getitem_view
return AnnData(self, oidx=oidx, vidx=vidx, asview=True)
File
"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/anndata/base.py",
line 635, in init
self._init_as_view(X, oidx, vidx)
File
"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/anndata/base.py",
line 661, in _init_as_view
var_sub = adata_ref.var.iloc[vidx_normalized]
File
"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas/core/indexing.py",
line 1478, in getitem
return self._getitem_axis(maybe_callable, axis=axis)
File
"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas/core/indexing.py",
line 2087, in _getitem_axis
return self._getbool_axis(key, axis=axis)
File
"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas/core/indexing.py",
line 1494, in _getbool_axis
inds, = key.nonzero()
ValueError: too many values to unpack (expected 1)

I've tried several variations of this yet I don't see why your command
wouldn't work, it seems like it should do what you intend ..

Note however, I ran :

print(np.any(adata.X.sum(axis=0) == 0)) # True
print(np.any(adata.X.sum(axis=1) == 0)) # False

right after loading the dataset and it still shows True and False, yet if
I were to regress out WITHOUT removing cell types via:

Temp = [i for i in adata.obs.index if i not in adata_blood.obs.index]
adata = adata[Temp,:]

or

adata = adata[adata.obs['blood'] < 0.25, :] # classification score threshold for blood cells

... the regression will work. However, once I remove, it won't. I could
try to remove the 0 columns with R, unless you have another suggestion?

Thank you for any feedback.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/theislab/scanpy/issues/230#issuecomment-412098297,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEu_1bD5gUmq-bUVfAJT2AGlXKXEkOmxks5uPZfTgaJpZM4V0Faw
.

[falexwolf]

Also briefly consider that after

adata = adata[:,adata.X.sum(axis=0) > 0]

adata is a view. Call copy to make it an object with its own data.

adata = adata[:,adata.X.sum(axis=0) > 0].copy()

Why not just use

pp.filter_genes(adata, min_counts=1)

[jayypaul]

Thanks fellas, it worked.

pp.filter_genes(adata, min_counts=1)

Weird, because the other datasets, when loaded fresh have the same pattern:

print(np.any(adata.X.sum(axis=0) == 0)) # True 
print(np.any(adata.X.sum(axis=1) == 0)) # False

Before and after removal of cell types. Yet they still regress out fine.

Anyways, big help.

Cheers

[brianpenghe]

teresting that if I ran your PBMC tutorial without filtering out the non-HVG then I get this error. But I thought these filtering steps in the beginning already eliminated the empty rows and columns?

sc.pp.filter_cells(adata, min_genes=200)
sc.pp.filter_genes(adata, min_cells=3)