Fmriprep: Multi echo: tCompCor - numpy.linalg.linalg.LinAlgError: SVD did not converge

[gb4944]

Hello all,
An older error I saw in a previous solved issue seems to have reappeared when I tried to run fmriprep on a multi echo dataset. Version 1.2.3 previously ran on the same data without any errors, but when I updated to 1.2.4 today, I got an error saying that SVD did not converge. Previous work directories were removed, and outputs were moved elsewhere before running fmriprep 1.2.4.
The full output is as follows:

[Node] Error on "fmriprep_wf.single_subject_4154_wf.func_preproc_task_co_run_2_echo_1_wf.bold_confounds_wf.tcompcor" (/data/work/fmriprep_wf/single_subject_4154_wf/func_preproc_task_co_run_2_echo_1_wf/bold_confounds_wf/tcompcor)
181204-15:14:53,737 nipype.workflow ERROR:
     Node tcompcor failed to run on host pavlov.amc.nl.
181204-15:14:53,743 nipype.workflow ERROR:
     Saving crash info to /data/fmriprep/sub-4154/log/20181204-143741_072fe7eb-6c6e-4559-aafd-94fe1bdf66a4/crash-20181204-151453-gbrunner-tcompcor-7402206b-78a1-4ed0-b54e-53fe7c787030.txt
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 473, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 557, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 637, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 511, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/patches.py", line 51, in _run_interface
    runtime = super(RobustTCompCor, self)._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/algorithms/confounds.py", line 539, in _run_interface
    self.inputs.pre_filter, degree, self.inputs.high_pass_cutoff, TR)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/algorithms/confounds.py", line 1189, in compute_noise_components
    u, _, _ = linalg.svd(M, full_matrices=False)
  File "/usr/local/miniconda/lib/python3.6/site-packages/scipy/linalg/decomp_svd.py", line 132, in svd
    raise LinAlgError("SVD did not converge")
numpy.linalg.linalg.LinAlgError: SVD did not converge

Preprocessing did not finish successfully. Errors occurred while processing data from participants: 4154 (1). Check the HTML reports for details.
Traceback (most recent call last):
  File "/usr/local/miniconda/bin/fmriprep", line 11, in <module>
    sys.exit(main())
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/cli/run.py", line 431, in main
    fmriprep_wf.run(**plugin_settings)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/workflows.py", line 597, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/base.py", line 162, in run
    self._clean_queue(jobid, graph, result=result))
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/base.py", line 225, in _clean_queue
    raise RuntimeError("".join(result['traceback']))
RuntimeError: Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 473, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 557, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 637, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 511, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/patches.py", line 51, in _run_interface
    runtime = super(RobustTCompCor, self)._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/algorithms/confounds.py", line 539, in _run_interface
    self.inputs.pre_filter, degree, self.inputs.high_pass_cutoff, TR)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/algorithms/confounds.py", line 1189, in compute_noise_components
    u, _, _ = linalg.svd(M, full_matrices=False)
  File "/usr/local/miniconda/lib/python3.6/site-packages/scipy/linalg/decomp_svd.py", line 132, in svd
    raise LinAlgError("SVD did not converge")
numpy.linalg.linalg.LinAlgError: SVD did not converge

Sentry is attempting to send 3 pending error messages
Waiting up to 2.0 seconds
Press Ctrl-C to quit

[shirareznik]

Facing the same issue (multi-band dataset).
Any planning information would be appreciated.
Many thanks!

[effigies]

@shirareznik Is your dataset also multiecho? Multiecho handling changed between 1.2.3 and 1.2.4.

@emdupre Do you know if this is expected if we run tCompCor on the optimal combination of echoes? Are *CompCor even valid measures on combined BOLD runs?

[shirareznik]

@effigies ,
Yes, it is.
Thanks!

[effigies]

Looking at #1010, it may be that we should skip CompCor for multiecho datasets.

@shirareznik @gb4944 Just as a couple data points, were either of you intend to use the CompCor regressors?

[gb4944]

@shirareznik @gb4944 Just as a couple data points, were either of you intend to use the CompCor regressors?

Thanks for the response! ME-ICA would be preferable for noise correction, but I couldn't find any options regarding it in the latest version of the fmriprep documentation.

[effigies]

No, it is not currently implemented. The question is whether CompCor is useful for multiecho data, in its absence.

We're also happy to get contributions from people with more multiecho experience in implementing MEICA, of course.

[emdupre]

Hi,

Thanks for reporting these issues ! Just to confirm, is this affecting all multi-echo runs, or only specific ones ? I've run several datasets myself locally and not encountered this error, so just trying to understand why !

That being said, I would discourage _interpreting_ tCompCor measures from the optimal combination. This was the update in 1.2.4, that we switched from individual echos returned by default to the optimal combination. I'm not sure if this made it into the release notes, but hopefully the documentation is clear !

I would actually discourage implementing MEICA denoising just yet, since there is some confusion across the field about which specific component selection criteria to adopt. That being said, we're having those discussions in the tedana repo and would love your input !!

[gb4944]

@effigies I see, that explains that! I was not intending on using tCompCorr in place of it.
@emdupre That's good to know, thank you! I've only tested a subset of subjects the dataset, but the error appeared for all of them when processing the first functional run.

[effigies]

@gb4944 Thanks for the response. Just to be sure, your derivatives otherwise look sensible? You should be getting reports and derivatives that don't depend on the confounds, and the reports should show the masks of voxels passed to CompCor. With SVD failures, the main thing we can control is whether we're giving sensible data to the process; if the data is good and it's still just failing, all we can try to do is handle that case more gracefully. @shirareznik Same questions for you.

I would propose that we catch LinAlgErrors in CompCor interfaces, and return columns of NaNs, producing a warning that SVD did not converge.

It sounds like we don't want to disable CompCor altogether with ME, at present, as we don't have MEICA to replace it, and there's no guarantee that users aren't using it. I also found this Power, et al. paper from February that seems to indicate from a cursory reading that there's value to CompCor in ME datasets.

cc @oesteban @chrisfilo

[emdupre]

Yes, I'd agree with that assessment ! There is definitely value, it's just
different from what I think most ME users are looking for if we're coming
from ME-informed denoising.

[chrisgorgo]

+1 for graceful handling non converging SVD (since there is no known solution). There were 12 such errors reported in the past couple of weeks (https://sentry.io/poldracklab/fmriprep/issues/774850143/?query=is%3Aunresolved)

[shirareznik]

Thanks for the response. Just to be sure, your derivatives otherwise look sensible?

I Will check afterwards and report.

I would propose that we catch LinAlgErrors in CompCor interfaces, and return columns of NaNs, producing a warning that SVD did not converge.

This is the best option I can think about.

It sounds like we don't want to disable CompCor altogether with ME, at present, as we don't have MEICA to replace it, and there's no guarantee that users aren't using it. I also found this Power, et al. paper from February that seems to indicate from a cursory reading that there's value to CompCor in ME datasets.

Many thanks for the reference.
I would not disable it for ME on the meantime- we might use it.

Many thanks and sorry for the delay.

[gb4944]

@effigies Thank you for the suggestion! I re-ran fmriprep for one subject to look at the rest of the derivatives, and it appears that the input to CompCorr doesn't look entirely correct, especially in the top right corner of this image. No errors occurred using the same files with 1.2.3.

[emdupre]

Just to clarify: the reason this is new with 1.2.4 is that effectively you are processing a different file, since you are deriving CompCor measures from the optimal combination rather than any individual echo. If you would like to process individual echos instead, I've added the --echo-idx flag.

But what you're seeing is actually an older issue, see: https://neurostars.org/t/weird-brain-mask-in-fmriprep-report/1812

It'd be great if you could confirm this looks right in your own data by looking at the brainmask.nii.gz and ensuring its shape is sensible.

[gb4944]

@emdupre Good point, the grid pattern is not present in the brain mask. Other parts that are missing or outside of the brain seem consistent with the image from the report that I posted.

[effigies]

@gb4944 Your aCompCor (magenta) and tCompCor (blue) masks look reasonable. I note that your BOLD mask seems to be cropping rather more than it should... This image is from the 1.2.4 run?

[gb4944]

@effigies Yes, it's from 1.2.4.

[effigies]

Could you go ahead and open a new issue for that? Also, are you using single-band references, or just the BOLD series?

[gb4944]

@gb4944 Just the BOLD series.
I'll open a new issue. Thanks for the help!

[effigies]

@gb4944 @shirareznik If either of you is running with fmriprep-docker, could you do the following, to run in developer mode?

git clone https://github.com/effigies/fmriprep.git
git clone https://github.com/effigies/nipype.git
pushd fmriprep; git checkout fix/compcor_nans; popd
pushd nipype; git checkout enh/compcor_svd_failure; popd

pip install --upgrade fmriprep-docker
fmriprep-docker -p nipype -f fmriprep [YOUR NORMAL FMRIPREP OPTIONS]

[shirareznik]

Of course- I'll run it soon and update you.

On Thu, Dec 13, 2018 at 2:30 PM Chris Markiewicz [e-mail]
wrote:

@gb4944 https://github.com/gb4944 @shirareznik
https://github.com/shirareznik If either of you is running with
fmriprep-docker, could you do the following, to run in developer mode?

git clone https://github.com/effigies/fmriprep.git
git clone https://github.com/effigies/nipype.gitpushd fmriprep; git checkout fix/compcor_nans; popdpushd nipype; git checkout enh/compcor_svd_failure; popd

pip install --upgrade fmriprep-docker
fmriprep-docker -p nipype -f fmriprep [YOUR NORMAL FMRIPREP OPTIONS]

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/poldracklab/fmriprep/issues/1433#issuecomment-446951901,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABMYijlHACh4huiNpr0k4QQ6hftj1kJ5ks5u4khqgaJpZM4ZBCxe
.

[effigies]

@gb4944 @shirareznik Hi, any updates? I can't off the top of my head think of a way of generating a synthetic dataset that will consistently fail to converge with SVD, so if you could verify that this produces NaN confounds and doesn't crash, that would be a good test.

[gb4944]

@effigies Apologies for the delay, unfortunately I don't think I have the permissions required to run this on the server where the data is stored. I will ask about it and update you if I can get it to run.

[effigies]

@gb4944 No worries. If you're running it on a cluster, it'll be difficult. We can always just try it and see if the next release fixes your issue.

[oesteban]

Hi @gb4944, @shirareznik have you experienced this again with latest versions of fMRIPrep? Could I ask you to try on the same failing data, but with version 1.4.0 (the latest)?

Thanks

[shirareznik]

Hi,

Sorry for being "slow"- I'll test this during this week and update (in case
I'll be overloaded, it might take another day or two).

I've read the change log yesterday and was very curious to try it already!

Many thanks

On Tue, May 21, 2019, 21:57 Oscar Esteban [e-mail] wrote:

Hi @gb4944 https://github.com/gb4944, @shirareznik
https://github.com/shirareznik have you experienced this again with
latest versions of fMRIPrep? Could I ask you to try on the same failing
data, but with version 1.4.0 (the latest)?

Thanks

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/poldracklab/fmriprep/issues/1433?email_source=notifications&email_token=AAJRRCRPATPL7WOLEAV7KM3PWRAZ7A5CNFSM4GIEFRPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODV43J2Y#issuecomment-494515435,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAJRRCQ7UMDWU5TZG3PWF33PWRAZ7ANCNFSM4GIEFRPA
.