Fmriprep: Crash at 'validate' after finished bold_reg_wf.fsl_bbr_wf.flt_bbr_init

[hcp4715]

Dear experts,
I am new to fmriprep. I am trying the latest version of fmriprep on the neurodebian VirtualBox. I assigned 4 cores and 10, 000 M RAM for the virtue machine, so theoretically it should enough for fmriprep. I tried to preprocess a single participant's data using fMRIprep, which include one T1w scan, one resting fMRI, and one task fMRI.

I used following code to do the preprocessing:

sudo docker run -it --rm -v ~/host/coiltest/bids:/data:ro -v ~/host/coiltest/prep:/out -v ~/host/freesurf_license:/opt/freesurfer/license.txt poldracklab/fmriprep:latest /data /out/ participant --participant-label sub-02 --fs-license-file /opt/freesurfer/license.txt --fs-no-reconall

After about one hour, I encountered the following error:

[Node] Finished "fmriprep_wf.single_subject_02_wf.func_preproc_task_rest_wf.bold_reg_wf.fsl_bbr_wf.flt_bbr_init".
180502-19:16:44,679 workflow INFO:
     [Node] Finished "fmriprep_wf.single_subject_02_wf.func_preproc_task_test_wf.bold_bold_trans_wf.merge".
180502-19:16:45,916 workflow INFO:
     [Node] Setting-up "fmriprep_wf.single_subject_02_wf.func_preproc_task_test_wf.bold_confounds_wf.non_steady_state" in "/root/src/fmriprep/work/fmriprep_wf/single_subject_02_wf/func_preproc_task_test_wf/bold_confounds_wf/non_steady_state".
180502-19:16:45,919 workflow INFO:
     [Node] Running "non_steady_state" ("niworkflows.nipype.algorithms.confounds.NonSteadyStateDetector")
180502-19:16:46,793 workflow INFO:
     [Node] Setting-up "fmriprep_wf.single_subject_02_wf.func_preproc_task_test_wf.bold_bold_trans_wf.bold_reference_wf.validate" in "/root/src/fmriprep/work/fmriprep_wf/single_subject_02_wf/func_preproc_task_test_wf/bold_bold_trans_wf/bold_reference_wf/validate".
180502-19:16:46,794 workflow INFO:
     [Node] Running "validate" ("fmriprep.interfaces.images.ValidateImage")
180502-19:17:10,460 workflow WARNING:
     [Node] Error on "fmriprep_wf.single_subject_02_wf.func_preproc_task_test_wf.bold_bold_trans_wf.bold_reference_wf.validate" (/root/src/fmriprep/work/fmriprep_wf/single_subject_02_wf/func_preproc_task_test_wf/bold_bold_trans_wf/bold_reference_wf/validate)
Traceback (most recent call last):
  File "/usr/local/miniconda/bin/fmriprep", line 11, in <module>
    load_entry_point('fmriprep==1.0.11', 'console_scripts', 'fmriprep')()
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/cli/run.py", line 274, in main
    fmriprep_wf.run(**plugin_settings)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/workflows.py", line 602, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/base.py", line 168, in run
    self._clean_queue(jobid, graph, result=result))
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/base.py", line 227, in _clean_queue
    raise RuntimeError("".join(result['traceback']))
RuntimeError: Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py", line 68, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 487, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 571, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 650, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base/core.py", line 516, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/images.py", line 461, in _run_interface
    img.to_filename(out_fname)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nibabel/filebasedimages.py", line 334, in to_filename
    self.to_file_map()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nibabel/analyze.py", line 1053, in to_file_map
    data = self.get_data()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nibabel/dataobj_images.py", line 202, in get_data
    data = np.asanyarray(self._dataobj)
  File "/usr/local/miniconda/lib/python3.6/site-packages/numpy/core/numeric.py", line 583, in asanyarray
    return array(a, dtype, copy=False, order=order, subok=True)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nibabel/arrayproxy.py", line 291, in __array__
    return apply_read_scaling(raw_data, self._slope, self._inter)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nibabel/volumeutils.py", line 967, in apply_read_scaling
    arr = arr + inter
MemoryError

Also, this data is validated by http://incf.github.io/bids-validator/#

Looking forward to help.

[effigies]

How large are your functional files (compressed and uncompressed)? It looks like it had to modify the headers and the process of loading and resaving the image exceeded the memory bounds.

[hcp4715]

Hi, Chris,
Thanks for your reply.
All data were transformed as .nii.gz format
resting fMRI: ~ 192 MB;
task fMRI: ~ 370 MB;

If this is the case, is there an available solution for me?

[oesteban]

I'd say 10GB is probably too low. But you can try the --mem_mb option, something like --mem_mb 8000.

[hcp4715]

Hi, Oscar,
Thanks, I guess you are right, 10 GB is too low.

I tried:

sudo docker run -it --rm -v ~/host/coiltest/bids:/data:ro -v ~/host/coiltest/prep:/out -v ~/host/freesurf_license:/opt/freesurfer/license.txt poldracklab/fmriprep:latest /data /out/ participant --participant-label sub-02 --fs-license-file /opt/freesurfer/license.txt --fs-no-reconall --mem_mb 9000

But it seems still have memory error:

[Node] Running "_fsl_to_lta1" ("niworkflows.nipype.interfaces.freesurfer.utils.LTAConvert"), a CommandLine Interface with command:
lta_convert --infsl /root/src/fmriprep/work/fmriprep_wf/single_subject_02_wf/func_preproc_task_rest_wf/bold_reg_wf/fsl_bbr_wf/flt_bbr_init/uni_xform_masked_flirt.mat --outlta /root/src/fmriprep/out.lta --src /root/src/fmriprep/work/fmriprep_wf/single_subject_02_wf/func_preproc_task_rest_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/apply_mask/uni_xform_masked.nii.gz --trg /root/src/fmriprep/work/fmriprep_wf/single_subject_02_wf/anat_preproc_wf/skullstrip_ants_wf/t1_skull_strip/highres001_BrainExtractionBrain.nii.gz
180503-09:53:11,991 workflow INFO:
     [Node] Finished "_fsl_to_lta1".
180503-09:53:13,705 workflow INFO:
     [Node] Setting-up "fmriprep_wf.single_subject_02_wf.func_preproc_task_test_wf.bold_confounds_wf.signals" in "/root/src/fmriprep/work/fmriprep_wf/single_subject_02_wf/func_preproc_task_test_wf/bold_confounds_wf/signals".
180503-09:53:13,728 workflow INFO:
     [Node] Running "signals" ("niworkflows.nipype.interfaces.nilearn.SignalExtraction")
180503-09:53:58,282 workflow WARNING:
     [Node] Error on "fmriprep_wf.single_subject_02_wf.func_preproc_task_test_wf.bold_confounds_wf.signals" (/root/src/fmriprep/work/fmriprep_wf/single_subject_02_wf/func_preproc_task_test_wf/bold_confounds_wf/signals)
Traceback (most recent call last):
  File "/usr/local/miniconda/bin/fmriprep", line 11, in <module>
    load_entry_point('fmriprep==1.0.11', 'console_scripts', 'fmriprep')()
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/cli/run.py", line 274, in main
    fmriprep_wf.run(**plugin_settings)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/workflows.py", line 602, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/base.py", line 168, in run
    self._clean_queue(jobid, graph, result=result))
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/base.py", line 227, in _clean_queue
    raise RuntimeError("".join(result['traceback']))
RuntimeError: Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py", line 68, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 487, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 571, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 650, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base/core.py", line 516, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/nilearn.py", line 102, in _run_interface
    signals.append(masker.fit_transform(self.inputs.in_file))
  File "/usr/local/miniconda/lib/python3.6/site-packages/nilearn/input_data/nifti_maps_masker.py", line 213, in fit_transform
    return self.fit().transform(imgs, confounds=confounds)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nilearn/input_data/base_masker.py", line 176, in transform
    return self.transform_single_imgs(imgs, confounds)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nilearn/input_data/nifti_maps_masker.py", line 319, in transform_single_imgs
    verbose=self.verbose)
  File "/usr/local/miniconda/lib/python3.6/site-packages/sklearn/externals/joblib/memory.py", line 283, in __call__
    return self.func(*args, **kwargs)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nilearn/input_data/base_masker.py", line 98, in filter_and_extract
    memory_level=memory_level)(imgs)
  File "/usr/local/miniconda/lib/python3.6/site-packages/sklearn/externals/joblib/memory.py", line 283, in __call__
    return self.func(*args, **kwargs)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nilearn/input_data/nifti_maps_masker.py", line 29, in __call__
    mask_img=self._resampled_mask_img_)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nilearn/regions/signal_extraction.py", line 270, in img_to_signals_maps
    data[maps_mask, :])[0].T
  File "/usr/local/miniconda/lib/python3.6/site-packages/scipy/linalg/basic.py", line 894, in lstsq
    iwork, cond, False, False)
MemoryError

[effigies]

What version of fmriprep are you using? Also, I would try the above command with the --low-mem flag.

[hcp4715]

Hi, Chris,
Thanks。
I used the latest fmriprep.
I almost give up. but run it on my pc again. will give you feedback.

same as above:

[Node] Finished "_fsl_to_lta1".
180503-21:48:10,830 workflow INFO:
     [Node] Setting-up "fmriprep_wf.single_subject_02_wf.func_preproc_task_test_wf.bold_confounds_wf.signals" in "/root/src/fmriprep/work/fmriprep_wf/single_subject_02_wf/func_preproc_task_test_wf/bold_confounds_wf/signals".
180503-21:48:10,832 workflow INFO:
     [Node] Running "signals" ("niworkflows.nipype.interfaces.nilearn.SignalExtraction")
180503-21:48:39,956 workflow WARNING:
     [Node] Error on "fmriprep_wf.single_subject_02_wf.func_preproc_task_test_wf.bold_confounds_wf.signals" (/root/src/fmriprep/work/fmriprep_wf/single_subject_02_wf/func_preproc_task_test_wf/bold_confounds_wf/signals)
Traceback (most recent call last):
  File "/usr/local/miniconda/bin/fmriprep", line 11, in <module>
    load_entry_point('fmriprep==1.0.11', 'console_scripts', 'fmriprep')()
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/cli/run.py", line 274, in main
    fmriprep_wf.run(**plugin_settings)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/workflows.py", line 602, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/base.py", line 168, in run
    self._clean_queue(jobid, graph, result=result))
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/base.py", line 227, in _clean_queue
    raise RuntimeError("".join(result['traceback']))
RuntimeError: Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py", line 68, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 487, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 571, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 650, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base/core.py", line 516, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/nilearn.py", line 102, in _run_interface
    signals.append(masker.fit_transform(self.inputs.in_file))
  File "/usr/local/miniconda/lib/python3.6/site-packages/nilearn/input_data/nifti_maps_masker.py", line 213, in fit_transform
    return self.fit().transform(imgs, confounds=confounds)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nilearn/input_data/base_masker.py", line 176, in transform
    return self.transform_single_imgs(imgs, confounds)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nilearn/input_data/nifti_maps_masker.py", line 319, in transform_single_imgs
    verbose=self.verbose)
  File "/usr/local/miniconda/lib/python3.6/site-packages/sklearn/externals/joblib/memory.py", line 283, in __call__
    return self.func(*args, **kwargs)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nilearn/input_data/base_masker.py", line 98, in filter_and_extract
    memory_level=memory_level)(imgs)
  File "/usr/local/miniconda/lib/python3.6/site-packages/sklearn/externals/joblib/memory.py", line 283, in __call__
    return self.func(*args, **kwargs)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nilearn/input_data/nifti_maps_masker.py", line 29, in __call__
    mask_img=self._resampled_mask_img_)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nilearn/regions/signal_extraction.py", line 270, in img_to_signals_maps
    data[maps_mask, :])[0].T
  File "/usr/local/miniconda/lib/python3.6/site-packages/scipy/linalg/basic.py", line 894, in lstsq
    iwork, cond, False, False)
MemoryError

[oesteban]

Okay, the only thing left to try is --nthreads 1, am I wrong @effigies ? Otherwise, I guess that 12 GB RAM is a hard lower bound for FMRIPREP to run.

[oesteban]

Hi @hcp4715, is this still an issue? We will release 1.1.1 as soon as possible (hopefully later today) and it should alleviate these memory issues. Please let us know if that worked for you.

[oesteban]

Hi @hcp4715 we have just released 1.1.3 which includes some new mechanisms to prevent this problem. Please reopen if you keep experiencing these memory errors with the new version.