fmriprep hangs after finished with no response

[asqwerty666]

I'm trying to run fmriprep in my cluster. First I made a singularity image,

$ singularity build /nas/usr/local/opt/bin/fmriprep-latest.simg docker://poldracklab/fmriprep:latest

Not problem at all. I got the 20.0.2 version. Then I looked at https://gist.github.com/oesteban/5947d28caf6c3750e0a2f2aa09102702 and choose the proper configuration in order to make and sbatch script

!/bin/bash

SBATCH -J fmriprep_mopead

SBATCH --time=24:00:00

SBATCH --mail-type=ALL

SBATCH -o /nas/data/mopead/slurm/fmriprep-%j

SBATCH -n 1

SBATCH --cpus-per-task=16

SBATCH --mem-per-cpu=4G

SBATCH -p fast

SBATCH --mail-user=osotolongo

singularity run --cleanenv -B /nas:/nas /nas/usr/local/opt/bin/fmriprep-latest.simg /nas/data/mopead/bids /nas/data/mopead/fmriprep_out participant --participant-label 0002 --ignore slicetiming --skip_bids_validation --fs-license-file /nas/usr/local/opt/freesurfer/license.txt --nthreads 16 --omp-nthreads 8 --mem-mb 30000 --output-spaces T1w MNI152Lin fsnative --use-aroma -w /nas/data/mopead/fmriprep_work

I copied the freesurfer directory to fmriprep_out/freesurfer/sub-0002 and I ran it with sbatch.
A day later I get TIMEOUT from SLURM. However, based on the contents of work and output directories, it seems to me that fmriprep have done the job. I attach the log here because I'm not sure what is happenning.

¿What I'm doing wrong? I don't think is a lack of resources issues since the machine have 64 CPU, 256 GB of RAM and is plenty of disk space in the network storage. I'm stuck here beacuse I can't integrate fmriprep into the cluster this way.

fmriprep-133694.txt

[mgxd]

Have you run recon-all before or is fmriprep running that?

Also i believe you'll have to set --mem to your jobs max memory requirement, otherwise a default will be used.

Looking at the log, there was also a failure:

200316-10:01:16,540 nipype.workflow WARNING:
     [Node] Error on "fmriprep_wf.single_subject_0002_wf.func_preproc_task_rest_dir_PA_sbref_wf.bold_reference_wf.gen_ref" (/nas/data/mopead/fmriprep_work/fmriprep_wf/single_subject_0002_wf/func_preproc_task_rest_dir_PA_sbref_wf/bold_reference_wf/gen_ref)
200316-10:01:16,555 nipype.workflow WARNING:
     [Node] Error on "fmriprep_wf.single_subject_0002_wf.func_preproc_task_rest_dir_AP_sbref_wf.bold_reference_wf.gen_ref" (/nas/data/mopead/fmriprep_work/fmriprep_wf/single_subject_0002_wf/func_preproc_task_rest_dir_AP_sbref_wf/bold_reference_wf/gen_ref)
exception calling callback for <Future at 0x7f43ef101e10 state=finished raised FileNotFoundError>
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 486, in run
    self._get_hashval()
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 538, in _get_hashval
    self._get_inputs()
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 609, in _get_inputs
    self.set_input(key, deepcopy(output_value))
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 302, in set_input
    setattr(self.inputs, parameter, deepcopy(val))
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/traits_extension.py", line 329, in validate
    value = super(File, self).validate(objekt, name, value, return_pathlike=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/traits_extension.py", line 137, in validate
    self.error(objekt, name, str(value))
  File "/usr/local/miniconda/lib/python3.7/site-packages/traits/trait_handlers.py", line 172, in error
    value )
traits.trait_errors.TraitError: The 'moving_image' trait of a _GenerateSamplingReferenceInputSpec instance must be a pathlike object or string representing an existing file, but a value of '/' <class 'str'> was specified.

Error setting node input:
Node: gen_ref
input: moving_image
results_file: /nas/data/mopead/fmriprep_work/fmriprep_wf/single_subject_0002_wf/func_preproc_task_rest_dir_AP_sbref_wf/bold_split/result_bold_split.pklz
value: /

[asqwerty666]

Thanks very much for your answer. I ran recon-all previously an copied the entire subject directory into fmriprep_out/freesurfer/sub-0002. I'm almost sure fmriprep find the freesurfer data.

I will look into those errors, I think the problem is there. I will report back!

[asqwerty666]

Thanks again. I removed the error causing files from the BIDS directory and now the whole proccess finished in 2 hours aprox.