fmriprep-docker hanging

[danielkimmel]

I'm using fmriprep-docker.py against a validated BIDS dataset. The program appears to work for a number of steps (i.e., many lines output to terminal, high CPU usage, some output files written), then stops printing new output to the terminal and stops using CPU cycles. There is no error message. It will stay in this hung state for >20 hours until I interrupt the process, at which point some python errors will generate.

I've attached a text file with the terminal output and a screenshot of the partially completed output files.
I'm using fmriprep 1.0.8 in a docker container.

Any suggestions?
Thanks in advance!
fmriprep-docker-terminal-output.txt

[effigies]

Could you add the --debug flag to your command? That should add some status data every couple seconds, which may help identify the issue.

[danielkimmel]

Sure, I'd be happy to. But it takes hours to get to the point at which it hangs, so it'll probably be tomorrow until I have that output. Any other suggestions in the meantime (or to try along with another attempt)?

Thanks!
Daniel

On Apr 10, 2018, at 1:31 PM, Chris Markiewicz [e-mail] wrote:

Could you add the --debug flag to your command? That should add some status data every couple seconds, which may help identify the issue.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/poldracklab/fmriprep/issues/1045#issuecomment-380183858, or mute the thread https://github.com/notifications/unsubscribe-auth/Ae1vFjMOLz5GvT87HfrdE67tSsIyWiRTks5tnOxygaJpZM4TOp-O.

[effigies]

It should get back to the point where it stopped fairly quickly, since you're using a scratch directory. As long as your inputs don't change, that allows you to skip steps that have already been run.

[danielkimmel]

ah, so in my new call, should I pass the same output directory and same scratch directory (-w flag) as for the most recent run that failed?

Thanks,
d

On Apr 10, 2018, at 1:40 PM, Chris Markiewicz [e-mail] wrote:

It should get back to the point where it stopped fairly quickly, since you're using a scratch directory. As long as your inputs don't change, that allows you to skip steps that have already been run.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/poldracklab/fmriprep/issues/1045#issuecomment-380186678, or mute the thread https://github.com/notifications/unsubscribe-auth/Ae1vFl6OtQ5O4AdyJu4tryx6FbROoqf9ks5tnO6PgaJpZM4TOp-O.

[effigies]

Yes.

[danielkimmel]

Thanks, Chris.

I did like you said:
python /Users/Daniel/anaconda2/lib/python2.7/site-packages/fmriprep/fmriprep_docker.py /Users/Daniel/DATA/ContextTask/fMRI/BIDS/ /Users/Daniel/DATA/ContextTask/fMRI/fmriprep/out3/ participant --fs-no-reconall --no-submm-recon --fs-license-file /Users/Daniel/anaconda2/lib/python2.7/site-packages/fmriprep/freesurfer_license.txt -w /Users/Daniel/DATA/ContextTask/fMRI/fmriprep/work3/ --participant_label JG --debug

It's running now, but the terminal output has been the same for the past 10min or so:
180410-17:46:24,332 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_01_wf.bold_preproc_report_wf.bold_rpt".

I say this because I think you mentioned that with the "--debug" flag, we'd get a status update every few seconds. Am I doing something wrong?

Thanks again,
d

On Apr 10, 2018, at 1:52 PM, Chris Markiewicz [e-mail] wrote:

Yes.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/poldracklab/fmriprep/issues/1045#issuecomment-380190533, or mute the thread https://github.com/notifications/unsubscribe-auth/Ae1vFrAur1I6OjAbzV01eLo-ou4YiLS_ks5tnPF1gaJpZM4TOp-O.

[effigies]

Hmm. @oesteban Shouldn't --debug turn on the nipype status messages?

[danielkimmel]

Hi Chris and @oesteban https://github.com/oesteban,

I re-ran fmriprep-docker.py with the "--debug" switch and pointing to the same output dir and scratch dir as the prior failed run.

It's been stuck on the last line of the terminal output for a while and the docker process is not claiming any CPU cycles.

Attached is the terminal output and screenshot of the file output.

Any suggestions?

Thanks very much,
Daniel

On Apr 10, 2018, at 2:04 PM, Chris Markiewicz [e-mail] wrote:

Hmm. @oesteban https://github.com/oesteban Shouldn't --debug turn on the nipype status messages?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/poldracklab/fmriprep/issues/1045#issuecomment-380194221, or mute the thread https://github.com/notifications/unsubscribe-auth/Ae1vFnLUr-Jzdyz44M5UJZQsdQ_rm9SCks5tnPQtgaJpZM4TOp-O.

dyn-160-39-134-187:helper Daniel$ python /Users/Daniel/anaconda2/lib/python2.7/site-packages/fmriprep/fmriprep_docker.py /Users/Daniel/DATA/ContextTask/fMRI/BIDS/ /Users/Daniel/DATA/ContextTask/fMRI/fmriprep/out3/ participant --fs-no-reconall --no-submm-recon --fs-license-file /Users/Daniel/anaconda2/lib/python2.7/site-packages/fmriprep/freesurfer_license.txt -w /Users/Daniel/DATA/ContextTask/fMRI/fmriprep/work3/ --participant_label JG --debug
RUNNING: docker run --rm -it -v /Users/Daniel/anaconda2/lib/python2.7/site-packages/fmriprep/freesurfer_license.txt:/opt/freesurfer/license.txt:ro -v /Users/Daniel/DATA/ContextTask/fMRI/BIDS/:/data:ro -v /Users/Daniel/DATA/ContextTask/fMRI/fmriprep/out3/:/out -v /Users/Daniel/DATA/ContextTask/fMRI/fmriprep/work3/:/scratch poldracklab/fmriprep:1.0.8 /data /out participant --fs-no-reconall --no-submm-recon --participant_label JG --debug -w /scratch
180410-17:43:37,82 workflow IMPORTANT:

Running fMRIPREP version 1.0.8:
  * BIDS dataset path: /data.
  * Participant list: ['JG'].
  * Run identifier: 20180410-174337_9cf185b0-34ed-41f2-adc3-5635dc5d60ca.

180410-17:43:37,608 workflow IMPORTANT:
Creating bold processing workflow for "/data/sub-JG/func/sub-JG_task-context_run-01_bold.nii.gz" (0.26 GB / 425 TRs). Memory resampled/largemem=1.03/2.13 GB.
180410-17:43:38,44 workflow WARNING:
No fieldmaps found or they were ignored, building base workflow for dataset /data/sub-JG/func/sub-JG_task-context_run-01_bold.nii.gz.
180410-17:43:38,940 workflow IMPORTANT:
Creating bold processing workflow for "/data/sub-JG/func/sub-JG_task-context_run-02_bold.nii.gz" (0.04 GB / 25 TRs). Memory resampled/largemem=0.16/0.21 GB.
180410-17:43:39,364 workflow WARNING:
No fieldmaps found or they were ignored, building base workflow for dataset /data/sub-JG/func/sub-JG_task-context_run-02_bold.nii.gz.
180410-17:43:40,266 workflow IMPORTANT:
Creating bold processing workflow for "/data/sub-JG/func/sub-JG_task-context_run-03_bold.nii.gz" (0.34 GB / 310 TRs). Memory resampled/largemem=1.35/2.40 GB.
180410-17:43:40,776 workflow WARNING:
No fieldmaps found or they were ignored, building base workflow for dataset /data/sub-JG/func/sub-JG_task-context_run-03_bold.nii.gz.
180410-17:43:41,599 workflow IMPORTANT:
Creating bold processing workflow for "/data/sub-JG/func/sub-JG_task-context_run-04_bold.nii.gz" (0.42 GB / 380 TRs). Memory resampled/largemem=1.66/3.24 GB.
180410-17:43:42,12 workflow WARNING:
No fieldmaps found or they were ignored, building base workflow for dataset /data/sub-JG/func/sub-JG_task-context_run-04_bold.nii.gz.
180410-17:43:42,847 workflow IMPORTANT:
Creating bold processing workflow for "/data/sub-JG/func/sub-JG_task-context_run-05_bold.nii.gz" (0.17 GB / 240 TRs). Memory resampled/largemem=0.68/1.09 GB.
180410-17:43:43,350 workflow WARNING:
No fieldmaps found or they were ignored, building base workflow for dataset /data/sub-JG/func/sub-JG_task-context_run-05_bold.nii.gz.
180410-17:43:58,625 workflow WARNING:
Some nodes exceed the total amount of memory available (8.77GB).
180410-17:44:13,661 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_01_wf.bold_preproc_report_wf.pos_tsnr" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_01_wf/bold_preproc_report_wf/pos_tsnr".
180410-17:44:13,696 workflow INFO:
[Node] Running "pos_tsnr" ("niworkflows.nipype.algorithms.confounds.TSNR")

180410-17:45:53,786 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_01_wf.bold_preproc_report_wf.pos_tsnr".
180410-17:45:56,461 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_01_wf.bold_preproc_report_wf.bold_rpt" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_01_wf/bold_preproc_report_wf/bold_rpt".
180410-17:45:56,490 workflow INFO:
[Node] Running "bold_rpt" ("niworkflows.interfaces.registration.SimpleBeforeAfterRPT")
180410-17:45:56,496 niworkflows INFO:
Report - setting before (/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_01_wf/bold_preproc_report_wf/pos_tsnr/stdev.nii.gz) and after (/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_01_wf/bold_preproc_report_wf/pre_tsnr/stdev.nii.gz) images
180410-17:45:56,496 niworkflows INFO:
Generating visual report
180410-17:45:59,547 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.bold_transform" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_bold_trans_wf/bold_transform".
180410-17:46:00,60 workflow INFO:
[Node] Running "bold_transform" ("fmriprep.interfaces.itk.MultiApplyTransforms")
180410-17:46:24,289 niworkflows INFO:
Successfully created report (/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_01_wf/bold_preproc_report_wf/bold_rpt/report.svg)
180410-17:46:24,332 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_01_wf.bold_preproc_report_wf.bold_rpt".
180410-18:03:24,772 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.bold_transform".
180410-18:03:28,477 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.merge" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_bold_trans_wf/merge".
180410-18:03:28,785 workflow INFO:
[Node] Running "merge" ("fmriprep.interfaces.nilearn.Merge")
180410-18:04:08,568 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.anat_preproc_wf.skullstrip_ants_wf.t1_skull_strip" in "/scratch/fmriprep_wf/single_subject_JG_wf/anat_preproc_wf/skullstrip_ants_wf/t1_skull_strip".
180410-18:04:08,576 workflow INFO:
[Node] Outdated cache found for "fmriprep_wf.single_subject_JG_wf.anat_preproc_wf.skullstrip_ants_wf.t1_skull_strip".
180410-18:04:08,680 workflow INFO:
[Node] Running "t1_skull_strip" ("niworkflows.nipype.interfaces.ants.segmentation.BrainExtraction"), a CommandLine Interface with command:
antsBrainExtraction.sh -a /data/sub-JG/anat/sub-JG_run-01_T1w.nii.gz -m /scratch/fmriprep_wf/single_subject_JG_wf/anat_preproc_wf/skullstrip_ants_wf/t1_skull_strip/T_template0_BrainCerebellumProbabilityMask.nii.gz -e /niworkflows_data/ants_oasis_template_ras/T_template0.nii.gz -z 1 -d 3 -f /niworkflows_data/ants_oasis_template_ras/T_template0_BrainCerebellumRegistrationMask.nii.gz -s nii.gz -k 1 -o highres001_ -q 1 180410-18:05:45,391 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.merge".
180410-18:05:47,964 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_preproc_report_wf.pos_tsnr" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_preproc_report_wf/pos_tsnr".
180410-18:05:47,984 workflow INFO:
[Node] Running "pos_tsnr" ("niworkflows.nipype.algorithms.confounds.TSNR")
180410-18:08:07,170 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_preproc_report_wf.pos_tsnr".
180410-18:08:20,179 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_preproc_report_wf.bold_rpt" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_preproc_report_wf/bold_rpt".
180410-18:08:20,218 workflow INFO:
[Node] Running "bold_rpt" ("niworkflows.interfaces.registration.SimpleBeforeAfterRPT")
180410-18:08:20,221 niworkflows INFO:
Report - setting before (/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_preproc_report_wf/pos_tsnr/stdev.nii.gz) and after (/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_preproc_report_wf/pre_tsnr/stdev.nii.gz) images
180410-18:08:20,221 niworkflows INFO:
Generating visual report
180410-18:08:52,412 niworkflows INFO:
Successfully created report (/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_preproc_report_wf/bold_rpt/report.svg)
180410-18:08:52,566 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_preproc_report_wf.bold_rpt".
180410-18:08:54,988 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_confounds_wf.non_steady_state" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_confounds_wf/non_steady_state".
180410-18:08:55,8 workflow INFO:
[Node] Running "non_steady_state" ("niworkflows.nipype.algorithms.confounds.NonSteadyStateDetector")
180410-18:09:17,36 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_confounds_wf.non_steady_state".
180410-18:09:20,564 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.bold_reference_wf.validate" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_bold_trans_wf/bold_reference_wf/validate".
180410-18:09:20,581 workflow INFO:
[Node] Running "validate" ("fmriprep.interfaces.images.ValidateImage")
180410-18:10:07,96 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.bold_reference_wf.validate".
180410-18:10:09,661 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.bold_reference_wf.gen_ref" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_bold_trans_wf/bold_reference_wf/gen_ref".
180410-18:10:09,676 workflow INFO:
[Node] Running "gen_ref" ("niworkflows.interfaces.registration.EstimateReferenceImage")
180410-18:10:13,675 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.bold_reference_wf.gen_ref".
180410-18:10:15,308 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.n4_mask" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/n4_mask".
180410-18:10:15,329 workflow INFO:
[Node] Running "n4_mask" ("fmriprep.interfaces.nilearn.MaskEPI")
180410-18:10:17,526 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.n4_mask".
180410-18:10:18,420 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.n4_correct" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/n4_correct".
180410-18:10:18,441 workflow INFO:
[Node] Running "n4_correct" ("niworkflows.nipype.interfaces.ants.segmentation.N4BiasFieldCorrection"), a CommandLine Interface with command:
N4BiasFieldCorrection -d 3 --input-image /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_bold_trans_wf/bold_reference_wf/gen_ref/ref_image.nii.gz --mask-image /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/n4_mask/ref_image_mask.nii.gz --output ref_image_corrected.nii.gz
180410-18:10:25,897 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.n4_correct".
180410-18:10:26,583 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.skullstrip_first_pass" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/skullstrip_first_pass".
180410-18:10:26,606 workflow INFO:
[Node] Running "skullstrip_first_pass" ("niworkflows.nipype.interfaces.fsl.preprocess.BET"), a CommandLine Interface with command:
bet /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/n4_correct/ref_image_corrected.nii.gz /root/src/fmriprep/ref_image_corrected_brain.nii.gz -f 0.20 -m
180410-18:10:29,82 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.skullstrip_first_pass".
180410-18:10:30,470 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.skullstrip_first_dilate" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/skullstrip_first_dilate".
180410-18:10:30,495 workflow INFO:
[Node] Running "skullstrip_first_dilate" ("niworkflows.nipype.interfaces.fsl.maths.DilateImage"), a CommandLine Interface with command:
fslmaths -dt char /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/skullstrip_first_pass/ref_image_corrected_brain_mask.nii.gz -kernel sphere 6.0000 -dilF /root/src/fmriprep/ref_image_corrected_brain_mask_dil.nii.gz
180410-18:10:40,654 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.skullstrip_first_dilate".
180410-18:10:42,328 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.skullstrip_first_mask" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/skullstrip_first_mask".
180410-18:10:42,344 workflow INFO:
[Node] Running "skullstrip_first_mask" ("niworkflows.nipype.interfaces.fsl.maths.ApplyMask"), a CommandLine Interface with command:
fslmaths /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/skullstrip_first_pass/ref_image_corrected_brain.nii.gz -mas /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/skullstrip_first_dilate/ref_image_corrected_brain_mask_dil.nii.gz /root/src/fmriprep/ref_image_corrected_brain_masked.nii.gz
180410-18:10:42,643 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.skullstrip_first_mask".
180410-18:10:44,295 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.unifize" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/unifize".
180410-18:10:44,323 workflow INFO:
[Node] Running "unifize" ("niworkflows.nipype.interfaces.afni.utils.Unifize"), a CommandLine Interface with command:
3dUnifize -clfrac 0.2 -rbt 18.3 65.0 90.0 -prefix uni.nii.gz -T2 -input /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/unifize/ref_image_corrected_brain_masked.nii.gz
180410-18:10:53,280 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.unifize".
180410-18:10:55,192 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.fixhdr_unifize" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/fixhdr_unifize".
180410-18:10:55,203 workflow INFO:
[Node] Running "fixhdr_unifize" ("niworkflows.interfaces.utils.CopyXForm")
180410-18:10:55,378 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.fixhdr_unifize".
180410-18:10:56,323 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.skullstrip_second_pass" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/skullstrip_second_pass".
180410-18:10:56,345 workflow INFO:
[Node] Running "skullstrip_second_pass" ("niworkflows.nipype.interfaces.afni.preprocess.Automask"), a CommandLine Interface with command:
3dAutomask -apply_prefix uni_xform_masked.nii.gz -dilate 1 -prefix uni_xform_mask.nii.gz /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/skullstrip_second_pass/uni_xform.nii.gz
180410-18:10:57,41 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.skullstrip_second_pass".
180410-18:10:59,75 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.fixhdr_skullstrip2" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/fixhdr_skullstrip2".
180410-18:10:59,114 workflow INFO:
[Node] Running "fixhdr_skullstrip2" ("niworkflows.interfaces.utils.CopyXForm")
180410-18:10:59,252 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.fixhdr_skullstrip2".
180410-18:11:00,480 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.combine_masks" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/combine_masks".
180410-18:11:00,500 workflow INFO:
[Node] Running "combine_masks" ("niworkflows.nipype.interfaces.fsl.maths.BinaryMaths"), a CommandLine Interface with command:
fslmaths /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/skullstrip_first_pass/ref_image_corrected_brain_mask.nii.gz -mul /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/fixhdr_skullstrip2/uni_xform_mask_xform.nii.gz /root/src/fmriprep/ref_image_corrected_brain_mask_maths.nii.gz
180410-18:11:00,810 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.combine_masks".
180410-18:11:02,297 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_confounds_wf.dvars" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_confounds_wf/dvars".
180410-18:11:02,315 workflow INFO:
[Node] Running "dvars" ("niworkflows.nipype.algorithms.confounds.ComputeDVARS")
180410-18:12:42,77 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_confounds_wf.dvars".
180410-18:12:43,397 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.apply_mask" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/apply_mask".
180410-18:12:43,427 workflow INFO:
[Node] Running "apply_mask" ("niworkflows.nipype.interfaces.fsl.maths.ApplyMask"), a CommandLine Interface with command:
fslmaths /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/fixhdr_unifize/uni_xform.nii.gz -mas /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/combine_masks/ref_image_corrected_brain_mask_maths.nii.gz /root/src/fmriprep/uni_xform_masked.nii.gz
180410-18:12:44,2 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.apply_mask".
180410-18:34:02,567 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.anat_preproc_wf.skullstrip_ants_wf.t1_skull_strip".
180410-18:34:08,259 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.bold_transform" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_transform".
180410-18:34:08,829 workflow INFO:
[Node] Running "bold_transform" ("fmriprep.interfaces.itk.MultiApplyTransforms")
180410-18:57:32,546 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.bold_transform".
180410-18:57:37,271 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.merge" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/merge".
180410-18:57:37,827 workflow INFO:
[Node] Running "merge" ("fmriprep.interfaces.nilearn.Merge")
180410-18:57:38,424 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_05_wf.bold_reg_wf.gen_ref" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_05_wf/bold_reg_wf/gen_ref".
180410-18:57:38,433 workflow INFO:
[Node] Outdated cache found for "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_05_wf.bold_reg_wf.gen_ref".
180410-18:57:38,495 workflow INFO:
[Node] Running "gen_ref" ("niworkflows.interfaces.utils.GenerateSamplingReference")
180410-18:57:38,636 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_05_wf.bold_reg_wf.fsl_bbr_wf.flt_bbr_init" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_05_wf/bold_reg_wf/fsl_bbr_wf/flt_bbr_init".
180410-18:57:38,646 workflow INFO:
[Node] Outdated cache found for "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_05_wf.bold_reg_wf.fsl_bbr_wf.flt_bbr_init".
180410-18:57:38,764 workflow INFO:
[Node] Running "flt_bbr_init" ("niworkflows.interfaces.registration.FLIRTRPT"), a CommandLine Interface with command:
flirt -in /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_05_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/apply_mask/uni_xform_masked.nii.gz -ref /scratch/fmriprep_wf/single_subject_JG_wf/anat_preproc_wf/skullstrip_ants_wf/t1_skull_strip/highres001_BrainExtractionBrain.nii.gz -out uni_xform_masked_flirt.nii.gz -omat uni_xform_masked_flirt.mat -dof 6 -usesqform
180410-18:57:41,486 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_05_wf.bold_reg_wf.gen_ref".
180410-18:57:45,712 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_reg_wf.fsl_bbr_wf.flt_bbr_init" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_reg_wf/fsl_bbr_wf/flt_bbr_init".
180410-18:57:45,737 workflow INFO:
[Node] Outdated cache found for "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_reg_wf.fsl_bbr_wf.flt_bbr_init".
180410-18:57:45,848 workflow INFO:
[Node] Running "flt_bbr_init" ("niworkflows.interfaces.registration.FLIRTRPT"), a CommandLine Interface with command:
flirt -in /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/apply_mask/uni_xform_masked.nii.gz -ref /scratch/fmriprep_wf/single_subject_JG_wf/anat_preproc_wf/skullstrip_ants_wf/t1_skull_strip/highres001_BrainExtractionBrain.nii.gz -out uni_xform_masked_flirt.nii.gz -omat uni_xform_masked_flirt.mat -dof 6 -usesqform
180410-18:58:09,625 niworkflows INFO:
Report - setting fixed (/scratch/fmriprep_wf/single_subject_JG_wf/anat_preproc_wf/skullstrip_ants_wf/t1_skull_strip/highres001_BrainExtractionBrain.nii.gz) and moving (/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_05_wf/bold_reg_wf/fsl_bbr_wf/flt_bbr_init/uni_xform_masked_flirt.nii.gz) images
180410-18:58:09,626 niworkflows INFO:
Generating visual report
180410-18:58:16,466 niworkflows INFO:
Report - setting fixed (/scratch/fmriprep_wf/single_subject_JG_wf/anat_preproc_wf/skullstrip_ants_wf/t1_skull_strip/highres001_BrainExtractionBrain.nii.gz) and moving (/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_reg_wf/fsl_bbr_wf/flt_bbr_init/uni_xform_masked_flirt.nii.gz) images
180410-18:58:16,466 niworkflows INFO:
Generating visual report
180410-18:59:19,235 niworkflows INFO:
Successfully created report (/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_05_wf/bold_reg_wf/fsl_bbr_wf/flt_bbr_init/report.svg)
180410-18:59:19,347 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_05_wf.bold_reg_wf.fsl_bbr_wf.flt_bbr_init".
180410-18:59:22,795 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_reg_wf.gen_ref" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_reg_wf/gen_ref".
180410-18:59:22,800 workflow INFO:
[Node] Outdated cache found for "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_reg_wf.gen_ref".
180410-18:59:22,873 workflow INFO:
[Node] Running "gen_ref" ("niworkflows.interfaces.utils.GenerateSamplingReference")
180410-18:59:25,134 niworkflows INFO:
Successfully created report (/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_reg_wf/fsl_bbr_wf/flt_bbr_init/report.svg)
180410-18:59:25,249 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_reg_wf.fsl_bbr_wf.flt_bbr_init".
180410-18:59:26,584 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_reg_wf.gen_ref".
180410-18:59:28,59 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_reg_wf.fsl_bbr_wf.flt_bbr_init" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_reg_wf/fsl_bbr_wf/flt_bbr_init".
180410-18:59:28,63 workflow INFO:
[Node] Outdated cache found for "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_reg_wf.fsl_bbr_wf.flt_bbr_init".
180410-18:59:28,180 workflow INFO:
[Node] Running "flt_bbr_init" ("niworkflows.interfaces.registration.FLIRTRPT"), a CommandLine Interface with command:
flirt -in /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/apply_mask/uni_xform_masked.nii.gz -ref /scratch/fmriprep_wf/single_subject_JG_wf/anat_preproc_wf/skullstrip_ants_wf/t1_skull_strip/highres001_BrainExtractionBrain.nii.gz -out uni_xform_masked_flirt.nii.gz -omat uni_xform_masked_flirt.mat -dof 6 -usesqform
180410-18:59:29,611 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_reg_wf.gen_ref" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_reg_wf/gen_ref".
180410-18:59:29,619 workflow INFO:
[Node] Outdated cache found for "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_reg_wf.gen_ref".
180410-18:59:29,704 workflow INFO:
[Node] Running "gen_ref" ("niworkflows.interfaces.utils.GenerateSamplingReference")
180410-18:59:32,522 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_reg_wf.gen_ref".
180410-18:59:35,669 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_02_wf.bold_reg_wf.fsl_bbr_wf.flt_bbr_init" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_02_wf/bold_reg_wf/fsl_bbr_wf/flt_bbr_init".
180410-18:59:35,678 workflow INFO:
[Node] Outdated cache found for "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_02_wf.bold_reg_wf.fsl_bbr_wf.flt_bbr_init".
180410-18:59:35,861 workflow INFO:
[Node] Running "flt_bbr_init" ("niworkflows.interfaces.registration.FLIRTRPT"), a CommandLine Interface with command:
flirt -in /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_02_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/apply_mask/uni_xform_masked.nii.gz -ref /scratch/fmriprep_wf/single_subject_JG_wf/anat_preproc_wf/skullstrip_ants_wf/t1_skull_strip/highres001_BrainExtractionBrain.nii.gz -out uni_xform_masked_flirt.nii.gz -omat uni_xform_masked_flirt.mat -dof 6 -usesqform
180410-18:59:56,452 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.merge".
180410-18:59:58,418 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_preproc_report_wf.pos_tsnr" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_preproc_report_wf/pos_tsnr".
180410-18:59:58,446 workflow INFO:
[Node] Running "pos_tsnr" ("niworkflows.nipype.algorithms.confounds.TSNR")
180410-19:00:01,291 niworkflows INFO:
Report - setting fixed (/scratch/fmriprep_wf/single_subject_JG_wf/anat_preproc_wf/skullstrip_ants_wf/t1_skull_strip/highres001_BrainExtractionBrain.nii.gz) and moving (/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_reg_wf/fsl_bbr_wf/flt_bbr_init/uni_xform_masked_flirt.nii.gz) images
180410-19:00:01,291 niworkflows INFO:
Generating visual report
180410-19:00:11,817 niworkflows INFO:
Report - setting fixed (/scratch/fmriprep_wf/single_subject_JG_wf/anat_preproc_wf/skullstrip_ants_wf/t1_skull_strip/highres001_BrainExtractionBrain.nii.gz) and moving (/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_02_wf/bold_reg_wf/fsl_bbr_wf/flt_bbr_init/uni_xform_masked_flirt.nii.gz) images
180410-19:00:11,817 niworkflows INFO:
Generating visual report
180410-19:03:15,0 niworkflows INFO:
Successfully created report (/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_03_wf/bold_reg_wf/fsl_bbr_wf/flt_bbr_init/report.svg)
180410-19:03:15,235 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_03_wf.bold_reg_wf.fsl_bbr_wf.flt_bbr_init".
180410-19:03:22,982 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_confounds_wf.non_steady_state" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_confounds_wf/non_steady_state".
180410-19:03:23,10 workflow INFO:
[Node] Running "non_steady_state" ("niworkflows.nipype.algorithms.confounds.NonSteadyStateDetector")
180410-19:03:56,724 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_confounds_wf.non_steady_state".
180410-19:04:30,613 niworkflows INFO:
Successfully created report (/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_02_wf/bold_reg_wf/fsl_bbr_wf/flt_bbr_init/report.svg)
180410-19:04:31,160 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_02_wf.bold_reg_wf.fsl_bbr_wf.flt_bbr_init".
180410-19:04:39,702 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.bold_reference_wf.validate" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_reference_wf/validate".
180410-19:04:39,721 workflow INFO:
[Node] Running "validate" ("fmriprep.interfaces.images.ValidateImage")
180410-19:04:40,669 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_02_wf.bold_reg_wf.gen_ref" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_02_wf/bold_reg_wf/gen_ref".
180410-19:04:40,676 workflow INFO:
[Node] Outdated cache found for "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_02_wf.bold_reg_wf.gen_ref".
180410-19:04:40,953 workflow INFO:
[Node] Running "gen_ref" ("niworkflows.interfaces.utils.GenerateSamplingReference")
180410-19:04:46,179 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_02_wf.bold_reg_wf.gen_ref".
180410-19:04:49,576 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_01_wf.bold_reg_wf.fsl_bbr_wf.flt_bbr_init" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_01_wf/bold_reg_wf/fsl_bbr_wf/flt_bbr_init".
180410-19:04:49,581 workflow INFO:
[Node] Outdated cache found for "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_01_wf.bold_reg_wf.fsl_bbr_wf.flt_bbr_init".
180410-19:04:49,847 workflow INFO:
[Node] Running "flt_bbr_init" ("niworkflows.interfaces.registration.FLIRTRPT"), a CommandLine Interface with command:
flirt -in /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_01_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/apply_mask/uni_xform_masked.nii.gz -ref /scratch/fmriprep_wf/single_subject_JG_wf/anat_preproc_wf/skullstrip_ants_wf/t1_skull_strip/highres001_BrainExtractionBrain.nii.gz -out uni_xform_masked_flirt.nii.gz -omat uni_xform_masked_flirt.mat -dof 6 -usesqform
180410-19:05:14,127 niworkflows INFO:
Report - setting fixed (/scratch/fmriprep_wf/single_subject_JG_wf/anat_preproc_wf/skullstrip_ants_wf/t1_skull_strip/highres001_BrainExtractionBrain.nii.gz) and moving (/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_01_wf/bold_reg_wf/fsl_bbr_wf/flt_bbr_init/uni_xform_masked_flirt.nii.gz) images
180410-19:05:14,128 niworkflows INFO:
Generating visual report
180410-19:05:47,253 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.bold_reference_wf.validate".
180410-19:05:49,275 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.bold_reference_wf.gen_ref" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_reference_wf/gen_ref".
180410-19:05:49,296 workflow INFO:
[Node] Running "gen_ref" ("niworkflows.interfaces.registration.EstimateReferenceImage")
180410-19:05:55,238 interface INFO:
stderr 2018-04-10T19:05:55.237967:++ 3dvolreg: AFNI version=Debian-16.2.07~dfsg.1-5~nd16.04+1 (Jun 12 2017) [64-bit]
180410-19:05:55,239 interface INFO:
stderr 2018-04-10T19:05:55.237967:++ Authored by: RW Cox
180410-19:05:55,256 interface INFO:
stderr 2018-04-10T19:05:55.256760:* AFNI converts NIFTI_datatype=4 (INT16) in file /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_reference_wf/gen_ref/slice.nii.gz to FLOAT32
180410-19:05:55,257 interface INFO:
stderr 2018-04-10T19:05:55.256760: Warnings of this type will be muted for this session.
180410-19:05:55,258 interface INFO:
stderr 2018-04-10T19:05:55.256760: Set AFNI_NIFTI_TYPE_WARN to YES to see them all, NO to see none.
180410-19:05:55,273 interface INFO:
stderr 2018-04-10T19:05:55.273880:+ WARNING: If you are performing spatial transformations on an oblique dset,
180410-19:05:55,274 interface INFO:
stderr 2018-04-10T19:05:55.273880: such as /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_reference_wf/gen_ref/slice.nii.gz,
180410-19:05:55,274 interface INFO:
stderr 2018-04-10T19:05:55.273880: or viewing/combining it with volumes of differing obliquity,
180410-19:05:55,275 interface INFO:
stderr 2018-04-10T19:05:55.273880: you should consider running:
180410-19:05:55,275 interface INFO:
stderr 2018-04-10T19:05:55.273880: 3dWarp -deoblique
180410-19:05:55,277 interface INFO:
stderr 2018-04-10T19:05:55.273880: on this and other oblique datasets in the same session.
180410-19:05:55,278 interface INFO:
stderr 2018-04-10T19:05:55.273880: See 3dWarp -help for details.
180410-19:05:55,279 interface INFO:
stderr 2018-04-10T19:05:55.273880:++ Oblique dataset:/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_reference_wf/gen_ref/slice.nii.gz is 6.258273 degrees from plumb.
180410-19:05:55,281 interface INFO:
stderr 2018-04-10T19:05:55.281414:++ Coarse del was 10, replaced with 8
180410-19:06:16,641 niworkflows INFO:
Successfully created report (/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_01_wf/bold_reg_wf/fsl_bbr_wf/flt_bbr_init/report.svg)
180410-19:06:16,759 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_01_wf.bold_reg_wf.fsl_bbr_wf.flt_bbr_init".
180410-19:07:04,128 interface INFO:
stderr 2018-04-10T19:07:04.127992:++ Max displacement in automask = 0.08 (mm) at sub-brick 13
180410-19:07:04,133 interface INFO:
stderr 2018-04-10T19:07:04.127992:++ Max delta displ in automask = 0.10 (mm) at sub-brick 18
180410-19:07:17,984 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.bold_reference_wf.gen_ref".
180410-19:07:21,163 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.n4_mask" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/n4_mask".
180410-19:07:21,190 workflow INFO:
[Node] Running "n4_mask" ("fmriprep.interfaces.nilearn.MaskEPI")
180410-19:07:21,567 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_01_wf.bold_reg_wf.gen_ref" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_01_wf/bold_reg_wf/gen_ref".
180410-19:07:21,572 workflow INFO:
[Node] Outdated cache found for "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_01_wf.bold_reg_wf.gen_ref".
180410-19:07:21,613 workflow INFO:
[Node] Running "gen_ref" ("niworkflows.interfaces.utils.GenerateSamplingReference")
180410-19:07:23,64 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.n4_mask".
180410-19:07:23,313 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_01_wf.bold_reg_wf.gen_ref".
180410-19:07:24,42 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.n4_correct" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/n4_correct".
180410-19:07:24,75 workflow INFO:
[Node] Running "n4_correct" ("niworkflows.nipype.interfaces.ants.segmentation.N4BiasFieldCorrection"), a CommandLine Interface with command:
N4BiasFieldCorrection -d 3 --input-image /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_reference_wf/gen_ref/ref_image.nii.gz --mask-image /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/n4_mask/ref_image_mask.nii.gz --output ref_image_corrected.nii.gz
180410-19:07:32,140 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.n4_correct".
180410-19:07:34,127 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.skullstrip_first_pass" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/skullstrip_first_pass".
180410-19:07:34,157 workflow INFO:
[Node] Running "skullstrip_first_pass" ("niworkflows.nipype.interfaces.fsl.preprocess.BET"), a CommandLine Interface with command:
bet /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/n4_correct/ref_image_corrected.nii.gz /root/src/fmriprep/ref_image_corrected_brain.nii.gz -f 0.20 -m
180410-19:07:36,838 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.skullstrip_first_pass".
180410-19:07:38,148 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.skullstrip_first_dilate" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/skullstrip_first_dilate".
180410-19:07:38,181 workflow INFO:
[Node] Running "skullstrip_first_dilate" ("niworkflows.nipype.interfaces.fsl.maths.DilateImage"), a CommandLine Interface with command:
fslmaths -dt char /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/skullstrip_first_pass/ref_image_corrected_brain_mask.nii.gz -kernel sphere 6.0000 -dilF /root/src/fmriprep/ref_image_corrected_brain_mask_dil.nii.gz
180410-19:07:45,539 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.skullstrip_first_dilate".
180410-19:07:46,192 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.skullstrip_first_mask" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/skullstrip_first_mask".
180410-19:07:46,217 workflow INFO:
[Node] Running "skullstrip_first_mask" ("niworkflows.nipype.interfaces.fsl.maths.ApplyMask"), a CommandLine Interface with command:
fslmaths /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/skullstrip_first_pass/ref_image_corrected_brain.nii.gz -mas /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/skullstrip_first_dilate/ref_image_corrected_brain_mask_dil.nii.gz /root/src/fmriprep/ref_image_corrected_brain_masked.nii.gz
180410-19:07:46,565 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.skullstrip_first_mask".
180410-19:07:48,271 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.unifize" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/unifize".
180410-19:07:48,398 workflow INFO:
[Node] Running "unifize" ("niworkflows.nipype.interfaces.afni.utils.Unifize"), a CommandLine Interface with command:
3dUnifize -clfrac 0.2 -rbt 18.3 65.0 90.0 -prefix uni.nii.gz -T2 -input /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/unifize/ref_image_corrected_brain_masked.nii.gz
180410-19:07:56,424 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.unifize".
180410-19:07:58,763 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.fixhdr_unifize" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/fixhdr_unifize".
180410-19:07:58,783 workflow INFO:
[Node] Running "fixhdr_unifize" ("niworkflows.interfaces.utils.CopyXForm")
180410-19:07:59,9 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.fixhdr_unifize".
180410-19:08:00,94 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.skullstrip_second_pass" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/skullstrip_second_pass".
180410-19:08:00,134 workflow INFO:
[Node] Running "skullstrip_second_pass" ("niworkflows.nipype.interfaces.afni.preprocess.Automask"), a CommandLine Interface with command:
3dAutomask -apply_prefix uni_xform_masked.nii.gz -dilate 1 -prefix uni_xform_mask.nii.gz /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/skullstrip_second_pass/uni_xform.nii.gz
180410-19:08:00,735 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.skullstrip_second_pass".
180410-19:08:02,798 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.fixhdr_skullstrip2" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/fixhdr_skullstrip2".
180410-19:08:02,912 workflow INFO:
[Node] Running "fixhdr_skullstrip2" ("niworkflows.interfaces.utils.CopyXForm")
180410-19:08:03,37 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.fixhdr_skullstrip2".
180410-19:08:04,156 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.combine_masks" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/combine_masks".
180410-19:08:04,187 workflow INFO:
[Node] Running "combine_masks" ("niworkflows.nipype.interfaces.fsl.maths.BinaryMaths"), a CommandLine Interface with command:
fslmaths /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/skullstrip_first_pass/ref_image_corrected_brain_mask.nii.gz -mul /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/fixhdr_skullstrip2/uni_xform_mask_xform.nii.gz /root/src/fmriprep/ref_image_corrected_brain_mask_maths.nii.gz
180410-19:08:04,367 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.combine_masks".
180410-19:08:06,162 workflow INFO:
[Node] Setting-up "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.apply_mask" in "/scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/apply_mask".
180410-19:08:06,194 workflow INFO:
[Node] Running "apply_mask" ("niworkflows.nipype.interfaces.fsl.maths.ApplyMask"), a CommandLine Interface with command:
fslmaths /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/fixhdr_unifize/uni_xform.nii.gz -mas /scratch/fmriprep_wf/single_subject_JG_wf/func_preproc_task_context_run_04_wf/bold_bold_trans_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/combine_masks/ref_image_corrected_brain_mask_maths.nii.gz /root/src/fmriprep/uni_xform_masked.nii.gz
180410-19:08:06,687 workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_JG_wf.func_preproc_task_context_run_04_wf.bold_bold_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.apply_mask".

[oesteban]

Hmm. @oesteban Shouldn't --debug turn on the nipype status messages?

Not anymore. The correct way of increasing verbosity is using more -v

So, for debug mode you should use at least three vs. E.g.:

<your fmriprep invocation> -vvvv

for a very verbose DEBUG information.

[effigies]

Ah, thanks for the reminder. Hopefully I don't tell people the wrong thing again.

[oricon]

I seem to be having the same problem when I try running docker locally on my mac. Subjects from the same study have been run without this problem on a cluster through singularity.

I have Docker 18.03.0-ce-mac60 on Mac 10.13.4. I configured docker with 2 CPUs, 10 GB of RAM, and 2 GB of swap. My command for docker is
docker run -it --rm -v /Users/josephorr/adaptcog/bids:/data:delegated -v /Users/josephorr/adaptcog/preproc:/out:delegated -v /Users/josephorr/adaptcog/work:/scratch:delegated -v ~/adaptcog/license.txt:/opt/freesurfer/license.txt poldracklab/fmriprep:latest /data /out participant -w /scratch --fs-no-reconall --stop-on-first-crash --nthreads 2 --mem-mb 8000 --force-syn --ignore slicetiming --use-aroma --output-space {T1w,template} --template MNI152NLin2009cAsym --write-graph --participant-label M80398983 --debug -vvvv

I've tried running it several times now and it always get to friprep_wf.single_subject_M80398983_wf.func_preproc_task_ritl_acq_MB_run_06_wf.bold_confounds_wf.signals before stalling. With debug set to -vvvv I get 180411-23:33:49,883 workflow DEBUG: No resources available every 2 seconds so it filled by screen pretty quickly so I can't quite see the last terminal output before it ran out of resources. I'm going to try again with tee so I can see the terminal output.

[effigies]

I think -vv might be enough to see what's currently running without spamming the screen.

[danielkimmel]

So … I've tried re-running fmriprep several times, always passing along the scratch output from the previous run. The processing gets farther along in the dataset each time, until now. Now, it seems it can't get past the current stage.

I've been running with verbose output (-vvvv and -vvv). I'm still not getting definitive error messages. I will frequently get the warning "Cannot allocate job xxx (9.97GB, 2 threads)." Generally, it moves past these warnings and eventually allocates a job and completes a task. However, this does suggest that fmriprep is encountering a resource limitation. The strange thing is that eventually the CPU usage for the docker container will go to near-zero, but the terminal output will cease generating new messages and no new output files are created. One would think that if the problem were resource availability, that once the resources became available, the job(s) would continue. Perhaps I'm exceeding some timeout threshold? If so, the program is not stating that explicitly.

I've attached the terminal output from each successive call to fmriprep in a zip file ("…-10" is the latest). I've also attached a screenshot of the fmriprep output dir. It hasn't generated any additional files in the last 2 calls to fmriprep. It seems that the functional output is incomplete for imaging runs 03 and 04. All 5 of the runs have different imaging parameters (TR, voxel size, TE), which are specified in the respective JSON sidecars. Runs 03 and 04 are the largest .nii files (363 and 446 MB, respectively). The other runs are 44 - 277 MB each.

I'd be really grateful for any advice. I can't move forward on our study until processing these scans.

Thanks again,
Daniel

On Apr 11, 2018, at 8:14 PM, Chris Markiewicz [e-mail] wrote:

I think -vv might be enough to see what's currently running without spamming the screen.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/poldracklab/fmriprep/issues/1045#issuecomment-380634518, or mute the thread https://github.com/notifications/unsubscribe-auth/Ae1vFv-Olj5NobSyvo3PrvWgHerGT8P-ks5tnpx7gaJpZM4TOp-O.

[effigies]

Sorry, I don't think GitHub allows email attachments. Can you try posting your logs somewhere and linking to them?

[oricon]

I was able to save out my whole log and it stopped in the same spot (func_preproc_task_ritl_acq_MB_run_06_wf.bold_confounds_wf.signals) before No resources available. The log is available here: https://goo.gl/KPKAZa.

The last message before that was

[Node] Running "signals" ("niworkflows.nipype.interfaces.nilearn.SignalExtraction")
180412-14:42:25,444 workflow DEBUG:
Progress: 1066 jobs, 773/2/16 (done/running/ready), 2/291 (pending_tasks/waiting).
180412-14:42:25,444 workflow DEBUG:
Tasks currently running: 2. Pending: 2.
180412-14:42:25,452 workflow INFO:
[MultiProc] Running 2 tasks, and 16 jobs ready. Free memory (GB): 2.45/7.81, Free processors: 0/2.
Currently running:
* fmriprep_wf.single_subject_M80398983_wf.func_preproc_task_ritl_acq_MB_run_05_wf.bold_confounds_wf.signals
* fmriprep_wf.single_subject_M80398983_wf.func_preproc_task_ritl_acq_MB_run_06_wf.bold_confounds_wf.signals
180412-14:42:25,453 workflow DEBUG:
No resources available

[effigies]

Ah, if it's signals, I think I know the problem. Try using the --low-mem flag. If that fixes it, the issue is that you're running out of memory trying to load the entire BOLD series into memory. (We fix it by not compressing the resampled BOLD series, which allows us to load parts of the file at a time.)

This would explain why you're not seeing much CPU usage; probably the bottleneck was pushing active memory into swap space.

@danielkimmel This may be your issue as well, which would explain why only the larger BOLD series are affected.

[danielkimmel]

Attached are the terminal outputs from successive calls to fmriprep.

On the last call, I increased my docker container to 15 of 16 GB RAM and the process completed!

So it does seem to be a memory issue. I'll try again with the --low-mem flag. What's the downside to using the flag?

fmriprep-docker-terminal-output.zip

[effigies]

It will increase hard disk usage in the scratch directory. It's a pretty low price to pay, so perhaps we should consider making it the default.

[danielkimmel]

Definitely your call about defaults. For what it's worth, increased disk usage may not be trivial either. For instance, I'm running on a new MBP 13 with 16GB RAM and 500GB HDD. (I know this setup is subpar for robust imaging analysis, but it was helpful for testing.) When I started, I had 80GB free on the HDD, including the original imaging files. But after initiating the Docker container and running fmriprep, HDD free space dropped to close to zero and I had to free up some space.

I'm not sure if it's possible, but it would be really helpful to know upfront if resources limitations are likely to be an issue. As it was, I didn't encounter either the RAM problem or the HDD problem until after hours of running. And with the RAM problem, I never got an explicit error (hence the troubleshooting). Is it possible to "preview" the job to estimate how much RAM and/or HDD space would be necessary for the job? If not, it would be still be great if an explicit error were raised when the resource limitation was encountered.

Anyhow, thanks for making and maintaining all these tools. It's easy to ask for things, hard to give them!

Best,
d

On Apr 12, 2018, at 7:44 PM, Chris Markiewicz [e-mail] wrote:

It can increase hard disk usage in the scratch directory. It's a pretty low price to pay, so perhaps we should consider making it the default.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/poldracklab/fmriprep/issues/1045#issuecomment-380977094, or mute the thread https://github.com/notifications/unsubscribe-auth/Ae1vFoaRV_AWmKT7u0MdX_2sVbyyRub_ks5tn-bygaJpZM4TOp-O.

[oesteban]

@effigies there are extra files written if low_mem=True? I guess that the largest difference is I/O direct to the hard-disk, not in space or number of inodes right?

[effigies]

The full BOLD series (in BOLD, T1w and MNI space) are save uncompressed so we can mmap them. No extra files are written.

[oesteban]

Oh, then it could be a big deal for large datasets

On Fri, Apr 13, 2018, 13:42 Chris Markiewicz [e-mail]
wrote:

The full BOLD series (in BOLD, T1w and MNI space) are save uncompressed so
we can mmap them. No extra files are written.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/poldracklab/fmriprep/issues/1045#issuecomment-381255505,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAkhxrtEuWMNow1gWbB95y9dT8Trlsnlks5toQ3AgaJpZM4TOp-O
.

[effigies]

Okay, then we'll leave it compressed by default.

We should add this to a FAQ, as this has come up a few times.

[effigies]

Added to the FAQ.