Fmriprep: Number of files written to working directory

[jokedurnez]

Running fmriprep on MyConnectome generated at least 1,794,910 files in the working directory. My job stopped because I exceeded the maximum number of files on my system (sherlock:oak - 8 mil). Something to keep in mind :-)

[effigies]

Wow. Good to know. I'm guessing this is because of the large number of EPI volumes?

I think there was discussion about re-jigging HMC to avoid the explosion of files, but if anything ever came of that, it sure wasn't code.

[effigies]

Assuming that that is the source of the problem, I think your best bet for now is to specify --task-id for separate tasks, and delete the corresponding func_preproc_task_<task-label>_run_<run-label>_wf directories from the working directory, once it's completed.

[oesteban]

@jokedurnez I think you should also use a local scratch mount point for the work directory. It'll be faster and I don't think there'll be any limitations on size and numbers of files.

[chrisgorgo]

I doubt local scratch would be big enough for this dataset.

@effigies Indeed switching to pypreprocess (https://github.com/poldracklab/fmriprep/issues/444) would avoid the need to convert MCFLIRT affine matrices to ITK and thus decrease the number of intermediate files.

[jokedurnez]

@oesteban I can't run on a local scratch, because I'm hitting the maxtime, so I need to be able to restart freesurfer...
(I can - of course - still move to TACC ;-) )

[effigies]

My recommendation in that case would be to first run with --anat-only --output-space T1w, so that almost all of the time is devoted to FreeSurfer, until it's finished. Once that's done (and FreeSurfer is in the output directory), you could run the rest on local scratch.

Also, is somebody already building a 0.6.0 singularity image? That will resolve your issues with the ses-001 T1w images. I can get it building if nobody else is. Will take about 45 minutes to build and transfer to Sherlock.

[oesteban]

@jokedurnez Gotcha. The only alternative I can think of is running FreeSurfer first, move the results to the appropriate location in OAK and then run fmriprep.

@chrisfilo ITK accepts writing all the transforms in only one file. We could remove that MapNode (number of ITK transforms times num. of files per node generated by nipype and the c3d interface). We could replace all of that with just one node. WDYT?, that'd be probably faster than moving to pypreprocess

PS. @effigies just posted while I was writing, probably his opinion is much better informed than mine.

[effigies]

Having issues with my build machine, if somebody else wants to push a new Singularity image...

[chrisgorgo]

On Mon, Jul 31, 2017 at 8:47 AM, Oscar Esteban [e-mail]
wrote:

@jokedurnez https://github.com/jokedurnez Gotcha. The only alternative
I can think of is running FreeSurfer first, move the results to the
appropriate location in OAK and then run fmriprep.

@chrisfilo https://github.com/chrisfilo ITK accepts writing all the
transforms in only one file. We could remove that MapNode (number of ITK
transforms times num. of files per node generated by nipype and the c3d
interface). We could replace all of that with just one node. WDYT?, that'd
be probably faster than moving to pypreprocess

But does c3d support such conversion? BTW this will reduce the number of
files, but I expect that myconnectome will still be problematic.

PS. @effigies https://github.com/effigies just posted while I was
writing, so probably his opinion is much better informed than mine.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/poldracklab/fmriprep/issues/623#issuecomment-319110745,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAOkp7hb6pru73Slgxw33Q5GMTLZcwiqks5sTfcmgaJpZM4OnaNb
.