Fmriprep: Reruns (all?) workflows when only performance flags change

[psadil]

Just trying to confirm that this is the expected behavior, and if so maybe request a feature for different behavior. When I run fmriprep in Windows 10 by calling docker directly

docker run --rm -ti -v C:\Users\psadil\data\vtf:/data:ro -v C:\Users\psadil\data\vtf\derivatives:/out -v C:\Users\psadil\data\vtf\derivatives\work:/scratch poldracklab/fmriprep:1.0.0-rc2 /data /out participant --output-space T1w template fsnative fsaverage --write-graph -w /scratch --n_cpus=4 --mem_mb=10240

and then rerun without flags to handle performance

docker run --rm -ti -v C:\Users\psadil\data\vtf:/data:ro -v C:\Users\psadil\data\vtf\derivatives:/out -v C:\Users\psadil\data\vtf\derivatives\work:/scratch poldracklab/fmriprep:1.0.0-rc2 /data /out participant --output-space T1w template fsnative fsaverage --write-graph -w /scratch

frmiprep begins rerunning ANTs. Is that expected? Does calling fmriprep with different flags always invalidate its cache? I'm only able to run on a local machine and so would like to be able to avoid as much duplicated processing as possible.

Thanks,

[effigies]

That should not invalidate the cache, no. I wonder if there's a Windows/Docker issue here. Are you able to replicate this on a Linux machine?

[oesteban]

@effigies: we should also check the hash metadata of all inputs of that node, DYT?

[psadil]

@effigies - I could not replicate this on a Linux machine. I ran that first command (with performance flags) in Ubuntu 17.04, waited until seeing

[Job finished] jobname: t1_skull_strip: 47

crtl+c'd, then ran the second command (without flags). fmriprep didn't pause during execution of the t1_skull_strip node, so I'm assuming it was able to grab what had already been run.

[effigies]

@oesteban I'm not entirely sure what you mean, there.

@psadil Okay, I'm going to guess this is an issue with mounting Windows filesystems (@chrisfilo, have you seen this before?). You may be able to resolve it by creating/updating a nipype.cfg file the following lines:

[execution]
hash_method = timestamp

This can be hooked into Docker using fmriprep-docker --config <config_file>, or adding -v /path/to/nipype.cfg:/root/.nipype/nipype.cfg:ro to a Docker command line.

[chrisgorgo]

I don't recall seeing this and I depend on cache quite a bit.

On Aug 16, 2017 8:06 AM, "Chris Markiewicz" [e-mail]
wrote:

@oesteban https://github.com/oesteban I'm not entirely sure what you
mean, there.

@psadil https://github.com/psadil Okay, I'm going to guess this is an
issue with mounting Windows filesystems (@chrisfilo
https://github.com/chrisfilo, have you seen this before?). You may be
able to resolve it by creating/updating a nipype.cfg
https://nipype.readthedocs.io/en/latest/users/config_file.html file the
following lines:

[execution]
hash_method = timestamp

This can be hooked into Docker using fmriprep-docker --config
, or adding -v /path/to/nipype.cfg:/root/.
nipype/nipype.cfg:ro to a Docker command line.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/poldracklab/fmriprep/issues/668#issuecomment-322622158,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAOkp6CnkGVcwCGWQobkykzovRv0-esGks5sYjJqgaJpZM4O4CdE
.

[psadil]

@chrisfilo - Here's to hoping that I haven't just set things up in a weird way...Sorry to be a bother if that's the case. For what it's worth I get the same behavior when I change the number of cores allocated to docker through their internal settings.

@effigies - Would the fmriprep docker image have come with a nipype.cfg file? I haven't created one for it. I'm just looking to confirm that I can safely just mount a new one and not overwrite anything?

[effigies]

Oh.... the issue may be that the "number of threads" input to the ANTs programs is being changed, causing nipype to think the node needs re-running.

I'll look to see if we can exclude these inputs from the cache invalidation check. It may require a nipype patch.

Edit: Also, no, you wouldn't be overwriting anything.

[chrisgorgo]

Hmm - nohash=True should've prevented this from happening.

On Wed, Aug 16, 2017 at 9:30 PM, Chris Markiewicz [e-mail]
wrote:

Oh.... the issue may be that the "number of threads" input to the ANTs
programs is being changed, causing nipype to think the node needs
re-running.

I'll look to see if we can exclude these inputs from the cache
invalidation check. It may require a nipype patch.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/poldracklab/fmriprep/issues/668#issuecomment-322771181,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAOkp2V9rZfy1uoX0ieQroQAuR02DJzWks5sYu7fgaJpZM4O4CdE
.

[effigies]

@chrisfilo You're right. ANTsCommandInputSpec sets nohash=True.

@psadil So I guess I'm back to my guess that there's a timestamp issue related to CIFS mounts, but I'm not sure of an easy way to test this hypothesis. If the issue replicates with hashmethod = content, that will disconfirm it. (Note that you will have to rerun any nodes that read/write files once; since what's being hashed is changing. It's the second time around that will test the idea.)

[psadil]

@effigies - okay. It's going to be a day or two before I can test that, but I'll report back when I've tried.

[effigies]

Alright. Thanks for your help and patience. Sorry that this isn't working as expected.

[oesteban]

@effigies I meant the nohash metadata. But for some reason I said hash.

[effigies]

@psadil Has this issue been resolved?

[psadil]

@effigies just ran the test today (sorry for the delay, I'm away from work for a bit). I ran fmriprep via docker, as above, trying both hash_method = timestamp and hash_method = content. In both cases, I called docker and set --n_cpus=4, waited until t1_skull_strip jobid: 47 was reported as [finished], and then reran without the --n_cpus flag. I didn't notice fmriprep working on any of the other nodes that had been marked as finished, but in both cases it had to redo the t1_skull_strip jobid: 47 node.

Just to check, I reran fmriprep a second time without the --n_cpus flag (i.e., a third time with hash_method = content, after stopping the process once 47 was finished on the first run without --n_cpus). fmriprep didn't pause on 47, so at least there it was able to use the cache when the --n_cpus flag didn't change.

[oesteban]

Please reopen if this has not been resolved.