Fmriprep: index out of range errors due to non-anatomical .nii derivatives being read as anatomical images

[nblauch]

Rerunning FMRIPREP on a subject who has gone through various post-processing pipelines poses an issue due apparently to FMRIPREP looking in any bids_dir/**/sub-{subid} folders. this requires keeping separate "pre-processed only" and "post-processing" BIDS directories to be able to rerun FMRIPREP (e.g. for func files in a new space) on subjects for whom post-processing has been done.

i think this could be fixed if only files in data/sub-{subid} were considered as candidates for anatomical images. although if bids_dir/derivatives/freesurfer/sub-{subid}/mri had to be off limits, this would seem fine too. i do not see why e.g. bids_dir/derivatives/matlab should ever be looked at.

see the attached crash for an example.

thanks for any help.

crash-20180723-161023-root-t1_name-dd98e0c1-c2cb-4906-8a8c-c783da097069.txt

[chrisgorgo]

Agreed. This behavior is weird - those should be excluded:
https://github.com/poldracklab/fmriprep/blob/master/fmriprep/utils/bids.py#L154

It could related to changes in pybids (CCing @tyarkoni) . Are you using the FMRIPREP version from Docker Hub? Are you using docker or singularity?

[nblauch]

v1.0.12 through docker

[tyarkoni]

Yeah this is probably due to changes in pybids--it's currently scanning derivatives/ by default, and probably shouldn't be. Will try to push a fix later today.

[tyarkoni]

Oh, sorry--I didn't see that you're explicitly passing exclude. This could be a bug, separate from what I mentioned in my last comment (which is not a bug, just an undesirable feature); will look into it.

[chrisgorgo]

for the reference we are using 0.6.3 release of pybids

[tyarkoni]

There's going to be a new release in the next day or two (for unrelated reasons), so I'll try to include a fix for this in that.

[chrisgorgo]

Awesome! Thanks!
Best,
Chris

On Tue, Jul 24, 2018 at 11:15 AM Tal Yarkoni [e-mail]
wrote:

There's going to be a new release in the next day or two (for unrelated
reasons), so I'll try to include a fix for this in that.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/poldracklab/fmriprep/issues/1220#issuecomment-407502318,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAOkp6pobjY0wO7Szsz91_8HH6mSQrGrks5uJ2QsgaJpZM4VdKFz
.

[tyarkoni]

Okay I'm looking at this, and I'm not sure it's a pybids bug... we have tests for inclusion/exclusion arguments, and those seem to be working fine. If you're passing exclude=['derivatives'], that should filter out files that contain 'derivatives' anywhere in the name. Before I get too deep into this, are there other plausible places the issue could be arising?

[chrisgorgo]

I just read through the code again and I don't see where else this could come from. @nblauch do you have access to the working directory? Could you share the report.rst from the bidssrc node?

[tyarkoni]

Okay, I'll write some extra tests to simulate your use case--just for you!

[nblauch]

it was autodeleted but i will try to reproduce it. hang tight.

[nblauch]

I had to convert the report to .txt to attach but here it is.
report.txt

this warning upon starting fmriprep might also be relevant:

/usr/local/miniconda/lib/python3.6/site-packages/grabbit/core.py:410: UserWarning: No valid root directory found for domain 'derivatives'. Falling back on the Layout's root directory. If this isn't the intended behavior, make sure the config file for this domain includes a 'root' key.

[tyarkoni]

@chrisfilo what version of grabbit are you using? The warning @nblauch pasted is from several releases ago, I believe. This may explain the problem you're having; there were pretty big changes to grabbit internally.

I still can't find any reason why exclusions would be failing. I suggest waiting till the next pybids release (probably in a couple of hours) and then trying with the latest. It's possible the issue will magically resolve itself.

[chrisgorgo]

fmriprep 1.1.2 uses pybids 0.6.3 - we did not specify grabbit version. @nblauch which version of fmriprep are you using?

[tyarkoni]

Oh, if @nblauch is using an older version of fmriprep, this error would make sense. The issue should be fixed in recent builds.

[nblauch]

v1.0.12.

is there any way to fix this without upgrading? i'd like to stick to the same FMRIPREP for this dataset, and would greatly prefer not to rerun everything (including extensive post-processing). or @chrisfilo , would you not worry about switching versions from 1.0.12 to 1.1.2 within dataset?

[tyarkoni]

Sure, you could probably just pip install --upgrade pybids.

[nblauch]

hm, within the docker container?

i can also just keep my folders separate for now.

sorry for the late discovery, but it looks like this was fixed in #1141 . Earlier, exclude='derivatives' was not included in the call to BIDSlayout in bids.py.

[chrisgorgo]

Glad it's sorted!

[nblauch]

thanks for the help @chrisfilo & @tyarkoni