Fmriprep: During the process: indexed_gzip.indexed_gzip.ZranError: zran_read returned error (-3)

[ali3654]

Hello Experts,
i have a bids format folder (I checked this in BIDS validatior) and tried to use smriprep (later also fmriprep) in the docker on a single subject.

Exact command line:

 docker run --rm -e DOCKER_VERSION_8395080871=19.03.13 -it -v /Users/../license.txt:/opt/freesurfer/license.txt:ro -v /Users/../smri-test:/data:ro -v /Users/.../smri-test/fsl-license:/out -v /Users/ali:/scratch nipreps/smriprep:0.7.0 /data /out participant --participant-label OAS30001 --fs-no-reconall --nthreads 4 --stop-on-first-crash --mem_gb 25 -w /scratch

It seems to work in the beginning but while it is running a strange error message appears:

201110-18:08:20,645 nipype.interface WARNING:
     Changing /out/smriprep/sub-OAS30001/ses-d0129/anat/sub-OAS30001_ses-d0129_run-2_desc-brain_mask.nii.gz dtype from int16 to uint8
201110-18:08:23,354 nipype.workflow WARNING:
     Storing result file without outputs
201110-18:08:23,358 nipype.workflow WARNING:
     [Node] Error on "smriprep_wf.single_subject_OAS30001_wf.anat_preproc_wf.anat_derivatives_wf.ds_t1w_preproc" (/scratch/smriprep_wf/single_subject_OAS30001_wf/anat_preproc_wf/anat_derivatives_wf/ds_t1w_preproc)
201110-18:08:23,363 nipype.workflow ERROR:
     Node ds_t1w_preproc failed to run on host ec88f6a5959a.
201110-18:08:23,369 nipype.workflow ERROR:
     Saving crash info to /out/smriprep/sub-OAS30001/log/20201110-173803_7849de25-30b7-4b1b-8514-b9c61462a666/crash-20201110-180823-root-ds_t1w_preproc-bcd936f9-2e79-4771-ae1d-6d28e5f57437.txt
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 344, in _send_procs_to_workers
    self.procs[jobid].run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/bids.py", line 588, in _run_interface
    nii = nb.load(out_file, mmap=False)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nibabel/loadsave.py", line 50, in load
    is_valid, sniff = image_klass.path_maybe_image(filename, sniff)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nibabel/filebasedimages.py", line 510, in path_maybe_image
    sniff)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nibabel/filebasedimages.py", line 458, in _sniff_meta_for
    binaryblock = fobj.read(sniff_nbytes)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nibabel/openers.py", line 160, in read
    return self.fobj.read(*args, **kwargs)
  File "indexed_gzip/indexed_gzip.pyx", line 578, in indexed_gzip.indexed_gzip._IndexedGzipFile.readinto
indexed_gzip.indexed_gzip.ZranError: zran_read returned error (-3)

Preprocessing did not finish successfully. Errors occurred while processing data from participants: OAS30001 (1). Check the HTML reports for details.
sMRIPrep: Please report errors to https://github.com/nipreps/smriprep/issues

I really don't know what to do, it doesn't work with fmriprep either.
Docker also has a memory of 16 GB and 12 CPU.

But I noticed that in the new folder smriprep (fmriprep also) created is a nii.gz file that can't be unzipped.
The created not working nii.gz of smriprep : sub-OAS30001_ses-d0129_run-2_desc-preproc_T1w.nii.gz

it seems that there is a nii.gz created by smriprep or fmriprep that is not zipped but ends with .gz.

I can't unzip this file even with "gunzip".

I have smriprep version 0.7.0 and fmriprep version 20.2.1

I am very grateful for your help.

[mgxd]

this has popped up before: https://github.com/nipreps/fmriprep/issues/1717

can you verify your input T1w file is compressed?

[effigies]

This is something that needs to be better handled in indexed_gzip or nibabel.

Even if we do get improved errors from indexed_gzip, it might be worth patching nibabel to detect when an uncompressed file is being passed as compressed.

[ali3654]

Thanks for the quick feedback.

@mgxd yes the input T1w ist compressed and from Oasis

@effigies Do you have any idea what I can do exactly because this happens during the process.
Because in the end I don't have any pre-processed Brain MRI scans.

[effigies]

I've requested access for the OASIS datasets so I can try to reproduce the issue.

To be clear, how are you confirming that the T1w image is compressed?

[ali3654]

i can see that the input T1w file ends with nii.gz at the end and i can unzip it with "gunzip" and i can open it with nibabel.

What I noticed then is that during the process smriprep or fmriprep created a new folder with 2 nii.gz files:
sub-OAS30001_ses-d0129_run-2_desc-brain_mask.nii.gz
sub-OAS30001_ses-d0129_run-2_descpreproc_T1w.nii.gz

I can also unzip the file sub-OAS30001_ses-d0129_run-2_desc-brain_mask.nii.gz and open it with nibabel.

But the file sub-OAS30001_ses-d0129_run-2_descpreproc_T1w.nii.gz I can neither unzip it nor open it with nibabel, this shows an error message, it looks like the file has not been compressed by fmriprep or so because that would be created during the process. I cannot do anything with this file.

[effigies]

Thanks. If I can get hold of the data, I'll try to reproduce.

[ali3654]

i have downloaded the subject from oasis as bids format.

It looks like this :

+---Smri-test
| +---OAS30001_MR_d0129.tar
| |
| +---sub-OAS3001
| | +---ses-d0129
| | | +---anat
| | | | ---sub-OAS30001_ses-d0129_run-02_T1w.json
| | | | ---sub-OAS30001_ses-d0129_run-02_T1w.nii.gz
| | | ---dataset_description.json

I noticed that I didn't pack it properly in the Bids Validator. I did that now and it shows 2 errors but I think they have nothing to do with the error message above.

First error: Error 1: [Code 1] NOT_INCLUDED
That is because of smri-test/OAS30001_MR_d0129.tar and smri-test/sub-OAS30001/ses-d0129/dataset_description.json

Error 2: [Code 57] DATASET_DESCRIPTION_JSON_MISSING
-->The compulsory file /dataset_description.json is missing. See Section 03 (Modality agnostic files) of the BIDS specification.

These are the 2 errors but as said above I think it has nothing to do with the error message indexed_gzip.indexed_gzip.ZranError: zran_read returned error (-3).

I will gladly wait until you can test this with the data set from Oasis.

thank you for the help and the engangment.

[effigies]

Yeah, it looks like the dataset is invalid, so it's possible there are some pybids issues resulting. If you move datast_description.json up to the dataset root, then you should be able to validate more thoroughly.

[ali3654]

ok i will cover this up thank you and just for info, this is the Oasis-3 data set.

[ali3654]

have done the datast_description.json file in dataset root and tested it again with the preprocessing.

unfortunately the same error message as mentioned above occurred.
-> ..... indexed_gzip.indexed_gzip.ZranError: zran_read returned error (-3)

Note:
but if I now check this in the Bids validator then I have 2 errors but others:

Error 1: Error 1: [Code 55] JSON_SCHEMA_VALIDATION_ERROR
-> Invalid JSON file. The file is not formatted according to the schema.

Error 2: [Code 1] NOT_INCLUDED
->location : smri-test/OAS30001_MR_e0129.tar

Thanks again and I will wait until you have access to the data.

[ali3654]

Hey @effigies
Did you find anything about this problem yet?

Thanks in advance.

[effigies]

Ah, thanks for the prod. The OASIS registration email hit my junk folder, so I appreciate the reminder.

Will run tests tonight.

[effigies]

I'm not able to replicate this issue. Can you try deleting your working directory? It's possible that there was a writing failure that led to a malformed file. Another possibility is that you ran out of space on the drive containing your working directory, so the file was truncated before it was completed.

[ali3654]

Hmm i work in a OneDrive Directory an in a virtual env in Python.
Do have the bids Format directory like i had?

[ali3654]

Hmm very Strange because fmriprep create the zip file

[ali3654]

In my Root Directory i have the subject folder, and a folder that Contains the bash script and i have an another folder with the Fsl license and in this folder fmriprep create the new folder with the preprocessing files

[effigies]

I have the BIDS dataset as follows:

$ tree /data/bids/oasis 
/data/bids/oasis
└── oasis3
    ├── dataset_description.json
    ├── sub-OAS30001
    │   └── ses-d0129
    │       ├── anat
    │       │   ├── sub-OAS30001_ses-d0129_acq-TSE_T2w.json
    │       │   ├── sub-OAS30001_ses-d0129_acq-TSE_T2w.nii.gz
    │       │   ├── sub-OAS30001_ses-d0129_run-01_T1w.json
    │       │   ├── sub-OAS30001_ses-d0129_run-01_T1w.nii.gz
    │       │   ├── sub-OAS30001_ses-d0129_run-02_T1w.json
    │       │   ├── sub-OAS30001_ses-d0129_run-02_T1w.nii.gz
    │       │   ├── sub-OAS30001_ses-d0129_T2w.json
    │       │   └── sub-OAS30001_ses-d0129_T2w.nii.gz
    │       └── func
    │           ├── sub-OAS30001_ses-d0129_task-rest_run-01_bold.json
    │           ├── sub-OAS30001_ses-d0129_task-rest_run-01_bold.nii.gz
    │           ├── sub-OAS30001_ses-d0129_task-rest_run-02_bold.json
    │           ├── sub-OAS30001_ses-d0129_task-rest_run-02_bold.nii.gz
    │           ├── sub-OAS30001_ses-d0129_task-rest_run-03_bold.json
    │           └── sub-OAS30001_ses-d0129_task-rest_run-03_bold.nii.gz
    └── task-rest_bold.json

It took some rearranging from what I got out of XNAT, including:

mv OAS30001_MR_d0129/scans/anat?/resources/*/* anat/
mv OAS30001_MR_d0129/scans/func?/resources/*/* func/

I had to modify dataset_description.json and create task-rest_bold.json in order to validate.

Then I ran:

$ smriprep-docker /data/bids/oasis/oasis3 /data/out/oasis3-smriprep participant --fs-no-recon-all
RUNNING: docker run --rm -e DOCKER_VERSION_8395080871=19.03.13 -it -v /home/cjmarkie/Documents/freesurfer_license.txt:/opt/freesurfer/license.txt:ro -v /data/bids/oasis/oasis3:/data:ro -v /data/out/oasis3-smriprep:/out nipreps/smriprep:0.7.0 /data /out participant --fs-no-recon-all

[ali3654]

Thanks.
Hmm i Downloaded just the t1w scans because i wanted to make just an anatomical preprocessing.
Should i modify also the Dataset_descripton.json file and create the task-rest_bold.json?
And how do u modify the description.json file and create the task-rest_bold.json?

[ali3654]

because I downloaded it directly in bids format with these instructions https://github.com/NrgXnat/oasis-scripts

[effigies]

Okay, I'll give that script a try. I'll send the details of my dataset when I get back to that computer tomorrow.

[ali3654]

I have really tried everything now.
I took a different folder and a different path and I also downloaded the file from the XNAt and tested it in the Bids Validator and had no error. I also uninstalled and reinstalled smriprep docker.

My Bids folder:
└── oasis
           ├── dataset_description.json
           └── sub-OAS30001
               └── ses-d0129
                   └── anat
                       ├── sub-OAS30001_ses-d0129_run-02_T1w.json
                       └── sub-OAS30001_ses-d0129_run-02_T1w.nii.gz

Exact command line:

RUNNING: docker run --rm -e DOCKER_VERSION_8395080871=19.03.13 -it -v /Users/alihassoun/Desktop/bids/fsl-license/license.txt:/opt/freesurfer/license.txt:ro -v /Users/alihassoun/Desktop/bids/oasis:/data:ro -v /Users/alihassoun/Desktop/bids/output:/out nipreps/smriprep:0.7.0 /data /out participant --fs-no-reconall

Smriprep-docker is stored in this path:
$HOME/Library/Python/3.8/bin

Since you tried this on your computer it is probably not because of the data because I downloaded it from XNAT and as I said before the same error message.
is it maybe because of my PC or my python ?
I have a Mac.

[effigies]

$ cat oasis3/dataset_description.json 
{
    "DatasetDOI": "https://central.xnat.org/data/archive/projects/OASIS3/subjects/CENTRAL_S05094/experiments/CENTRAL_E09253",
    "HowToAcknowledge": "Refer to documentation at www.oasis-brains.org",
    "License": "Refer to Data Use Agreement at www.oasis-brains.org",
    "Name": "OASIS3",
    "BIDSVersion": "1.0.1"
}
$ cat oasis3/task-rest_bold.json 
{
  "TaskName": "Resting State"
}

And FWIW:

find /data/bids/oasis -type f -exec md5sum "{}" ";"
6c1f2b8a20978a759375483ea4c7cb8d  /data/bids/oasis/oasis3/sub-OAS30001/ses-d0129/func/sub-OAS30001_ses-d0129_task-rest_run-03_bold.json
bd6959fdbd57aac8c7ef2f4b65593e65  /data/bids/oasis/oasis3/sub-OAS30001/ses-d0129/func/sub-OAS30001_ses-d0129_task-rest_run-01_bold.nii.gz
9e2f6ac9c9f228e4ffd03e347b69c351  /data/bids/oasis/oasis3/sub-OAS30001/ses-d0129/func/sub-OAS30001_ses-d0129_task-rest_run-02_bold.nii.gz
ceb364b9606550f9ff64007b8bff1576  /data/bids/oasis/oasis3/sub-OAS30001/ses-d0129/func/sub-OAS30001_ses-d0129_task-rest_run-01_bold.json
5494d821583460a33460fc4421d500fa  /data/bids/oasis/oasis3/sub-OAS30001/ses-d0129/func/sub-OAS30001_ses-d0129_task-rest_run-02_bold.json
c830b8847387f8110f1502bc48f98de9  /data/bids/oasis/oasis3/sub-OAS30001/ses-d0129/func/sub-OAS30001_ses-d0129_task-rest_run-03_bold.nii.gz
151c2d56766be57ab4bdc4252f26008f  /data/bids/oasis/oasis3/sub-OAS30001/ses-d0129/anat/sub-OAS30001_ses-d0129_run-01_T1w.nii.gz
764e9fb6dfd207b45b85eb18b4c7ce43  /data/bids/oasis/oasis3/sub-OAS30001/ses-d0129/anat/sub-OAS30001_ses-d0129_run-01_T1w.json
c4eca5d93473638602759bde56d7f96b  /data/bids/oasis/oasis3/sub-OAS30001/ses-d0129/anat/sub-OAS30001_ses-d0129_acq-TSE_T2w.nii.gz
4fe648c03cd0bffb532447895e6e300b  /data/bids/oasis/oasis3/sub-OAS30001/ses-d0129/anat/sub-OAS30001_ses-d0129_run-02_T1w.json
34f31bfa11f57eb0264e38fcbc2a998d  /data/bids/oasis/oasis3/sub-OAS30001/ses-d0129/anat/sub-OAS30001_ses-d0129_T2w.json
558799535d887ccf93b539f4427f3343  /data/bids/oasis/oasis3/sub-OAS30001/ses-d0129/anat/sub-OAS30001_ses-d0129_run-02_T1w.nii.gz
7960f67280d98c4c832e4ee661f0bba6  /data/bids/oasis/oasis3/sub-OAS30001/ses-d0129/anat/sub-OAS30001_ses-d0129_acq-TSE_T2w.json
76fb23ddc07fe2cc49409cb3144a5d02  /data/bids/oasis/oasis3/sub-OAS30001/ses-d0129/anat/sub-OAS30001_ses-d0129_T2w.nii.gz
2cc460ed46b4719c8ea8aa32352e6edd  /data/bids/oasis/oasis3/task-rest_bold.json
540dd97fc133746e0ab4b0a228846817  /data/bids/oasis/oasis3/dataset_description.json

I've created a reduced version of the dataset with only sub-OAS30001_ses-d0129_run-02_T1w.nii.gz as you show, and am running that now:

$ smriprep-docker /data/bids/oasis/ali3654 /data/out/ali3654-smriprep participant --fs-no-reconall --sloppy
RUNNING: docker run --rm -e DOCKER_VERSION_8395080871=19.03.13 -it -v /home/cjmarkie/Documents/freesurfer_license.txt:/opt/freesurfer/license.txt:ro -v /data/bids/oasis/ali3654:/data:ro -v /data/out/ali3654-smriprep:/out nipreps/smriprep:0.7.0 /data /out participant --fs-no-reconall --sloppy

[effigies]

is it maybe because of my PC or my python ?
I have a Mac.

Should have no effect. The Python used to run smriprep-docker has no impact on the Python inside the Docker container. And I've run on macs plenty. If my Linux test fails, I can switch to a mac for testing.

[ali3654]

hmm yes it is a good idea to try it like my Bids folder.
because i only have a t1w scan because i just wanted to do a very simple anatomical preprocessing.
I will test this again with the task-rest_bold.json file.

thank you very much for this great support.

[effigies]

Ran without issue. Switching to mac.

[ali3654]

i have tested it again with thetask-rest_bold.json file and unfortunately this error message still appears.

My Bids folder:

├── oasis
│   ├── dataset_description.json
│   ├── sub-OAS30001
│   │   └── ses-d0129
│   │       └── anat
│   │           ├── sub-OAS30001_ses-d0129_run-02_T1w.json
│   │           └── sub-OAS30001_ses-d0129_run-02_T1w.nii.gz
│   └── task-rest_bold.json

Smriprep creates 2 nii.gz files.
once these:

 sub-OAS30001_ses-d0129_run-2_desc-brain_mask.nii.gz

I can unzip this file and it is also compressed.

The other last created file is:

sub-OAS30001_ses-d0129_run-2_desc-preproc_T1w.nii.gz

I can't unzip this file or open it with nibabel, I get an error message. My Mac can't open it either.

When I look at the information of the last created sub-OAS30001_ses-d0129_run-2_desc-preproc_T1w.nii.gz file I see that I only have access rights to read and write.
at "everyone" it says "no rights".

i don't know now if this has anything to do with it but i can't explain it any other way.

[effigies]

Okay, if it might be a configuration issue, are you giving your Docker plenty of space ("Disk image size" in Preferences -> Resources)? And in case it's permissions, you could add -u $UID to your smriprep-docker command to ensure that any outputs are created as your own user. (This might also require passing -w. I'm not sure if we handle non-root users with an in-image working directory well.)

[ali3654]

at docker resources I have:
CPUs: 12

Memory: 16.00 GB

Disk image size: 59.6 GB (16.8 GB used)

Swap: 1 GB

I hope this is sufficient

I will now do this with -u $UID and -w and will tell you immediately if this works.

I just run this with this command:

RUNNING: docker run --rm -e DOCKER_VERSION_8395080871=19.03.13 -it -u 501 -v /Users/alihassoun/Desktop/bids/fsl-license/license.txt:/opt/freesurfer/license.txt:ro -v /Users/alihassoun/Desktop/bids/oasis:/data:ro -v /Users/alihassoun/Desktop/bids/output:/out -v /Users/alihassoun:/scratch nipreps/smriprep:0.7.0 /data /out participant --fs-no-reconall --sloppy -w /scratch

[ali3654]

unfortunately it did not work and the same error message appeared

My Bids folder:

├── oasis
│   ├── dataset_description.json
│   ├── sub-OAS30001
│   │   └── ses-d0129
│   │       └── anat
│   │           ├── sub-OAS30001_ses-d0129_run-02_T1w.json
│   │           └── sub-OAS30001_ses-d0129_run-02_T1w.nii.gz
│   └── task-rest_bold.json

Did it work for you with the Mac ?
This is so strange, do you have any idea why this is only the case with me?

I have actually considered everything.

[effigies]

Yeah, it is running to completion on my mac.

smriprep/
├── dataset_description.json
├── logs
│   ├── CITATION.bib
│   ├── CITATION.html
│   ├── CITATION.md
│   └── CITATION.tex
├── sub-OAS30001
│   ├── figures
│   │   ├── sub-OAS30001_desc-about_T1w.html
│   │   ├── sub-OAS30001_desc-summary_T1w.html
│   │   ├── sub-OAS30001_ses-d0129_run-2_desc-conform_T1w.html
│   │   └── sub-OAS30001_ses-d0129_run-2_dseg.svg
│   └── ses-d0129
│       └── anat
│           ├── sub-OAS30001_ses-d0129_run-2_desc-brain_mask.json
│           ├── sub-OAS30001_ses-d0129_run-2_desc-brain_mask.nii.gz
│           ├── sub-OAS30001_ses-d0129_run-2_desc-preproc_T1w.json
│           ├── sub-OAS30001_ses-d0129_run-2_desc-preproc_T1w.nii.gz
│           ├── sub-OAS30001_ses-d0129_run-2_dseg.nii.gz
│           ├── sub-OAS30001_ses-d0129_run-2_label-CSF_probseg.nii.gz
│           ├── sub-OAS30001_ses-d0129_run-2_label-GM_probseg.nii.gz
│           └── sub-OAS30001_ses-d0129_run-2_label-WM_probseg.nii.gz
└── sub-OAS30001.html

I really don't know what else to suggest. Have you checked the md5 of your T1w file to ensure it's identical to mine?

558799535d887ccf93b539f4427f3343  sub-OAS30001/ses-d0129/anat/sub-OAS30001_ses-d0129_run-02_T1w.nii.gz

Only other thing I can think is just to try on another machine. Maybe there's something really baroque about your setup?

[ali3654]

yes I have controlled it:

MD5 (sub-OAS30001_ses-d0129_run-02_T1w.nii.gz) = 558799535d887ccf93b539f4427f3343

So Strange, the Mac is also only 5 months old and is a Mac Catalina.

thanks a lot for your great support.

[ali3654]

do you think this has something to do with MacOs Catalina?

[effigies]

My mac is also on catalina, so I doubt it's that. But there are a lot of weird permissions issues that I've had to deal with over 3 years that may explain the difference.

[ali3654]

yes I can imagine.
what i just found out is that if i remove the .gz extension of the file sub-OAS30001_ses-d0129_run-2_desc-preproc_T1w.nii.gz i can zip it again and the file is executable and i can unzip it again and i can even open it with nibabel.

Hahaha I don't know if something like that helps you but it's worth telling you.

And I am still very grateful for the fast and good support.

[effigies]

Yeah, my rough diagnosis is that somewhere internal to the workflow, we're writing out an uncompressed NIfTI file to .nii.gz, which is getting passed around until finally some interface tries to look at it.

The fact that you're seeing it consistently while I am utterly unable to reproduce it using the same Docker image on the same file is what makes this so hard. I may not be able to until Monday, but I will try to give you a process that will allow you to bundle up your working directory so I can examine it.

[ali3654]

Yes I understand what you mean. I also find it strange that it works great for you and not for me although I tried everything.

I am looking forward to your process that you will give me.

Thanks a lot in advance.

[effigies]

Okay, here's the procedure:

1) Create a file nipype.cfg:

[execution]
crashfile_format = pklz

2) Execute with -w pointing somewhere and specifying the config file with --config:

smriprep-docker $DATA $OUT participant --fs-no-reconall --sloppy -w $SCRATCH --config nipype.cfg

3) Zip the $SCRATCH and the $OUT/smriprep/sub-OAS30001/log/ directories, and share them with Google Drive or DropBox or something. You can use this username @ gmail to share with me.

[ali3654]

ok thanks i will do that now.
But where should nipype.cfg be saved and should I include this file in the dropbox?

[effigies]

Oh, right. I'm sorry, updated step 2.

[ali3654]

i have now sent this to you by dropbox according to this email address.

If this did not arrive, let me know and I will share it with you again.

If you have any questions or if there is anything missing write me and I will send it to you later.

thank you again very much

[effigies]

Thanks for this. I think you got the wrong log directory, though.

[ali3654]

oh sry you are right, i will send you the right one right away

[ali3654]

i have deleted the other one and uploaded the correct log file. Did you get it ?

[effigies]

Got it, thanks. Looking at it, it looks like the nipype.cfg did not successfully configure it to produce pklz crash files instead of txt, which makes things difficult. The inputs to these interfaces seem reasonable, so nothing's clear yet...

We're going to need to patch sMRIPrep. If you don't have sMRIPrep cloned:

git clone [email protected]:nipreps/smriprep.git
cd smriprep

Then add my remote, fetch, and checkout a test branch:

git remote add effigies [email protected]:effigies/smriprep.git
git fetch effigies
git checkout effigies/fix/debugging

Then run smriprep-docker as you did before, but adding the flag: --patch-smriprep smriprep (Note that this is the directory smriprep inside the repository, not the repository itself. If you cloned to /projects/smriprep, then you could use /projects/smriprep/smriprep.)

This should produce *.pklz crashfiles that you can send.

[ali3654]

Ok I did that and I uploaded this in the shared dropbox folder.

If you didn't get it and need anything else please let me know.

Thanks a lot in advance.

[ali3654]

Hello @effigies,

and did you find the problem ?

Thanks in advance

[effigies]

Not yet, sorry. I'll try to have a quick look tomorrow, but if I can't, I will be out until Dec 1.

[ali3654]

Hello @effigies ,
sorry to bother you but have you seen what the problem is?

Thanks in advance

[effigies]

Thanks for the ping. I did not get a chance before the break. Let me check in on this.

[effigies]

Unfortunately, when I try rerunning those nodes, they succeed, presumably because the relevant files have been correctly generated on my machine.

I think the best bet is to try to package up /scratch/smriprep_wf/single_subject_OAS30001/anat_preproc_wf/brain_extraction_wf and send that along. Maybe we can track down where the failure was introduced. (This is ~580M on my machine.)

[ali3654]

ok i compressed the folder and uploaded it again in the dropbox folder. you should still have access to it.

[effigies]

Thanks. I looked at it, and the files inside are all fine. The error must be happening when copying the data to your output folder, and the failure is happening when trying to open that file to verify its contents. Could you try pointing your output directory somewhere else, possibly on a different drive?

[ali3654]

Ok i'll try to save my output file to another hard drive and let you know, thanks.

[ali3654]

wow, i ran this with the same laptop but on an external hard drive and it worked.
Thanks a lot but do you know what the problem with my built-in hard drive could be?

Do you think if I format my laptop, it will work or is there another solution?
I have one mac pro 2019-2020

[ali3654]

thanks a lot again now we know at least what the reason is.

[effigies]

I really don't know why it might be happening. It seems like it must be a filesystem issue, but especially when adding volume mounting via Docker, there are a bunch of abstraction layers between the write call and the disk (and then a subsequent read call) where something might go wrong.

It looks like on Mac you can check the SMART status through the disk utility and run some checks: https://osxdaily.com/2018/05/31/how-check-smart-status-mac-hard-disk/

I'd probably do that before considering reformatting.

[ali3654]

ok thanks, you really helped me a lot.
i will take your tips and try to solve this.