Itk: Issue when calling `read` in multi-threads program

[Nic-Ma]

Description

Hi @aylward and @thewtex ,

I am trying to call the itk.imread in a multi-threads python program to load images in parallel, and got error:

--> 128             img_.append(itk.imread(name, **kwargs))
    129         return img_ if len(filenames) > 1 else img_[0]
    130 

/opt/conda/lib/python3.6/site-packages/itkExtras.py in imread(filename, pixel_type, fallback_only)
    626         kwargs = {'FileNames': filename}
    627     else:
--> 628         TemplateReaderType = itk.ImageFileReader
    629         io_filename=filename
    630         increase_dimension=False

/opt/conda/lib/python3.6/site-packages/itkLazy.py in __getattribute__(self, attr)
     50             module = self.__belong_lazy_attributes[attr]
     51             namespace = {}
---> 52             itkBase.LoadModule(module, namespace)
     53             self.loaded_lazy_modules.add(module)
     54             for k, v in namespace.items():

/opt/conda/lib/python3.6/site-packages/itkBase.py in LoadModule(name, namespace)
     59         if namespace is not None:
     60             swig = namespace.setdefault('swig', {})
---> 61             swig.update(this_module.swig)
     62 
     63             # don't worry about overwriting the symbols in namespace -- any

Seems something crashed in C++ layer?
Could you please help take a look at this multi-threads issue?
@wyli and I still work on the development of MONAI v0.3, we are close to code-freeze next week.

Thanks in advance.

[blowekamp]

Which file type are you trying to read? Some ImageIOs are not concurrent thread safe.

[Nic-Ma]

I read Nifti data from Decathlon dataset, is it supported?
@aylward @thewtex and @wyli, if not thread safe, we may need to temporarily change the default reader of MONAI?

Thanks.

[blowekamp]

I have had concurrency problem with GDCM, and some on OSX with MetaImageIO. It is always possible to use heavy weight processes, then some type of pickling to get the results back to the main process.

[hjmjohnson]

The problem may not be with "Reading images" but rather with the itkLazy loading of modules.

@Nic-Ma would it be possible to do a small test? Change the import itk to

import itkConfig
itkConfig.LazyLoading = False
import itk

This will make loading the ITK module much slower, but perhaps may identify the problem better.

Additionally, could you provide an example code to allow us to better diagnose the problem?

Hans

[Nic-Ma]

Hi @hjmjohnson and @blowekamp ,

I tried disable the lazyloading as you suggested, still got the same error:

--> 131             self._img.append(itk.imread(name, **self.kwargs))
    132         return self._img if len(filenames) > 1 else self._img[0]
    133 

/opt/conda/lib/python3.6/site-packages/itkExtras.py in imread(filename, pixel_type, fallback_only)
    626         kwargs = {'FileNames': filename}
    627     else:
--> 628         TemplateReaderType = itk.ImageFileReader
    629         io_filename=filename
    630         increase_dimension=False

/opt/conda/lib/python3.6/site-packages/itkLazy.py in __getattribute__(self, attr)
     50             module = self.__belong_lazy_attributes[attr]
     51             namespace = {}
---> 52             itkBase.LoadModule(module, namespace)
     53             self.loaded_lazy_modules.add(module)
     54             for k, v in namespace.items():

/opt/conda/lib/python3.6/site-packages/itkBase.py in LoadModule(name, namespace)
     59         if namespace is not None:
     60             swig = namespace.setdefault('swig', {})
---> 61             swig.update(this_module.swig)
     62 
     63             # don't worry about overwriting the symbols in namespace -- any

You can easily run this notebook to reproduce the issue:
https://github.com/Project-MONAI/Tutorials/blob/add-nnunet-tutorial/nnunet_tutorial.ipynb
Just replace the LoadNiftid transform with LoadImaged, then it will crash in CacheDataset cell with num_workers=4.

Thanks.

[Nic-Ma]

Update the test notebook link: https://github.com/Project-MONAI/Tutorials/blob/master/nnunet_tutorial.ipynb
I think @hjmjohnson is very familiar with CacheDataset logic.
And let me temporarily change the default reader of LoadImage to Nibabel.

Thanks in advance.

[Nic-Ma]

Hi @hjmjohnson ,

Could you please help share any update?
Thanks.

[hjmjohnson]

Hi @hjmjohnson ,

Could you please help share any update?
Thanks.

@Nic-Ma Bad timing for this request. Several core ITK python developers are on vacation for 2 weeks, and I have a MAJOR funded grant deadline on October 1st. Normally we would be able to respond more quickly, but we just don't have the bandwidth to dedicate to digging into this right now. We have scheduled a meeting for September 28th as a backstop to investigate if we are unable to get to it sooner.

Sorry.
Hans

[Nic-Ma]

Got it, thanks @hjmjohnson .

Hi @aylward , @wyli,

Can we temporarily change to do some auto-select logic instead of using ITK as IO default reader for all formats:
use ITK as default reader for DICOM, use Nibabel as default reader for Nifti, use PIL as default reader for PNG, JPG and BMP.

What do you think?

Thanks.

[aylward]

Yes - we probably need to pass for the 0.3 release of MONAI, but
hopefully @Matthew
McCormick [e-mail] or @Dzenan Zukic
[e-mail] can start looking into it.

On Wed, Sep 16, 2020 at 8:08 PM Nic Ma [e-mail] wrote:

Got it, thanks @hjmjohnson https://github.com/hjmjohnson .

Hi @aylward https://github.com/aylward , @wyli https://github.com/wyli
,

Can we temporarily change to do some auto-select logic instead of using
ITK as IO default reader for all formats:
use ITK as default reader for DICOM, use Nibabel as default reader for
Nifti, use PIL as default reader for PNG, JPG and BMP.

What do you think?

Thanks.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/InsightSoftwareConsortium/ITK/issues/2003#issuecomment-693730494,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACEJL744O7YZETHBGXFYWLSGFHPPANCNFSM4RLIOIKA
.

--
Stephen R. Aylward, Ph.D.

Senior Director of Strategic Initiatives

Kitware: Advancing the frontiers of understanding
by developing innovative open-source software platforms
and integrating them into research, processes, and products.

[Nic-Ma]

Hi @aylward ,

After internal discussion with @hjmjohnson , we decided to keep ITK as all default IO reader for v0.3 release.
Of course, if you experts can identify the issue and fix it soon, that would be wonderful.

Thanks.

[aylward]

Great!

It is our busy time of the year in the US with US government fiscal year
ending Oct 1, grant deadlines in September and October, MICCAI in early
Oct, and more. Hopefully we will get some free time soon. Thanks for
understanding!

Stephen

On Mon, Sep 21, 2020 at 8:47 AM Nic Ma [e-mail] wrote:

Hi @aylward https://github.com/aylward ,

After internal discussion with @hjmjohnson https://github.com/hjmjohnson
, we decided to keep ITK as all default IO reader for v0.3 release.
Of course, if you experts can identify the issue and fix it soon, that
would be wonderful.

Thanks.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/InsightSoftwareConsortium/ITK/issues/2003#issuecomment-696091181,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACEJLZ4BNVMYKBZJLEDLFDSG5DPXANCNFSM4RLIOIKA
.

--
Stephen R. Aylward, Ph.D.

Senior Director of Strategic Initiatives

Kitware: Advancing the frontiers of understanding
by developing innovative open-source software platforms
and integrating them into research, processes, and products.

[dzenanz]

Note: https://github.com/Project-MONAI/Tutorials/blob/master/nnunet_tutorial.ipynb is now https://github.com/Project-MONAI/tutorials/blob/master/modules/dynunet_tutorial.ipynb

[dzenanz]

Another similar issue has been reported on the forum.

[Nic-Ma]

Hi Guys,

Do you have any update here?
Thanks.

[hjmjohnson]

I'll have a look into this over the next week.

[blowekamp]

To rule out another similar bug, but likely different, can you set the environment variable ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=1 before running the minimal example case?

[aylward]

You might want to also look at work to speed up pytorch dataloader on windows: MONAI Issue 1300

There are also problems with pytorch pickle on windows for transforms using non-pickled constructs as parameters - that could be confusing issues when testing IO: MONAI Issue 1240

[hjmjohnson]

@Nic-Ma Do you have a failing version? The version referenced by @dzenanz above runs without failure on my computer.

https://github.com/Project-MONAI/tutorials/blob/master/modules/dynunet_tutorial.ipynb seems to be the replacement notebook, but it does not have CacheDataset or

How can I make a failure to debug?

Thanks.

Hans

[Nic-Ma]

Hi @hjmjohnson ,

You can use the notebook to test, the DecathonDataset in it inherits from CacheDataset, and please make sure you are using MONAI v0.3.0 to test the LoadImaged transform in it, we replaced the default reader in master.

Thanks.

[hjmjohnson]

• [X] Problem identified.
• [X] Minimal test case created.
• [X] proposed initial work workaround identified and being tested!

[hjmjohnson]

@Nic-Ma Fix propsed in #2194.

I think there is a quick hack fix for monai. Something like the following psuedo-code:

import itk
if itk.__version__< '5.2.1':
         # Warning the ITK LazyLoading mechanism was not threadsafe until version 5.2.1
          xx = itk.imread # A silly call to force loading the required modules BEFORE the multi-threading is called

[aylward]

Nice. I think it should be 5.1.2 for the version, but otherwise looks good.

[Nic-Ma]

Thanks @hjmjohnson ,

I submitted PR to MONAI: https://github.com/Project-MONAI/MONAI/pull/1324
Could you please help review it?

Thanks.

[dzenanz]

https://github.com/Project-MONAI/MONAI/pull/1324 has been merged. Is this issue now resolved?