Deepreg: Illegal instruction (core dumped) with Intel(R) Pentium(R) CPU N3540 @ 2.16GHz

[ciphercharly]

Subject of the issue

getting Illegal instruction (core dumped) when installation is done on a laptop with Intel(R) Pentium(R) CPU N3540 @ 2.16GHz

If the bug is confirmed, would you be willing to submit a PR? _(Help can be provided if
you need assistance submitting a PR)_

No

Your environment

• DeepReg version (commit hash or tag): 110ef6f923bc79e710d7279797b01db61fcf1357

• OS: Linux 5.4.0-51-generic #56-Ubuntu x86_64 x86_64 x86_64 GNU/Linux

• Python Version: 3.7.9

• Tensorflow: 2.3.1 (as in requirements.txt)

Steps to reproduce

installed by cloning repo locally and creating virtual env with conda env create -f environment_cpu.yml
and then running pip install -e .
(but tried also with pypl pip install deepreg and same)
running any deepreg command results in the above errorIllegal instruction (core dumped)
e.g. python3 demos/paired_MMIV/demo_data.py
or deepreg_train --config_path demos/grouped_mr_heart/grouped_mr_heart.yaml --log_dir grouped_mr_heart

[mathpluscode]

Hi @ciphercharly, thank you for the report again!

Could you clarify the version of the code? As the conda env create -f environment_cpu.yml corresponds to the latest version which based on main while the tag you reported is 0.1.0b1 ;)

[ciphercharly]

good question... that is the version I get from running pip list after cloning the repo, creating the environment with conda env create -f environment_cpu.yml and then running pip install -e . from the created DeepReg folder.. I guess I cloned the wrong repo? I copy-pasted the link in the docs: git clone https://github.com/DeepRegNet/DeepReg.git

[mathpluscode]

good question... that is the version I get from running pip list after cloning the repo, creating the environment with conda env create -f environment_cpu.yml and then running pip install -e . from the created DeepReg folder.. I guess I cloned the wrong repo? I copy-pasted the link in the docs: git clone https://github.com/DeepRegNet/DeepReg.git

@ciphercharly

git clone will get the latest version, therefore you are using main, not the tag.
To verify, could you run git rev-parse HEAD to get the hash of the commit and update this issue (and maybe also #452)?

I'm trying to reproduce this locally.

[ciphercharly]

yep I will check that tomorrow as soon as I am back to the machine
I wonder why pip list would say 0.1.0b1 then.. but I'll run git rev-parse HEAD as you suggest and update correspondingly

[mathpluscode]

yep I will check that tomorrow as soon as I am back to the machine
I wonder why pip list would say 0.1.0b1 then.. but I'll run git rev-parse HEAD as you suggest

@ciphercharly

It's because when you do pip install -e . it will install the local version with tag from setup.py where we have put 0.1.0b1.

Btw, I just tried the example, one potential problem you have might be OOM (out of memory).

For instance, with the default config, I have

Epoch 1/6000
2020-10-15 19:25:20.974014: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 117440512 exceeds 10% of free system memory.
2020-10-15 19:25:21.064712: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 117440512 exceeds 10% of free system memory.
2020-10-15 19:25:21.197227: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 117440512 exceeds 10% of free system memory.
2020-10-15 19:25:21.283221: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 117440512 exceeds 10% of free system memory.
2020-10-15 19:25:21.719961: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 117440512 exceeds 10% of free system memory.

Then I went to the config file demos/grouped_mr_heart/grouped_mr_heart.yaml, change the image_shape to [32,32,28] instead of [128, 128, 28], it works fine for me.

Part of the config is as follows,

dataset:
  dir:
    train: "demos/grouped_mr_heart/dataset/train"
    valid: "demos/grouped_mr_heart/dataset/val"
    test: "demos/grouped_mr_heart/dataset/test"
  format: "nifti"
  type: "grouped" # paired / unpaired / grouped
  labeled: false
  intra_group_prob: 1
  intra_group_option: "unconstrained" # forward / backward / unconstrained
  sample_image_in_group: true
  image_shape: [32, 32, 28]

[ciphercharly]

yes I got similar out of memory errors from another machine (that I don't use anymore now), but on the laptop in question I think it might be a problem related to tensorflow > 2.2 somehow incompatible with that CPU specifically, but I am not sure..
anyway Illegal instruction (core dumped) is literally the only thing that gets printed at stdout and it happens immediately

[mathpluscode]

yes I got similar out of memory errors from another machine (that I don't use anymore now), but on the laptop in question I think it might be a problem related to tensorflow > 2.2 somehow incompatible with that CPU specifically, but I am not sure..
anyway Illegal instruction (core dumped) is literally the only thing that gets printed at stdout and it happens immediately

@ciphercharly OK. That's strange. If later you can try again and provide the full log, maybe we can check together? For now, as I couldn't reproduce it, I'm not sure whether if I can fix it.

[ciphercharly]

right, I'll try to see if more info are logged. I fear this might be a machine-specific issue which is out of your control..

[ciphercharly]

update: I have put in the correct version tag, as I said this might be a machine-specific issue but an older version of deepreg was working (like 2 weeks ago), so I don't know what went wrong recently, could be tensorflow (I tried to downgrade it to 2.2 and same)

here a simple test I tried from the python console (all direct calls to deepreg_train or deepreg_predict result in the same):

python
Python 3.7.9 (default, Aug 31 2020, 12:42:55) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> a="test"
>>> print(a)
test
>>> import deepreg
>>> import deepreg.train
Illegal instruction (core dumped)

there are no more info/logs that I can easily find (maybe something more from anaconda?)
but anyway, I'll just drop the attempt to use deepreg with this laptop for now

[ciphercharly]

I checked that tensorflow fails upon importing
python -c 'import tensorflow as tf; print(tf.__version__)'
Illegal instruction (core dumped)

the issue seems to be this https://tech.amikelive.com/node-887/how-to-resolve-error-illegal-instruction-core-dumped-when-running-import-tensorflow-in-a-python-program/, but then nothing above 1.5 should work and I don't think you were using 1.5 two weeks ago.. so I can't explain why I was able to use deepreg on this laptop previously

[mathpluscode]

I checked that tensorflow fails upon importing
python -c 'import tensorflow as tf; print(tf.__version__)'
Illegal instruction (core dumped)

the issue seems to be this https://tech.amikelive.com/node-887/how-to-resolve-error-illegal-instruction-core-dumped-when-running-import-tensorflow-in-a-python-program/, but then nothing above 1.5 should work and I don't think you were using 1.5 two weeks ago.. so I can't explain why I was able to use deepreg on this laptop previously

We've updated the tf from 2.2 to 2.3, so this might be the reason? If you pip install tensorflow==2.2.0, will it work?

[mathpluscode]

Hi, @ciphercharly are you still facing problems regarding this issue?
Btw, we've updated the tf to 2.3.1, which has the same requirement as 2.2.x.

[ciphercharly]

I have update to latest and same yeah, I can't use it anymore from this specific laptop, I think it's beyond repair

[mathpluscode]

I have update to latest and same yeah, I can't use it anymore from this specific laptop, I think it's beyond repair

Sorry to hear that.

In this case I will close this issue :)