Ivadomed: Prediction volume missing one slice with command test

[mariehbourget]

Issue description

While working on PR #813 and more generally on inference for microscopy #306, I found something odd in the code in the run_inference function of testing.py, and I would like a second opinion about that before making changes (@andreanne-lemay @charleygros)

In brief, the prediction volume created and saved into a nifti file is missing the last slice.

Current behavior

When running the command test, in the prediction for loop, predictions for each slices are added to pred_tmp_lst at line 253.

However, when running the loop for the last sample of a volume (last_sample_bool=True), the "new complete volume" is treated before that at line 213. So for example, with a volume of 16 slices, the pred_tmp_lst at this stage has a length of 15 (missing the last slice).
EDIT: for MRI, if there are more than one volume to segment this only happens for the last volume of the batch because of last_sample_bool=True.

The resulting nifti file saved in pred_to_nib and the output_data here have 16 slices, but the last one is empty (seems to be filled by zeros in pred_to_nib here)

Expected behavior

I would have expected the append to pred_tmp_lst to happen before "new complete volume" to include the last slice predictions.

For example in 2D microscopy, I have only 1 slice, so 1 iteration of the for loop and my pred_tmp_lst is empty as it gets to "new complete volume" before the append.
EDIT: for microscopy, there is only one slice per volume. So the behavior above happens if I have only one volume to segment. If I try segmenting 2 volumes, the first one is OK, but the second one doens't work because of last_sample_bool=True.

Moreover, because the assignement of fname_tmp is after "new complete volume", the fname_refpassed to pred_to_nib for the first volume is not the good one (my 2 class are inverted: axon ref with myelin pred, and myelin ref with axon pred). So it seems that the condition(fname_ref != fname_tmp or last_sample_bool) here cause two separate issues. EDIT2: the mixup between predictions and GT was related to another issue, see #836).

The command segment doesn't use run_inference but segment_volume and reconstruct_3d_object. While running the same model with the segment command, the volume is complete and all the 16 slices have predictions in them.

Also, if there is indeed a slice missing in the prediction, this will surely impact the computed 3D metrics, no?

Steps to reproduce

1. Run a 2D model with command test and check the resulting nifti file (last slice is all zeros).
2. Run the same 2D model with command segment and check the resulting nifti file (no missing slice in the prediction).

[dyt811]

@mariehbourget, hmm... since this is best validated in 2D images as that is mostly impacted, do you have an example data set/trained model/tutorial on 2D processing pipeline so we can start building a test apparatus around it to catch this bug in the future? If not, do you have a recommendation of any particular existing data set (in either data_testing / data_funcitonal_testing)? So we can see. I wonder if we can use a tutorial dataset to test this out as well.

[mariehbourget]

@dyt811, I'm working on this issue right now. I'll open a PR later today with a fix.
As for testing, we don't have a trained 2D model to test yet (as the pipeline is not entirely finisehd). We could test on a single 3D volume (the last slice would not be all zeros) with the current data_testing dataset. i'll check what is best for testing, if needed, I'll train a 2D model to add to the testing dataset.

[dyt811]

Yeah, good point. I mean, if you got some input/config.json setup and can train and model, we can help build test cases if you wish. Let us know. Happy to help. I will keep an eye out for this missing slice issue when I get to the prediction/testing stage. For now, still on the input side of things. :D

[mariehbourget]

@dyt811, turns out I was able to add a relevant test without adding anything to the testing dataset (see PR #835). Thank you for the offer, I'll let you know if I need help with test for other PRs.