Deepreg: Randomness in training - Unpaired data_loader

[acasamitjana]

Issue description

Randomness in batches during training. The unpaired data_loader loads the same pair of images (reference/floating) in every epoch.
This should be tested in issue #127

Type of Issue

• [x] Bug report

What's the expected result?

During training, a different combination of reference/floating are seen in every epoch.

[NMontanaBrown]

Also relevant to @s-sd: does this behaviour occur with the other loaders we have tested thus far (h5, nifti)?

[acasamitjana]

Indeed, with any loader called from the Unpaired_loader class. It will be solved a next PR.

[s-sd]

@acasamitjana, I tired to test the unpaired loader using the output test and tried printing the order used. For me, this gave me the a different order (or different image pair) on each epoch. Did you test using a seed value of None? If possible, could you let me know the steps to recreate the bug?

@NMontanaBrown, the nifti and h5 loaders are general file loaders so they do not generate the sample indices which is why they do not have any random behaviour on their own. Coupled with one of the other loaders (like unpaired) they should load files in a random order in train mode since the seed is set to None.

[acasamitjana]

@s-sd I run the test_unpaired_labeled and printing the output of data_loader_train.sample_index_generator for several epochs and it gave the same indices for all of them. I'm using seed=None but it has no effect in the output as it doesn't use the shuffled list. Am I right? How did you test it?

[s-sd]

@acasamitjana, Thanks! I tested by printing the moving_index and fixed_index just before line 78 in 'deepreg/dataset/loader/unpaired_loader.py'. I ran the test 'test/output/test_unpaired_labeled.py'. I got different indices while training and same indices in each epoch when in test mode. Is it possible that you observed the indices in test mode?

'test/output/test_unpaired_labeled.py' tests both the train mode and test mode. In train mode, the order is supposed to be different and in test mode the order is supposed to be the same due to a seed of 0 being used. You can also try to run only the training and observe what is printed out in that case. You can run only the training on the nifti data from a python prompt by entering this:

from deepreg.train import train
config_path=[
    "deepreg/config/test/ddf.yaml",
    "deepreg/config/test/unpaired_nifti.yaml",
    "deepreg/config/test/labeled.yaml",
]
gpu = ""
gpu_allow_growth = False
ckpt_path = ""
log_dir = "train_" + 'test1'
train(
    gpu=gpu,
    config_path=config_path,
    gpu_allow_growth=gpu_allow_growth,
    ckpt_path=ckpt_path,
    log_dir=log_dir,
)

Let me know if you get a different order when you run just the training on nifti data

[acasamitjana]

@s-sd thanks for the detailed explanation. I could run the script printing the indices in line 78 and I get the same image pairs [moving_index, fixed_index] for every epoch: [0,1], [2,3], ..., [12,13].
Are you using the master branch? In master, the indices, as far as I understand, are not random.

for sample_index in range(self.num_samples):
    moving_index, fixed_index = 2 * sample_index, 2 * sample_index + 1
    yield moving_index, fixed_index, [moving_index, fixed_index]

[s-sd]

@acasamitjana Yes you are right. Sorry, the printed outputs got a bit jumbled up in the output test. I have now just run only the training and have realised that that the shuffled list is not being used and the outputs are not shuffled at each epoch. Sorry for the confusion. Happy to help correct this if you want or if you need help with it :)

[acasamitjana]

@s-sd don't worry and thanks for the help and discussion. I already submitted a PR with updated changes and the unit testing for the unpaired data loader. I will close the issue when the PR is accepted.

[mathpluscode]

        image_indices = [i for i in range(self.num_images)]
        random.Random(self.seed).shuffle(image_indices)
        for sample_index in range(self.num_samples):
            moving_index, fixed_index = 2 * sample_index, 2 * sample_index + 1
            yield moving_index, fixed_index, [moving_index, fixed_index]

lol i shuffled it in place and forgot to use it...