Ivadomed: test_sampler.py failing unexpectedly

Created on 14 Jun 2021  路  4Comments  路  Source: ivadomed/ivadomed

Issue description

I merged PR #813 this morning and test_sampler.py is failing after the merge on master in one of the CI configuration (Matrix Test of Python 3.6 on ubuntu-18.04) here. The test was passing on my branch previous to the merge.

Looking back at previous merges and PRs, it seems that this test is failing often and in different configurations, recent examples here in Matrix Test of Python 3.6 on ubuntu-20.04 and here in test (3.7, integration-test). I'm not familiar with the feature but there may be an bug with it or with the test itself that makes it fail unexpectedly.

bug testing

Most helpful comment

So it seems the underlying process is probabilitic:
image
image
image

but the assertion ensures that unbalanced results will randomly become more balanced eventually but in some odd CI state, it did not. Best to run this issue many times locally and see what is up with that.

image

Testing 100x repeat:
image

Testing 1000x repeat:
image

This suggests, that we should probably redesign the test to avoid this 0.4% or about 1 in 200 chance of failure. Good catch @mariehbourget !

All fail scenarios are the same, with the edge case illustrated.
image

All 4 comments

So it seems the underlying process is probabilitic:
image
image
image

but the assertion ensures that unbalanced results will randomly become more balanced eventually but in some odd CI state, it did not. Best to run this issue many times locally and see what is up with that.

image

Testing 100x repeat:
image

Testing 1000x repeat:
image

This suggests, that we should probably redesign the test to avoid this 0.4% or about 1 in 200 chance of failure. Good catch @mariehbourget !

All fail scenarios are the same, with the edge case illustrated.
image

good digging @dyt811 !

the only difference between the two runs of this test is that one uses random sampler:
```
def __iter__(self):
n = len(self.data_source)
if self.replacement: #False
return iter(torch.randint(high=n, size=(self.num_samples,), dtype=torch.int64).tolist())
return iter(torch.randperm(n).tolist())

the other one uses balanced multinomial sampler:
def __iter__(self):
    return (self.indices[i] for i in torch.multinomial(
        self.weights, self.nb_samples, replacement=True))

```
and if they happen to generate the same numbers we have an error such as above, (if you replace the second __iter__ with torch.randperm(n).tolist() such as above you get the same error), suggest change < to <=

@cakester good catch.

Glad the fix wasn't super involved. Going to run a 3000+ retry to ensure it stays that way before approving the PR and we can wrap this up.

BTW @pytest.mark.repeat(1000) from pytest-repeat is PRETTY cool for this.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

andreanne-lemay picture andreanne-lemay  路  3Comments

copperwiring picture copperwiring  路  4Comments

andreanne-lemay picture andreanne-lemay  路  4Comments

AnBucquet picture AnBucquet  路  5Comments

charleygros picture charleygros  路  5Comments