I merged PR #813 this morning and test_sampler.py is failing after the merge on master in one of the CI configuration (Matrix Test of Python 3.6 on ubuntu-18.04) here. The test was passing on my branch previous to the merge.
Looking back at previous merges and PRs, it seems that this test is failing often and in different configurations, recent examples here in Matrix Test of Python 3.6 on ubuntu-20.04 and here in test (3.7, integration-test). I'm not familiar with the feature but there may be an bug with it or with the test itself that makes it fail unexpectedly.
So it seems the underlying process is probabilitic:



but the assertion ensures that unbalanced results will randomly become more balanced eventually but in some odd CI state, it did not. Best to run this issue many times locally and see what is up with that.

Testing 100x repeat:

Testing 1000x repeat:

This suggests, that we should probably redesign the test to avoid this 0.4% or about 1 in 200 chance of failure. Good catch @mariehbourget !
All fail scenarios are the same, with the edge case illustrated.

good digging @dyt811 !
the only difference between the two runs of this test is that one uses random sampler:
```
def __iter__(self):
n = len(self.data_source)
if self.replacement: #False
return iter(torch.randint(high=n, size=(self.num_samples,), dtype=torch.int64).tolist())
return iter(torch.randperm(n).tolist())
the other one uses balanced multinomial sampler:
def __iter__(self):
return (self.indices[i] for i in torch.multinomial(
self.weights, self.nb_samples, replacement=True))
```
and if they happen to generate the same numbers we have an error such as above, (if you replace the second __iter__ with torch.randperm(n).tolist() such as above you get the same error), suggest change < to <=
@cakester good catch.
Glad the fix wasn't super involved. Going to run a 3000+ retry to ensure it stays that way before approving the PR and we can wrap this up.
BTW @pytest.mark.repeat(1000) from pytest-repeat is PRETTY cool for this.
Most helpful comment
So it seems the underlying process is probabilitic:



but the assertion ensures that unbalanced results will randomly become more balanced eventually but in some odd CI state, it did not. Best to run this issue many times locally and see what is up with that.
Testing 100x repeat:

Testing 1000x repeat:

This suggests, that we should probably redesign the test to avoid this 0.4% or about 1 in 200 chance of failure. Good catch @mariehbourget !
All fail scenarios are the same, with the edge case illustrated.
