Datasets: Add a method to shuffle a dataset

Created on 19 May 2020  路  4Comments  路  Source: huggingface/datasets

Could maybe be a dataset.shuffle(generator=None, seed=None) signature method.

Also, we could maybe have a clear indication of which method modify in-place and which methods return/cache a modified dataset. I kinda like torch conversion of having an underscore suffix for all the methods which modify a dataset in-place. What do you think?

generic discussion

Most helpful comment

+1 for the naming convention

About the shuffle method, from my understanding it should be done in Dataloader (better separation between dataset processing - usage)

All 4 comments

+1 for the naming convention

About the shuffle method, from my understanding it should be done in Dataloader (better separation between dataset processing - usage)

+1 for shuffle in Dataloader.
Some Dataloader just store idxs of dataset and just shuffle those idxs, which might(?) be faster than do shuffle in dataset, especially when doing shuffle every epoch.

Also +1 for the naming convention.

As you might already know the issue of dataset shuffling came up in the nlp code walkthrough by Yannic Kilcher

We added the .shuffle method :)

Closing this one.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

astariul picture astariul  路  3Comments

david-waterworth picture david-waterworth  路  3Comments

NielsRogge picture NielsRogge  路  3Comments

astariul picture astariul  路  6Comments

ratthachat picture ratthachat  路  6Comments