Caffe: Train the dataset in random order each time?

[dreadlord1984]

I found that the train batch samples were trained by the order saved in level-db. The test dataset have the same law. I want to know how to train the samples in random order.

[Yangqing]

Randomly reshuffle your leveldb might be the easiest. the reason things are sequentially read is for performance purpose - random access on conventional HDDs is near disaster.

[dreadlord1984]

Thank you for your replay. I'm not familiar with level-db,so i don't know what's your mean "Randomly reshuffle your leveldb". I had save the samples in level-db by random order,but in the training process, the batchsize samples always feed the network in the same order. I don't know how much influence on network performance when just tarining it in same samples order (Because my trained network is not better,so i want to find the reason).
In Data_layer.cpp:InternalThreadEntry(), when go to the next iter, the program use "iter_->Next(); "whether have some way to read a random datum in level-db ?
Additionally, congratulate your team rank 1 in ILSVRC2014's classification task. Nice job.
Thanks.

[shelhamer]

You can't do random seeking in leveldb because it's counter to its purpose as a storage engine: maximize throughput for sequential reads.

If you want true random re-sampling of mini-batches you can feed the net with an ImageDataLayer when configured for shuffling. If prefetching with this configuration is still faster than your forward / backward passes then no problem, otherwise you'll be slowing down training.

[sguada]

I did some timings and using level_db it takes 250ms to do prefetching
while using ImageDataLayer with shuffle takes 2800ms. Since the
forward-backward pass takes ~1000ms then training with ImageDataLayer will
slow your training by a factor of 3-4x.

Sergio

2014-09-21 14:37 GMT-07:00 Evan Shelhamer [email protected]:

You can't do random seeking in leveldb because it's counter to its purpose
as a storage engine: maximize throughput for sequential reads.

If you want true random re-sampling of mini-batches you can feed the net
with an ImageDataLayer when configured for shuffling. If prefetching with
this configuration is still faster than your forward / backward passes then
no problem, otherwise you'll be slowing down training.

—
Reply to this email directly or view it on GitHub
https://github.com/BVLC/caffe/issues/1087#issuecomment-56313682.

[dreadlord1984]

Thank you for you advice. Now i know how to feed the network randomly.

[sajjo79]

I am using my own input data layer developed in python. I did two experiments one with sequential batch and other with random batch strategy. Sequential batch i.e. takes 240 images in one step and in next it takes next 240 images and when data set ends, it starts from 0th image again for second epoch. In random image batch, every image is selected at random from data image set. I found random batch strategy more converging as compared to sequential batch strategy. Can anyone comment on it?