Keras-retinanet: Viewing the training data size (shape) that is being fed into the network

[sghosal93]

I am working with a custom dataset and would like to view the shape of the data cube that is being fed into the network before training starts using something similar to X_train.shape. How do I do this while invoking the train.py script available in this repository?

Also what exactly does "steps" (default value = 10000) mean here?

[hgaiser]

This is where the input and target blobs are created. You could print their shape there.

Steps means the number of batches to process per epoch.

[sghosal93]

Thanks. So, 10k steps means it'd process 10k batches per epoch? Also, what's the relation between training data, batch size and steps here? For instance, if batch size = 1, does that mean that given I have 30k training images with their corresponding labels (bounding boxes in my case, where there are multiple bounding boxes per image), will a step size of 10k entail that only 10k out of my 30k image dataset will be used for training?

[mgwizdala]

Exactly like you said. With batch size equal to one, single step means that one image from your dataset will be processed.

Given a dataset of 30k images you should set step_size to 30k if you want to use all your data in each epoch (and i guess it's right thing to do). Number of annotations on image does not matter.

If you use larger batch size, you should decrease steps count by dividing images count / batch size

[sghosal93]

Got it. Thanks for confirming this. I have one more query here. What will the training situation be like if my data size is less than the step size? For example, if step size is 10k, where as my entire data size is 5k images? Will it just train the entire 5k images training data twice per epoch in that case?

[mgwizdala]

Yes it will. When CSV generator is created it splits your dataset into groups with size equal to your provided batch size. So if your dataset have 30k examples and you have choosed batch size of 1 it will create 30k of groups.
In file linked by Hans you can see in next() method that current group index is calculated like this:

   if self.group_index == 0 and self.shuffle_groups:
                # shuffle groups at start of epoch
                random.shuffle(self.groups)
            group = self.groups[self.group_index]
            self.group_index = (self.group_index + 1) % len(self.groups)

So when you start a new epoch or if your steps count is larger than you dataset, groups will be shuffled and then training will continue. But in general you should try to avoid that. Try to keep it simple and do not use same file twice in single epoch, just because it doesn't feel right.

[ShibaPrasad]

vizer1993 : How you changed the batch size on the main program. I have only 5500 images and need to train ....... but by default the number of epochs are 50 and also number of iterations are: 10000

[hgaiser]

Did you check retinanet-train --help?

On Mon, 3 Dec 2018, 18:50 Shiba Kuanar <[email protected] wrote:

vizer1993 : How you changed the batch size on the main program. I have
only 5500 images and need to train ....... but by default the number of
epochs are 50 and also number of iterations are: 10000

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/fizyr/keras-retinanet/issues/510#issuecomment-443803102,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AArtal3Ax2Tf1Z0HmVtdbIGtHXSH94p1ks5u1WR7gaJpZM4UszIR
.