Ssd.pytorch: StopIteration ERROR during training

[seongkyun]

My environment is

8GB RAM
Ubuntu 16.04 LTS
Pytorch 0.4 with CUDA 9.0 cuDNN v7
Python 3.5
Geforce GTX 1080 8GB.

I have geforce gtx 1080 8gb so i have tried to train network with 16 batch size.
And run the training with
python3 train.py --batch_size=16
after 1030 iteration,

.....
iter 1020 || Loss: 9.2115 || timer: 0.1873 sec.
iter 1030 || Loss: 8.1139 || Traceback (most recent call last):
File "train.py", line 255, in <module>
train()
File "train.py", line 165, in train
images, targets = next(batch_iterator)
File "/home/han/.local/lib/python3.5/site-packages/torch/utils/data/dataloader.py", line 326, in __next__
raise StopIteration
StopIteration

So i tried training with other batch size like 8, 20, everytime it prints out that signal.
and i calculated
batch_size * iteration step

then everytime the calculated number is around 16,480 with difference batch size and iter steps.

The problem occured part of pytorch dataloader is

if self.batches_outstanding == 0:
self._shutdown_workers()
raise StopIteration

in /home/han/.local/lib/python3.5/site-packages/torch/utils/data/dataloader.py.

Is it possible my 8GB RAM can occurs this problem?
(but i have checked RAM was enough with Ubuntu System Monitor)

Is there anybody can solve this problem?
Please help me guyz :)

[chenxinyang123]

I think it is because "batch_iterator" is used up.You may get len(batch_iterator)(it means the number of data/batch size) and use it to help you define a new batch_iterator. It is to say that you can use:batch_iterator = iter(data_loader) after len(batch_iterator) iterations

[seongkyun]

I fixed this problem.
I've changed the code in train.py
from
images, targets = next(batch_iterator)
to
try:
images, targets = next(batch_iterator)
except StopIteration:
batch_iterator = iter(data_loader)
images, targets = next(batch_iterator)

now, i checked all the losses are falling down :)

thanks. @chenxinyang123

[d-li14]

@seongkyun Unluckily, in my case, sometimes the training processs will diverge, i.e. the losses stopped falling down and became nan, and it seems to happen randomly

[seongkyun]

@d-li14 Maybe, the batch size could be problem. I don't know why this happened, but sometimes nan loss error is occurred with wrong batch size.

[d-li14]

Thanks a lot @seongkyun . FYI, I just try python train.py, i.e. with the default batch size number 32 in the script training on Pascal VOC dataset. And would you please elaborate how batch size can influence the convergence? We need to set batch size as large as possible to ensure robust training?

[sunny52juli]

It is because "batch_iterator" is used up, you should start a new "batch_iterator" as follows:
try:
image, mask, gt = [x.to(device) for x in next(iterator_train)]
except StopIteration:
iterator_train = iter(data.DataLoader(
dataset_train, batch_size=args.batch_size,shuffle=True, num_workers=int(args.n_threads),drop_last=True))
image, mask, gt = [x.to(device) for x in next(iterator_train)]

[sxyxf66]

@seongkyun Unluckily, in my case, sometimes the training processs will diverge, i.e. the losses stopped falling down and became nan, and it seems to happen randomly

I also have the same problem, do you solve it?

[seongkyun]

@sxyxf66 Maybe learning rate can occur that problem.
Why don't you change the learning rate smaller like half or 1/10 of your settings?

[sxyxf66]

@seongkyun You're right. It works. Do you know how to put the dataset in the code? I used other tools to download the dataset. I don’t know how to sent the –voc_root in eval.py and the –dataset_root in the train.py. Would like your reply.

[xlm998]

It is because "batch_iterator" is used up, you should start a new "batch_iterator" as follows:
try:
image, mask, gt = [x.to(device) for x in next(iterator_train)]
except StopIteration:
iterator_train = iter(data.DataLoader(
dataset_train, batch_size=args.batch_size,shuffle=True, num_workers=int(args.n_threads),drop_last=True))
image, mask, gt = [x.to(device) for x in next(iterator_train)]

where should this add to?trian.py or dataloader.py?

[chenjun2hao]

@seongkyun , thanks for your tips.

[yarkable]

@seongkyun Thanks a lot！It saved me .

[qhizershareef]

You wouldn't believe my server3.py file was in the folder datafeed i tried running it directly, it threw 2 errors one of them was discussed above i tried fixing until i tried to open the server3.py file outside the datafeed as python datafeed/server3.py, and phew it ran

[cotyyang]

I fixed this problem.
I've changed the code in train.py
from
images, targets = next(batch_iterator)
to
try:
images, targets = next(batch_iterator)
except StopIteration:
batch_iterator = iter(data_loader)
images, targets = next(batch_iterator)

now, i checked all the losses are falling down :)

thanks. @chenxinyang123

So, this is because of this round of batch_ Iterators run out, and then move on to the next round?

[Jonbroad15]

I fixed this problem.
I've changed the code in train.py
from
images, targets = next(batch_iterator)
to
try:
images, targets = next(batch_iterator)
except StopIteration:
batch_iterator = iter(data_loader)
images, targets = next(batch_iterator)

now, i checked all the losses are falling down :)

thanks. @chenxinyang123

Ive seen this solution in quite a few forums, My issue here is that then aren't you not iterating over your whole dataset and retraining on previous examples?