Ssd.pytorch: Bug in train.py

[bobo0810]

In the 165 or so lines of code:
Images,target = next(batch_iterator)

Bug Description:
The above code can only read the data set once. After reading through the data set, the program will stop.

Solution:
The code is changed to:
  
  # load train data
  Try:
      Images,target = next(batch_iterator)
  Except StopIteration: #Start a new iteration
      Batch_iterator = iter(data_loader)
      Images,target = next(batch_iterator)

[visor2020]

You should reload data. what I mean is that you should copy
data_loader = data.DataLoader(dataset, args.batch_size, num_workers=args.num_workers, shuffle=True, collate_fn=detection_collate, pin_memory=True)
to the except.

[chi0tzp]

As @bobo0810 mentioned, this bug is because the batch iterator run through the whole dataset eventually. So, I guess this means that it happens after exactly one epoch? Do I miss something here?

I just wonder why the developers did not implement an approach like for epoch in range(num_epochs): .... Wouldn't that make better sense?

[bobo0810]

@visor2020 After testing, in fact, do not need to reload the data set。

[visor2020]

@bobo0810 however, the weights is not trained and saved by yourself. Is this a fact?

[shufanwu]

You can see the code in line 122 of voc0712.py, return len(self.ids), the iterator only iterate once, so there are two methods to solve this problem:
1.replacing iterator with the form mentioned by @chi0tzp
2.modifying the function __len__() in voc0712.py with def __len__(self): return self.total_images
the self.total_images is (cfg['max_iter'] - args.start_iter) * args.batch_size

[yingdiLee]

@bobo0810 you should change batch_iterator = iter(data_loader) to batch_iterator = None
and then add this to the beginning of the for loop:
if (not batch_iterator) or (iteration % epoch_size == 0):
# create batch iterator
batch_iterator = iter(data_loader)

[hust-kevin]

@bobo0810 @visor2020 hey！ I have a question in train.py ,why there sometimes use 'ssd_net' sometimes 'net':
ssd_net = build_ssd('train', cfg['min_dim'], cfg['num_classes'])
net = ssd_net

if args.cuda:
    net = torch.nn.DataParallel(ssd_net)
    cudnn.benchmark = True

if args.resume:
    print('Resuming training, loading {}...'.format(args.resume))
    ssd_net.load_weights(args.resume)
else:
    vgg_weights = torch.load(args.save_folder + args.basenet)
    print('Loading base network...')
    ssd_net.vgg.load_state_dict(vgg_weights)  #

if args.cuda:
    net = net.cuda()

if not args.resume:
    print('Initializing weights...')
    # initialize newly added layers' weights with xavier method
    ssd_net.extras.apply(weights_init)
    ssd_net.loc.apply(weights_init)
    ssd_net.conf.apply(weights_init)

optimizer = optim.SGD(net.parameters(), lr=args.lr, momentum=args.momentum,
                      weight_decay=args.weight_decay)
criterion = MultiBoxLoss(cfg['num_classes'], 0.5, True, 0, True, 3, 0.5,
                         False, args.cuda)

I can't understand it,can you help me ? thank you

[bobo0810]

@hust-kevin torch.nn.DataParallel return a new model for multi-gpu

[kuaitoukid]

DataParallel

So why not use "ssd_net = torch.nn.DataParallel(ssd_net)"

[chaomartin]

Comments are quite wonderful and usefull.