Yolov3: labels4.append(labels) UnboundLocalError: local variable 'labels' referenced before assignment

Created on 11 Oct 2019  ยท  17Comments  ยท  Source: ultralytics/yolov3

I have replaced coco dataset with own datasets, which have only one class ('person'). While training, i got the following error.

`
(base) C:Userssamjith.cpDesktopyolov3>python train.py --data coco.data --cfg cfg/yolov3.cfg
Namespace(accumulate=2, adam=False, arc='defaultpw', batch_size=32, bucket='', cache_images=False, cfg='cfg/yolov3.cfg', data='coco.data', device='', epochs=273, evolve=False, img_size=416, img_weights=False, multi_scale=False, name='', nosave=False, notest=False, prebias=False, rect=False, resume=False, transfer=False, var=None, weights='')
Using CPU

WARNING:root:This caffe2 python run does not have GPU support. Will run in CPU only mode.
Reading labels (357 found, 0 missing, 4 empty for 361 images): 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 361/361 [00:00<00:00, 6489.34it/s]
Model Summary: 222 layers, 6.19491e+07 parameters, 6.19491e+07 gradients
Starting training for 273 epochs...

 Epoch   gpu_mem      GIoU       obj       cls     total   targets  img_size

Corrupt JPEG data: 2 extraneous bytes before marker 0xd9
0%| | 0/12 [00:00 Corrupt JPEG data: 1 extraneous bytes before marker 0xd9
Corrupt JPEG data: 1 extraneous bytes before marker 0xd9
Corrupt JPEG data: 2 extraneous bytes before marker 0xd9
Corrupt JPEG data: 1 extraneous bytes before marker 0xd9
Corrupt JPEG data: 2 extraneous bytes before marker 0xd9
Corrupt JPEG data: 1 extraneous bytes before marker 0xd9
Corrupt JPEG data: 1 extraneous bytes before marker 0xd9
Corrupt JPEG data: 1 extraneous bytes before marker 0xd9
Corrupt JPEG data: 2 extraneous bytes before marker 0xd9
Corrupt JPEG data: 2 extraneous bytes before marker 0xd9
Corrupt JPEG data: 1 extraneous bytes before marker 0xd9
Corrupt JPEG data: 1 extraneous bytes before marker 0xd9
Corrupt JPEG data: 2 extraneous bytes before marker 0xd9
Traceback (most recent call last):
File "train.py", line 426, in
train() # train normally
File "train.py", line 235, in train
for i, (imgs, targets, paths, _) in pbar: # batch -------------------------------------------------------------
File "C:Userssamjith.cpAppDataLocalContinuumanaconda3libsite-packagestqdm_tqdm.py", line 1005, in __iter__
for obj in iterable:
File "C:Userssamjith.cpAppDataLocalContinuumanaconda3libsite-packagestorchutilsdatadataloader.py", line 819, in __next__
return self._process_data(data)
File "C:Userssamjith.cpAppDataLocalContinuumanaconda3libsite-packagestorchutilsdatadataloader.py", line 846, in _process_data
data.reraise()
File "C:Userssamjith.cpAppDataLocalContinuumanaconda3libsite-packagestorch_utils.py", line 369, in reraise
raise self.exc_type(msg)
UnboundLocalError: Caught UnboundLocalError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "C:Userssamjith.cpAppDataLocalContinuumanaconda3libsite-packagestorchutilsdata_utilsworker.py", line 178, in _worker_loop
data = fetcher.fetch(index)
File "C:Userssamjith.cpAppDataLocalContinuumanaconda3libsite-packagestorchutilsdata_utilsfetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "C:Userssamjith.cpAppDataLocalContinuumanaconda3libsite-packagestorchutilsdata_utilsfetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "C:Userssamjith.cpDesktopyolov3utilsdatasets.py", line 416, in __getitem__
img, labels = load_mosaic(self, index)
File "C:Userssamjith.cpDesktopyolov3utilsdatasets.py", line 590, in load_mosaic
labels4.append(labels)
UnboundLocalError: local variable 'labels' referenced before assignment`

bug
Source
picture Samjith888

Most helpful comment

Examining the whole label files, finding files of size 0, and deleting the label file and corresponding image. Then, create a txt file containing the image paths again.

for label_path in label_paths:
    if os.stat(label_path).st_size == 0:
        img_path = label_path.replace('labels', 'images').replace('.txt', '.jpg')
        os.remove(label_path)
        os.remove(img_path)

# create a txt file containing the image paths again.
if os.path.exists(txtfile):
    os.remove(txtfile)

with open(txtfile, 'w+') as f:
    for img_name in os.listdir(img_root):
        img_path = os.path.join(img_root, img_name)
        f.write(img_path + '\n')
picture rms0329 on 13 Oct 2019
👍2

All 17 comments

I have the same bug:

 Epoch   gpu_mem      GIoU       obj       cls     total   targets  img_size
 0/272      3.7G      2.09      4.58         0      6.67       132       416:   0%|โ–                                                        | 5/1990 [00:17<2:19:31,  4.22s/it]
 0/272      3.7G      2.06      4.58         0      6.64       120       416:   0%|โ–                                                        | 5/1990 [00:17<2:19:31,  4.22s/it]
 0/272      3.7G      2.06      4.58         0      6.64       120       416:   0%|โ–                                                        | 6/1990 [00:17<1:41:41,  3.08s/it]
 0/272      3.7G      2.05      4.77         0      6.82       171       416:   0%|โ–                                                        | 6/1990 [00:18<1:41:41,  3.08s/it]
 0/272      3.7G      2.05      4.77         0      6.82       171       416:   0%|โ–                                                        | 7/1990 [00:18<1:15:18,  2.28s/it]
 0/272      3.7G      2.04       4.7         0      6.73       109       416:   0%|โ–                                                        | 7/1990 [00:18<1:15:18,  2.28s/it]
 0/272      3.7G      2.04       4.7         0      6.73       109       416:   0%|โ–                                                          | 8/1990 [00:18<56:38,  1.71s/it]
 0/272      3.7G      2.03      4.67         0       6.7       131       416:   0%|โ–                                                          | 8/1990 [00:24<56:38,  1.71s/it]
 0/272      3.7G      2.03      4.67         0       6.7       131       416:   0%|โ–Ž                                                        | 9/1990 [00:24<1:40:54,  3.06s/it]

Traceback (most recent call last):
File "train.py", line 433, in
train() # train normally
File "train.py", line 242, in train
for i, (imgs, targets, paths, _) in pbar: # batch -------------------------------------------------------------
File "E:softpython3.6libsite-packagestqdm_tqdm.py", line 1017, in __iter__
for obj in iterable:
File "E:softpython3.6libsite-packagestorchutilsdatadataloader.py", line 568, in __next__
return self._process_next_batch(batch)
File "E:softpython3.6libsite-packagestorchutilsdatadataloader.py", line 608, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
UnboundLocalError: Traceback (most recent call last):
File "E:softpython3.6libsite-packagestorchutilsdata_utilsworker.py", line 99, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "E:softpython3.6libsite-packagestorchutilsdata_utilsworker.py", line 99, in
samples = collate_fn([dataset[i] for i in batch_indices])
File "Y:trainyolov3-masterutilsdatasets.py", line 416, in __getitem__
img, labels = load_mosaic(self, index)
File "Y:trainyolov3-masterutilsdatasets.py", line 590, in load_mosaic
labels4.append(labels)
UnboundLocalError: local variable 'labels' referenced before assignment

Belinda-great picture Belinda-great on 12 Oct 2019

I only one class

Belinda-great picture Belinda-great on 12 Oct 2019

In my case, it was because the train dataset contained an empty label file.

rms0329 picture rms0329 on 12 Oct 2019
👍2

In my case, it was because the train dataset contained an empty label file.
how can you solve it?

xiaotian3 picture xiaotian3 on 13 Oct 2019

Examining the whole label files, finding files of size 0, and deleting the label file and corresponding image. Then, create a txt file containing the image paths again.

for label_path in label_paths:
    if os.stat(label_path).st_size == 0:
        img_path = label_path.replace('labels', 'images').replace('.txt', '.jpg')
        os.remove(label_path)
        os.remove(img_path)

# create a txt file containing the image paths again.
if os.path.exists(txtfile):
    os.remove(txtfile)

with open(txtfile, 'w+') as f:
    for img_name in os.listdir(img_root):
        img_path = os.path.join(img_root, img_name)
        f.write(img_path + '\n')
rms0329 picture rms0329 on 13 Oct 2019
👍2

@Belinda-great I'm seeing your same error (I have 24 classes instead of 1), but have no empty files, nor mismatch in files between labels & images. Did you have any luck resolving?

(py3) user$ python3 train.py --data data/coco.data --cfg cfg/yolov3.cfg
Namespace(accumulate=2, adam=False, arc='default', batch_size=32, bucket='', cache_images=False, cfg='cfg/yolov3.cfg', data='data/coco.data', device='', epochs=273, evolve=False, img_size=416, img_weights=False, multi_scale=False,
name='', nosave=False, notest=False, prebias=False, rect=False, resume=False, transfer=False, var=None, weights='')
Using CPU

Reading labels (4824 found, 16 missing, 0 empty for 4840 images): 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 4840/4840 [00:01<00:00, 3131.97it
/s]
Model Summary: 222 layers, 6.16476e+07 parameters, 6.16476e+07 gradients
Starting training for 273 epochs...

 Epoch   gpu_mem      GIoU       obj       cls     total   targets  img_size
 0/272        0G      2.04      1.35      10.3      13.7        77       416:   7%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–                                                                                                      | 10/152 [36:21<8:31:24, 216.09s/

it]Traceback (most recent call last):
File "train.py", line 432, in
train() # train normally
File "train.py", line 235, in train
for i, (imgs, targets, paths, _) in pbar: # batch -------------------------------------------------------------
File "//anaconda3/envs/py3/lib/python3.7/site-packages/tqdm/std.py", line 1081, in __iter__
for obj in iterable:
File "//anaconda3/envs/py3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 801, in __next__
return self._process_data(data)
File "//anaconda3/envs/py3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 846, in _process_data
data.reraise()
File "//anaconda3/envs/py3/lib/python3.7/site-packages/torch/_utils.py", line 385, in reraise
raise self.exc_type(msg)
UnboundLocalError: Caught UnboundLocalError in DataLoader worker process 10.
Original Traceback (most recent call last):
File "//anaconda3/envs/py3/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
data = fetcher.fetch(index)
File "//anaconda3/envs/py3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "//anaconda3/envs/py3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/Users/user.../yolov3/utils/datasets.py", line 416, in __getitem__
img, labels = load_mosaic(self, index)
File "/Users/user.../yolov3/utils/datasets.py", line 590, in load_mosaic
labels4.append(labels)
UnboundLocalError: local variable 'labels' referenced before assignment

willsroberts picture willsroberts on 28 Oct 2019

@Belinda-great I'm seeing your same error (I have 24 classes instead of 1), but have no empty files, nor mismatch in files between labels & images. Did you have any luck resolving?

(py3) user$ python3 train.py --data data/coco.data --cfg cfg/yolov3.cfg
Namespace(accumulate=2, adam=False, arc='default', batch_size=32, bucket='', cache_images=False, cfg='cfg/yolov3.cfg', data='data/coco.data', device='', epochs=273, evolve=False, img_size=416, img_weights=False, multi_scale=False,
name='', nosave=False, notest=False, prebias=False, rect=False, resume=False, transfer=False, var=None, weights='')
Using CPU

Reading labels (4824 found, 16 missing, 0 empty for 4840 images): 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 4840/4840 [00:01<00:00, 3131.97it
/s]
Model Summary: 222 layers, 6.16476e+07 parameters, 6.16476e+07 gradients
Starting training for 273 epochs...

 Epoch   gpu_mem      GIoU       obj       cls     total   targets  img_size
 0/272        0G      2.04      1.35      10.3      13.7        77       416:   7%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–                                                                                                      | 10/152 [36:21<8:31:24, 216.09s/

it]Traceback (most recent call last):
File "train.py", line 432, in
train() # train normally
File "train.py", line 235, in train
for i, (imgs, targets, paths, _) in pbar: # batch -------------------------------------------------------------
File "//anaconda3/envs/py3/lib/python3.7/site-packages/tqdm/std.py", line 1081, in iter
for obj in iterable:
File "//anaconda3/envs/py3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 801, in next
return self._process_data(data)
File "//anaconda3/envs/py3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 846, in _process_data
data.reraise()
File "//anaconda3/envs/py3/lib/python3.7/site-packages/torch/_utils.py", line 385, in reraise
raise self.exc_type(msg)
UnboundLocalError: Caught UnboundLocalError in DataLoader worker process 10.
Original Traceback (most recent call last):
File "//anaconda3/envs/py3/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
data = fetcher.fetch(index)
File "//anaconda3/envs/py3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "//anaconda3/envs/py3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/Users/user.../yolov3/utils/datasets.py", line 416, in getitem
img, labels = load_mosaic(self, index)
File "/Users/user.../yolov3/utils/datasets.py", line 590, in load_mosaic
labels4.append(labels)
UnboundLocalError: local variable 'labels' referenced before assignment

I fixed this error after I uncommented a line in datasets.py, (line 330) to discover the problematic files and deleted them. I had sufficient samples that this wasn't a problem, but for some datasets that might not be the case. I also could not deduce from close inspection what the difference was between these labels and other labels. Ideally, if this type of data error will prevent the model training, it should exit prior to starting the process.
Note: after deleting your images and label files, also update your list of files you point to in your version of coco.data
nm += 1 print('missing labels for image %s' % self.img_files[i]) # file missing continue

willsroberts picture willsroberts on 4 Nov 2019

@glenn-jocher, I just add a tab in line, maybe fix๏ผŸ
https://github.com/ultralytics/yolov3/blob/09ca721f888d55d7147599d09bd091c021470162/utils/datasets.py#L590

    if os.path.isfile(label_path):
        x = self.labels[index]
        if x is None:  # labels not preloaded
            with open(label_path, 'r') as f:
                x = np.array([x.split() for x in f.read().splitlines()], dtype=np.float32)

        if x.size > 0:
            # Normalized xywh to pixel xyxy format
            labels = x.copy()
            labels[:, 1] = w * (x[:, 1] - x[:, 3] / 2) + padw
            labels[:, 2] = h * (x[:, 2] - x[:, 4] / 2) + padh
            labels[:, 3] = w * (x[:, 1] + x[:, 3] / 2) + padw
            labels[:, 4] = h * (x[:, 2] + x[:, 4] / 2) + padh

            labels4.append(labels) # add a tab to fix issue #548
if len(labels4):
    labels4 = np.concatenate(labels4, 0)
mozpp picture mozpp on 7 Nov 2019

@mozpp @willsroberts @rms0329 @xiaotian3 @Belinda-great @Samjith888 the latest commit should fix this: https://github.com/ultralytics/yolov3/commit/aae39ca894300cea594b47f426020ad41b683371

The error was caused because some images in your custom dataset lacked labels, so this predefines an empty labels array for all images, which is replaced by actual labels if they are present. Can you git pull and try again?

glenn-jocher picture glenn-jocher on 7 Nov 2019

@glenn-jocher previously had the same error as those above. After pulling (so that I'm to date as of this post) and trying again, I run into the following error: 

File "/oak/stanford/groups/deho/benny/cafo/yolov3/utils/datasets.py", line 593, in load_mosaic
    labels4 = np.concatenate(labels4, 0)
  File "<__array_function__ internals>", line 6, in concatenate
ValueError: all the input arrays must have same number of dimensions, but the array at index 0 has 1 dimension(s) and the array at index 3 has 2 dimension(s)

I'm assuming this had something to do with the fix; the code runs fine if I remove the empty label files.

bchugg picture bchugg on 12 Nov 2019

@bchugg thanks for the feedback. This means it's trying to stack the np.arrays with the [] repopulated arrays. We want to make these 0xn np arrays, so I've updated the code to this now. Can you try again? Thanks!

https://github.com/ultralytics/yolov3/blob/470ef6bc9271637e1b49137d32dbe4698e9ab478/utils/datasets.py#L582-L592

glenn-jocher picture glenn-jocher on 12 Nov 2019

@glenn-jocher all good on my end now! Cheers :).

bchugg picture bchugg on 13 Nov 2019

Great! I'll close this issue for now as the original issue appears to have been resolved, and/or no activity has been seen for some time. Feel free to comment if this is not the case.

glenn-jocher picture glenn-jocher on 13 Nov 2019

yolov5/utils/general.py", line 75, in check_git_status
print(s)
UnboundLocalError: local variable 's' referenced before assignment

henbucuoshanghai picture henbucuoshanghai on 13 Jan 2021

the train set must not have a image that has no object in it?????

henbucuoshanghai picture henbucuoshanghai on 13 Jan 2021

@henbucuoshanghai thanks for the bug report. This is related to a recent PR https://github.com/ultralytics/yolov5/pull/1916 (unrelated to your dataset). I will take a look.

glenn-jocher picture glenn-jocher on 13 Jan 2021

@henbucuoshanghai we've identified the problem and created and merged a bug fix PR #1660 for this. Please git pull to receive this update and let us know if you spot any other issues!

Thank you for your contributions!

glenn-jocher picture glenn-jocher on 13 Jan 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

suarezjessie picture suarezjessie  ยท  5Comments
cyberclone12 picture cyberclone12  ยท  4Comments
leeyunhome picture leeyunhome  ยท  3Comments
CF2220160244 picture CF2220160244  ยท  5Comments
NgTuong picture NgTuong  ยท  4Comments