Yolov5: KeyError occur when start training

[Zzh-tju]

❔Question

@glenn-jocher
Currently, I work on a face detection. I use the following command to train.
python train.py --img 640 --batch 16 --epochs 5 --data ./data/face.yaml --cfg ./models/yolov5s.yaml --weights yolov5s.pt
All the training datasets are

../face/images/train/6000.jpg
../face/images/train/6001.jpg
../face/images/train/6002.jpg
......

And their coresponding labels are

../face/labels/train/6000.txt
../face/labels/train/6001.txt
../face/labels/train/6002.txt
......

But I have an error:

WARNING: /media/zzh/face/images/train/9994.jpg: setting an array element with a sequence.
WARNING: /media/zzh/face/images/train/9995.jpg: setting an array element with a sequence.
WARNING: /media/zzh/face/images/train/9996.jpg: setting an array element with a sequence.
WARNING: /media/zzh/face/images/train/9997.jpg: setting an array element with a sequence.
WARNING: /media/zzh/face/images/train/9998.jpg: setting an array element with a sequence.
WARNING: /media/zzh/face/images/train/9999.jpg: setting an array element with a sequence.
Scanning images: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6120/6120 [00:01<00:00, 4498.79it/s]
Traceback (most recent call last):
  File "train.py", line 456, in <module>
    train(hyp, opt, device, tb_writer)
  File "train.py", line 169, in train
    world_size=opt.world_size, workers=opt.workers)
  File "/media/zzh/yolov5/utils/datasets.py", line 61, in create_dataloader
    rank=rank)
  File "/media/zzh/yolov5/utils/datasets.py", line 380, in __init__
    labels, shapes = zip(*[cache[x] for x in self.label_files])
  File "/media/zzh/yolov5/utils/datasets.py", line 380, in <listcomp>
    labels, shapes = zip(*[cache[x] for x in self.label_files])
KeyError: '/media/zzh/face/labels/train/10000.txt'

And face/labels/train/10000.txt is
0 0.6062500000000001 0.6017543859649123 0.3775 0.5719298245614035

I don't know how can I solve this problem.

[github-actions[bot]]

Hello @Zzh-tju, thank you for your interest in our work! Please visit our Custom Training Tutorial to get started, and see our Jupyter Notebook , Docker Image, and Google Cloud Quickstart Guide for example environments.

If this is a bug report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom model or data training question, please note Ultralytics does not provide free personal support. As a leader in vision ML and AI, we do offer professional consulting, from simple expert advice up to delivery of fully customized, end-to-end production solutions for our clients, such as:

• Cloud-based AI systems operating on hundreds of HD video streams in realtime.
• Edge AI integrated into custom iOS and Android apps for realtime 30 FPS video inference.
• Custom data training, hyperparameter evolution, and model exportation to any destination.

For more information please visit https://www.ultralytics.com.

[Zzh-tju]

When I changed labels, shapes = zip(*[cache[x] for x in self.label_files])
back to labels, shapes = zip(*[cache[x] for x in self.img_files]) in utils/datasets.py
Error changed too:

20                -1  1    313088  models.common.BottleneckCSP             [256, 256, 1, False]
 21                -1  1    590336  models.common.Conv                      [256, 256, 3, 2]
 22          [-1, 10]  1         0  models.common.Concat                    [1]
 23                -1  1   1248768  models.common.BottleneckCSP             [512, 512, 1, False]
 24      [17, 20, 23]  1     18879  models.yolo.Detect                      [2, [[10, 13, 16, 30, 33, 23], [30, 61, 62, 45, 59, 119], [116, 90, 156, 198, 373, 326]], [128, 256, 512]]
Model Summary: 191 layers, 7.25779e+06 parameters, 7.25779e+06 gradients

Transferred 362/370 items from yolov5s.pt
Optimizer groups: 62 .bias, 70 conv.weight, 59 other
TypeError: float() argument must be a string or a number, not 'tuple'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "train.py", line 456, in <module>
    train(hyp, opt, device, tb_writer)
  File "train.py", line 169, in train
    world_size=opt.world_size, workers=opt.workers)
  File "/media/zzh/yolov5/utils/datasets.py", line 61, in create_dataloader
    rank=rank)
  File "/media/zzh/yolov5/utils/datasets.py", line 383, in __init__
    self.shapes = np.array(shapes, dtype=np.float64)
ValueError: setting an array element with a sequence.

[karen-gishyan]

Same issue @Zzh-tju, but the issue is new, had no problems until yesterday.

[glenn-jocher]

@Zzh-tju @karen-gishyan
Hello, thank you for your interest in our work! This issue seems to lack the minimum requirements for a proper response, or is insufficiently detailed for us to help you. Please note that most technical problems are due to:

• Your changes to the default repository. If your issue is not reproducible in a new git clone version of this repository we can not debug it. Before going further run this code and ensure your issue persists:

sudo rm -rf yolov5  # remove existing
git clone https://github.com/ultralytics/yolov5 && cd yolov5 # clone latest
python detect.py  # verify detection
# CODE TO REPRODUCE YOUR ISSUE HERE

• Your custom data. If your issue is not reproducible with COCO or COCO128 data we can not debug it. Visit our Custom Training Tutorial for guidelines on training your custom data. Examine train_batch0.jpg and test_batch0.jpg for a sanity check of training and testing data.

• Your environment. If your issue is not reproducible in one of the verified environments below we can not debug it. If you are running YOLOv5 locally, ensure your environment meets all of the requirements.txt dependencies specified below.

If none of these apply to you, we suggest you close this issue and raise a new one using the Bug Report template, providing screenshots and minimum viable code to reproduce your issue. Thank you!

Requirements

Python 3.8 or later with all requirements.txt dependencies installed, including torch>=1.6. To install run:

$ pip install -r requirements.txt

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Google Colab Notebook with free GPU:
• Kaggle Notebook with free GPU: https://www.kaggle.com/ultralytics/yolov5
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Docker Image https://hub.docker.com/r/ultralytics/yolov5. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are passing. These tests evaluate proper operation of basic YOLOv5 functionality, including training (train.py), testing (test.py), inference (detect.py) and export (export.py) on MacOS, Windows, and Ubuntu.

[karen-gishyan]

Hello @glenn-jocher , and thanks for your reply. I could see that the issue was with the way the labels were being read. I looked at your commit history in utils/datasets.py, and went back to your previous version, which solved the problem.
self.label_files = [x.replace('images', 'labels').replace(os.path.splitext(x)[-1], '.txt') for x in self.img_files]
I think the new change may certainly be the source of the issue. Thanks.

[sophiatmu]

@karen-gishyan I got the same problem as you, but it came out with "WARNING: /home/TrafficLight/JPEGImages/10141_0_1.jpg: image size <10 pixels" before, and which previous version were you use? thanks for your reply.

[lolpa1n]

@karen-gishyan same problem, how do u solve? fixes in datasets.py dont solved the problem

Optimizer groups: 86 .bias, 94 conv.weight, 83 other
TypeError: float() argument must be a string or a number, not 'tuple'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "train.py", line 456, in <module>
    train(hyp, opt, device, tb_writer)
  File "train.py", line 169, in train
    rank=rank, world_size=opt.world_size, workers=opt.workers)
  File "/content/yolov5/utils/datasets.py", line 61, in create_dataloader
    rank=rank)
  File "/content/yolov5/utils/datasets.py", line 379, in __init__
    self.shapes = np.array(shapes, dtype=np.float64)
ValueError: setting an array element with a sequence.

[karen-gishyan]

@sophiatmu I know this a temporary solution until the authors take a look at it, but in the utils/datasets.py, changed the code in lines 366,3
67 to the following code, which is the previous commit, and the model worked again.

        self.label_files = [x.replace('images', 'labels').replace(os.path.splitext(x)[-1], '.txt') for x in
                            self.img_files]

[glenn-jocher]

@Zzh-tju @karen-gishyan @lolpa1n @sophiatmu I've pushed a fix which should restore similar functionality to before.
https://github.com/ultralytics/yolov5/blob/806e75f2b1166a4a789e0ea70b0e48064005f5c9/utils/datasets.py#L365-L368

Note that label paths are defined as the image paths with a .replace() statement that will replace the last instance of /images/ with /labels/ in your image paths.

[glenn-jocher]

The dataset structure example provided by @Zzh-tju should work with no issues:

../face/images/train/6000.jpg
../face/images/train/6001.jpg
../face/images/train/6002.jpg
......

And their coresponding labels are

../face/labels/train/6000.txt
../face/labels/train/6001.txt
../face/labels/train/6002.txt
......

CI tests on fix https://github.com/ultralytics/yolov5/commit/806e75f2b1166a4a789e0ea70b0e48064005f5c9 are all green.
https://github.com/ultralytics/yolov5/actions/runs/252688252

[Zzh-tju]

OK, fixed.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[21143]

Hi. I am facing the same issue when training on my custom dataset in ubuntu 18.04. However, this issue does not come up with coco128 dataset.
On windows 10, I do not face this issue at all with either my custom dataset or coco128. Any thoughts on why this could be happening and where I should be looking to fix this?

[glenn-jocher]

@21143 it appears you may have environment problems. Please ensure you meet all dependency requirements if you are attempting to run YOLOv5 locally. If in doubt, create a new virtual Python 3.8 environment, clone the latest repo (code changes daily), and pip install -r requirements.txt again. We also highly recommend using one of our verified environments below.

Requirements

Python 3.8 or later with all requirements.txt dependencies installed, including torch>=1.6. To install run:

$ pip install -r requirements.txt

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Google Colab Notebook with free GPU:
• Kaggle Notebook with free GPU: https://www.kaggle.com/ultralytics/yolov5
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Docker Image https://hub.docker.com/r/ultralytics/yolov5. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are passing. These tests evaluate proper operation of basic YOLOv5 functionality, including training (train.py), testing (test.py), inference (detect.py) and export (export.py) on MacOS, Windows, and Ubuntu.

[21143]

Update: Fixed my issue by deleting the train.cache and val.cache files in the labels folder and re-running the training. I'm able to run training code now. Thanks !