Yolov5: Class-Incremental Training for saving training time

[rose-jinyang]

❔Question

Hello
How are you?
Thanks for contributing this project.
I have a question.
Does this project support class-incremental training for saving training time?
Thanks

Additional context

[github-actions[bot]]

Hello @rose-jinyang, thank you for your interest in our work! Please visit our Custom Training Tutorial to get started, and see our Jupyter Notebook , Docker Image, and Google Cloud Quickstart Guide for example environments.

If this is a bug report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom model or data training question, please note that Ultralytics does not provide free personal support. As a leader in vision ML and AI, we do offer professional consulting, from simple expert advice up to delivery of fully customized, end-to-end production solutions for our clients, such as:

• Cloud-based AI systems operating on hundreds of HD video streams in realtime.
• Edge AI integrated into custom iOS and Android apps for realtime 30 FPS video inference.
• Custom data training, hyperparameter evolution, and model exportation to any destination.

For more information please visit https://www.ultralytics.com.

[glenn-jocher]

@rose-jinyang sorry, I don't know what class incremental training is. If you'd like to add a feature we are open to PRs!

[rose-jinyang]

Hi
Thanks for your reply.
Let me explain in detail.
Let's suppose I've trained a model on the dataset with K classes for 7 days.
If a new class is added into this dataset, should we train a model with the expanded dataset (K+1 classes) from begin?
If so, it is so expensive, especially in case of object detection in retail store.
That's because a new class of good is added frequently in retail store.
Can we train a new model in short time with the original weight on the expanded dataset?
I think that this is very important function.
Please let me know if you have a willing to implement this.
Thanks

[glenn-jocher]

@rose-jinyang ah I see. Yes this is supported here, you don't need anything extra. Here are the steps:

1. Train a model path/best.pt
2. Run from utils.utils import *; create_pretrained(path/best.pt) will create an official yolov5 model suitable for use as pretrained weights in the future on any arbitrary new dataset, regardless of class count.

These are the same steps I use to create the official yolov5 models, and they are capable of training any new dataset using the basic train command:
python train.py --weights yolov5s.pt

[glenn-jocher]

@rose-jinyang was just thinking, perhaps I should integrate step 2 to run automatically at the end of training after the final epoch.

TODO: consider unifying strip_optimizer() and create_pretrained() to formally finalize training after completion of final epoch.

[glenn-jocher]

@rose-jinyang TODO is completed with PR https://github.com/ultralytics/yolov5/pull/509. Now last.pt and best.pt are automatically finalized as pretrained weights upon training completion (the same as the official weights), and can be used as starting points to train any dataset with any number of classes. Removing TODO and closing issue. Let me know if you have any further problems.

[rose-jinyang]

Hi
Thanks for your efforts.
I have a question.
Really, does this support the class-incremental object detection?
Can be the training time saved?
if so, does not the knowledge forgetting occur?
The following papers treated such problems in the incremental object detection.
_https://arxiv.org/abs/1708.06977
https://arxiv.org/abs/2003.04668
https://arxiv.org/abs/2003.08798_
Please let me know which paper/idea was implemented in this project?
Thanks

[glenn-jocher]

@rose-jinyang yes, you can add classes incrementally with no changes or special settings. You simply start training from your previously trained weights: For example to add an extra class on top of a 40 class trained model you update your datal.yaml and then run:

python train.py --weights last.pt --data.yaml

[dulre]

@rose-jinyang yes, you can add classes incrementally with no changes or special settings. You simply start training from your previously trained weights: For example to add an extra class on top of a 40 class trained model you update your datal.yaml and then run:

python train.py --weights last.pt --data.yaml

Yes but even without providing the previous images of classes on which the previous model was trained?
For example i want to extend yolov5s with 2 new classes but i do not want to provide all the old data of the old classes but just of the 2 new ones so the training is faster. Is this possible without forgetting the knowledge about the previous classes?

[Ownmarc]

No its not possible, maybe in the future of deep learning we will be able to concat models together ahah. Your best bet would be to do as Glenn said or even better, add your new images and do a full training again for best results.

[dulre]

No its not possible, maybe in the future of deep learning we will be able to concat models together ahah. Your best bet would be to do as Glenn said or even better, add your new images and do a full training again for best results.

Yeah i see that it is impossible indeed.

But what if i provide images for one of the classes i want the model to maintain previous information about?i I want to add to yolov5s 2 new classes painting and bust but i am interested in another class person for which the pretrained weights works really well. For class person i can provide some data obviously i can't compete with the dataset on which the pretrained weights were trained. Is there a solution for this problem?

Recently i have tried training by using a small dataset of 5000k images for classes person, painting and bust using the pretrained weights as a start but at the end of training persons are non detected very well so i guess the model didn't use any information about persons on which it was trained previously and was performing so well.

What i should do?

[Ownmarc]

The best results will be achieved by building your own dataset that will be as similar as possible to the images you will be doing inference on. I would also advise to try, as much as possible, to balance the classes. It is not required and it is handled pretty well in Yolov5, but still a best practice.

You could go out and gather a dataset of peoples that is as similar as possible to the people you will want to detect and add those images to your painting/brush dataset. Make sure you label any painting/brush in the people-images you pick or else it could downmine your results for other classes and do the same for people in your painting/brush images.

The closer your dataset will be to the images you will predict/inference on, the best it will be. You probably do not need 5k images of people if you choose them wisely!

Hope this helps!

[dulre]

The best results will be achieved by building your own dataset that will be as similar as possible to the images you will be doing inference on. I would also advise to try, as much as possible, to balance the classes. It is not required and it is handled pretty well in Yolov5, but still a best practice.

You could go out and gather a dataset of peoples that is as similar as possible to the people you will want to detect and add those images to your painting/brush dataset. Make sure you label any painting/brush in the people-images you pick or else it could downmine your results for other classes and do the same for people in your painting/brush images.

The closer your dataset will be to the images you will predict/inference on, the best it will be. You probably do not need 5k images of people if you choose them wisely!

Hope this helps!

Ok i'll try adding and manually labeling some images of people watching paintings or people in museums in general but i really hoped that i could use the knowledge about people gained by the pretrained weights.

[Ownmarc]

You could use the Coco model (that has the person category you want) to label your images for you and simply "correct" the labels if needed and then train your own model! This is called active learning and would lower the effort required to label your dataset!

[rose-jinyang]

Hi
I think that it is happy if the training time can be saved although the original training dataset is reused.
The main problem is to save the training time without forgetting knowledge as possible.

[Ownmarc]

From my experience with this repo, retraining from scratch (from the coco weights) has always given better results when wanting to add a class, add images to your dataset or even after you modify/correct some annotations.

[rose-jinyang]

Hi
I think that it is happy if the training time can be saved although the original training dataset is reused.
The main problem is to save the training time without forgetting knowledge as possible.

[rose-jinyang]

Hi
Thanks for your reply.
Let me explain in detail.
Let's suppose I've trained a model on the dataset with K classes for 7 days.
If a new class is added into this dataset, should we train a model with the expanded dataset (K+1 classes) from begin?
If so, it is so expensive, especially in case of object detection in retail store.
That's because a new class of good is added frequently in retail store.
Can we train a new model in short time with the original weight on the expanded dataset?
I think that this is very important function.
Thanks

[glenn-jocher]

@rose-jinyang the optimizer will modify all available weights and biases in a model to optimally reduce the losses between the output and and what you want it to output. Every neuron is connected to every output, therefore it is impossible to keep the model constant while also changing it. This is like spending your money while also keeping it in your bank account.

I don't see your situation as any different than the default use case: fine-tuning on a new dataset. Your dataset must naturally contain the objects you want to train on.

[rose-jinyang]

Hi
Do u mean that it is impossible to implement this?

[glenn-jocher]

@rose-jinyang I'm saying you should just follow the tutorials for what you need:
https://github.com/ultralytics/yolov5#tutorials