Darknet: Unable to understand how to train classifier with our own dataset?

[LolikaPadmanbhan]

Hey,
I'm trying to train a classifier model in Darknet with my own dataset, consists of 4 class data. Though i found a cifar classification example in darknet, but i am unable to find in what format i should provide my data (renaming is required i guess, with classname at end?). or just tell me how should i proceed farther to use my dataset for a classifier specifying the classes that i want.
I have followed this https://pjreddie.com/darknet/train-cifar/ link but still not clear how the classifier finds the class information.
Please help me in this regards. Thanks in advance.

[ahsan856jalal]

Your filenames should be like 1_class_name.png
where 1 in the above name could be any integer and class_name should be one of your class name such as dog, car etc. (remember, it should be one of the name from labels.list)
Then make a list 'train.list' containing all paths of the image files use the find ~pwd/... >train.list command from the train_cifar tutorial , if you have your image files in one folder.
Then make a labels.list file having all class_names
then modify cifar.data according to your dataset as well as the number of classes at the end of the cfg file .
Then run:
./darknet classifier train your_data.data your_cfg.cfg

[glowcone]

What should I name the image if it has no classes (a background image)?

[LolikaPadmanbhan]

@ahsan856jalal
Hey,
Thank you so much for the detailed explanation.
However i tried making all the changes but when i execute it gives me null in the output. thats due to classes index. looks like class index of cifar is made as 10 and even when we are changing it in cifar.data the class number its still not getting updated. need to knw where else i should change.
and one more thing there is no darknet.c in src file or classifier.c in src then from where it is happing? is it through example folder am totally confused.
the output is attached for your inforamtion below.
Help me out..
Thanks.

/Darket/darknet$ ./darknet classifier predict cfg/cifar.data cfg/cifar_small.cfg backup/cifar_small.weights data/ycym2.jpg
layer filters size input output
0 conv 32 3 x 3 / 1 28 x 28 x 3 -> 28 x 28 x 32 0.001 BFLOPs
1 max 2 x 2 / 2 28 x 28 x 32 -> 14 x 14 x 32
2 conv 16 1 x 1 / 1 14 x 14 x 32 -> 14 x 14 x 16 0.000 BFLOPs
3 conv 64 3 x 3 / 1 14 x 14 x 16 -> 14 x 14 x 64 0.004 BFLOPs
4 max 2 x 2 / 2 14 x 14 x 64 -> 7 x 7 x 64
5 conv 32 1 x 1 / 1 7 x 7 x 64 -> 7 x 7 x 32 0.000 BFLOPs
6 conv 128 3 x 3 / 1 7 x 7 x 32 -> 7 x 7 x 128 0.004 BFLOPs
7 conv 64 1 x 1 / 1 7 x 7 x 128 -> 7 x 7 x 64 0.001 BFLOPs
8 conv 10 1 x 1 / 1 7 x 7 x 64 -> 7 x 7 x 10 0.000 BFLOPs
9 avg 7 x 7 x 10 -> 10
10 softmax 10
Loading weights from backup/cifar_small.weights...Done!
data/ycym2.jpg: Predicted in 0.000695 seconds.
11.05%: (null)
10.81%: (null)

[ahsan856jalal]

1: what do have in cifar.data? what is in the labels.list file which you
have linked in cifar.data
2: yes detector and classifier 'c' files are in the examples folder.
Regards
Ahsan

On Fri, Jul 13, 2018 at 10:54 AM Lolika Padmanabhan <
[email protected]> wrote:

@ahsan856jalal https://github.com/ahsan856jalal
Hey,
Thank you so much for the detailed explanation.
However i tried making all the changes but when i execute it gives me null
in the output. thats due to classes index. looks like class index of cifar
is made as 10 and even when we are changing it in cifar.data the class
number its still not getting updated. need to knw where else i should
change.
and one more thing there is no darknet.c in src file or classifier.c in
src then from where it is happing? is it through example folder am totally
confused.
the output is attached for your inforamtion below.
Help me out..
Thanks.

/Darket/darknet$ ./darknet classifier predict cfg/cifar.data
cfg/cifar_small.cfg backup/cifar_small.weights data/ycym2.jpg
layer filters size input output
0 conv 32 3 x 3 / 1 28 x 28 x 3 -> 28 x 28 x 32 0.001 BFLOPs
1 max 2 x 2 / 2 28 x 28 x 32 -> 14 x 14 x 32
2 conv 16 1 x 1 / 1 14 x 14 x 32 -> 14 x 14 x 16 0.000 BFLOPs
3 conv 64 3 x 3 / 1 14 x 14 x 16 -> 14 x 14 x 64 0.004 BFLOPs
4 max 2 x 2 / 2 14 x 14 x 64 -> 7 x 7 x 64
5 conv 32 1 x 1 / 1 7 x 7 x 64 -> 7 x 7 x 32 0.000 BFLOPs
6 conv 128 3 x 3 / 1 7 x 7 x 32 -> 7 x 7 x 128 0.004 BFLOPs
7 conv 64 1 x 1 / 1 7 x 7 x 128 -> 7 x 7 x 64 0.001 BFLOPs
8 conv 10 1 x 1 / 1 7 x 7 x 64 -> 7 x 7 x 10 0.000 BFLOPs
9 avg 7 x 7 x 10 -> 10
10 softmax 10
Loading weights from backup/cifar_small.weights...Done!
data/ycym2.jpg: Predicted in 0.000695 seconds.
11.05%: (null)
10.81%: (null)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pjreddie/darknet/issues/962#issuecomment-404733406,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AK9zjj5YncJR5WG7V9rM-bLCiI4EWr-tks5uGDYSgaJpZM4VKm4t
.

[ahsan856jalal]

Moreover, did you train cifar_small by yourself. If yes, did you make sure
the naming scheme of your dataset as I told before ? e.g 1_class1name.png,
1_class2name.png,....
And labels.list contains
class1name
class2name
.
.
.

Regards
Ahsan

On Fri, Jul 13, 2018 at 11:32 AM Ahsan Jalal [e-mail]
wrote:

1: what do have in cifar.data? what is in the labels.list file which you
have linked in cifar.data
2: yes detector and classifier 'c' files are in the examples folder.
Regards
Ahsan

On Fri, Jul 13, 2018 at 10:54 AM Lolika Padmanabhan <
[email protected]> wrote:

@ahsan856jalal https://github.com/ahsan856jalal
Hey,
Thank you so much for the detailed explanation.
However i tried making all the changes but when i execute it gives me
null in the output. thats due to classes index. looks like class index of
cifar is made as 10 and even when we are changing it in cifar.data the
class number its still not getting updated. need to knw where else i should
change.
and one more thing there is no darknet.c in src file or classifier.c in
src then from where it is happing? is it through example folder am totally
confused.
the output is attached for your inforamtion below.
Help me out..
Thanks.

/Darket/darknet$ ./darknet classifier predict cfg/cifar.data
cfg/cifar_small.cfg backup/cifar_small.weights data/ycym2.jpg
layer filters size input output
0 conv 32 3 x 3 / 1 28 x 28 x 3 -> 28 x 28 x 32 0.001 BFLOPs
1 max 2 x 2 / 2 28 x 28 x 32 -> 14 x 14 x 32
2 conv 16 1 x 1 / 1 14 x 14 x 32 -> 14 x 14 x 16 0.000 BFLOPs
3 conv 64 3 x 3 / 1 14 x 14 x 16 -> 14 x 14 x 64 0.004 BFLOPs
4 max 2 x 2 / 2 14 x 14 x 64 -> 7 x 7 x 64
5 conv 32 1 x 1 / 1 7 x 7 x 64 -> 7 x 7 x 32 0.000 BFLOPs
6 conv 128 3 x 3 / 1 7 x 7 x 32 -> 7 x 7 x 128 0.004 BFLOPs
7 conv 64 1 x 1 / 1 7 x 7 x 128 -> 7 x 7 x 64 0.001 BFLOPs
8 conv 10 1 x 1 / 1 7 x 7 x 64 -> 7 x 7 x 10 0.000 BFLOPs
9 avg 7 x 7 x 10 -> 10
10 softmax 10
Loading weights from backup/cifar_small.weights...Done!
data/ycym2.jpg: Predicted in 0.000695 seconds.
11.05%: (null)
10.81%: (null)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pjreddie/darknet/issues/962#issuecomment-404733406,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AK9zjj5YncJR5WG7V9rM-bLCiI4EWr-tks5uGDYSgaJpZM4VKm4t
.

[LolikaPadmanbhan]

@ahsan856jalal
this is what i have in cifar.data.
classes=4
train = data/cifar/train.list
valid = data/cifar/test.list
labels = data/cifar/labels.txt
backup = backup/
top=2
labels.txt
capnomaskno
capnomaskyes
capyesmaskno
capyesmaskyes
cifar is in cfg folder and this labels.txt in cifar folder along with test.list and train.list

Ya the naming scheme is u have explained.
thanks

[LolikaPadmanbhan]

@ahsan856jalal
if am creating different folder and making this train, test list and labels .list, then its not able to read the values from the /data/mydb/labels.list.
this gives me error saying unable to find labels.txt..

[ahsan856jalal]

You are doing it right. This is strange that you are getting this sort of
error . Can you re-clone the directory in a new folder just to test this
code
git clone https://github.com/pjreddie/darknet darknet_test1
then compile and copy .data,cfg and weight file in the main dir and call
the function and see. To me, it should run fine now
Regards
Ahsan

On Fri, Jul 13, 2018 at 3:13 PM Lolika Padmanabhan [e-mail]
wrote:

@ahsan856jalal https://github.com/ahsan856jalal
if am creating different folder and making this train, test list and
labels .list, then its not able to read the values from the
/data/mydb/labels.list.
this gives me error saying unable to find labels.txt..

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pjreddie/darknet/issues/962#issuecomment-404790403,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AK9zju5LQ_htgTTex0FvmdZd_m6KMu4wks5uGHK3gaJpZM4VKm4t
.

[LolikaPadmanbhan]

@ahsan856jalal
Thanks for it. but am still getting the same. please help me.

[ahsan856jalal]

I will write the steps again , may be you forgot to do anyone of them
1: change the filtes to 4 at the end of cifar_small.cfg file instead of 10 as your classes are 10.
2: make a new labels list e.g test1.list and the contents will the your class names ( try to have one word label for each class , may be space is adding problems)
3: Copy all image files in one directory like cifar10 train folder and name them in the same manner as train folder of cifar 10 i.e 1_dog.png, 2_cat.png, 1_car.png, ....
4: Make a list train_img.list and copy absolute paths of these training images ( e.g /home/ahsan/custom_data/train/1_cat.png )
5: Your data file will looks like
classes=4
train = data/train.list
valid = data/test.list
labels = data/labels.list
backup = backup
top=2
6: Now do training
./darknet classifier train data/your_data.data cfg/your_cfg.cfg

[LolikaPadmanbhan]

@ahsan856jalal
yaa thank u so much i had missed changing the filter to 4 thats y may be. now kept for training hoping to see positive results.
and one more thing, i don't want that weights to get loaded after every 3 or 4 iterations. so, where will change it .
Thanks man thank you so much ur really super.. :-)

[LolikaPadmanbhan]

@ahsan856jalal
Hey,
I was able to run this successfully on my own dataset on my laptop with gpu and was very happy about it. but i have come across another problem. please help me.
The same project am trying to run on my Jetson TX2 board there is a conflict.
In the make fail i have made CuDNN as 0 because classifier was not working with CuDNN = 1,where as the Yolo object detector works only with CuDNN being 1. so When i integrate these two am unable to run this together and get output on the board.
However both are working fine as stand alone applications.
Kindly help me

[ahsan856jalal]

For me, Desktop PC with Nvidia 1080 Ti , classifier and detector both are
working with CudNN=1.
I am not sure why it is not behaving in a similar fashion on Tx2. Once I
used Tx2 with cudnn=1, but that time, I only did detection.
Regards
Ahsan

On Wed, Jul 18, 2018 at 9:13 PM Lolika Padmanabhan [e-mail]
wrote:

@ahsan856jalal https://github.com/ahsan856jalal
Hey,
I was able to run this successfully on my own dataset on my laptop with
gpu and was very happy about it. but i have come across another problem.
please help me.
The same project am trying to run on my Jetson TX2 board there is a
conflict.
In the make fail i have made CuDNN as 0 because classifier was not
working with CuDNN = 1,where as the Yolo object detector works only with
CuDNN being 1. so When i integrate these two am unable to run this together
and get output on the board.
However both are working fine as stand alone applications.
Kindly help me

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pjreddie/darknet/issues/962#issuecomment-405988207,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AK9zjsh5996SPFXnQj4F4YuEEjQGK7J-ks5uH16hgaJpZM4VKm4t
.

[LolikaPadmanbhan]

@ahsan856jalal
ooh is it .. which version of cuda and cudnn u had in the machine?.
and also is it possible to keep two different .so files for a project. if yes please tell me how?
thanks...

[LolikaPadmanbhan]

@ahsan856jalal
if u have time i can mail u my config and data file to ur mail if it is ok ?, so that u can just help me to run it on my TX2 that would be great help. what am asking for is like too much only, but am stuck with this from 2 days and help less not getting any help o this so.
Thanks in advance.

[ahsan856jalal]

Actually I am not working on Tx2 right now . So it will take sometime to
let you know the exact cuda and cudnn versions.
For having two different .so files , I believe either make one .so file
with certain parameters and then rename it and use accordingly .
Or make two separate darknet directories, make one with one set of
parameters and other with other set of parameters.
Regards
Ahsan

On Thu, Jul 19, 2018 at 11:01 AM Lolika Padmanabhan <
[email protected]> wrote:

@ahsan856jalal https://github.com/ahsan856jalal
if u have time i can mail u my config and data file to ur mail if it is ok
?, so that u can just help me to run it on my TX2 that would be great help.
what am asking for is like too much only, but am stuck with this from 2
days and help less not getting any help o this so.
Thanks in advance.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pjreddie/darknet/issues/962#issuecomment-406164163,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AK9zjn1QBrrDyVugBBW-hKjwVn4nXI2rks5uICDTgaJpZM4VKm4t
.

[LolikaPadmanbhan]

@ahsan856jalal
i had thought of it but there are some 50 + files which are common to both classifier and detection so i'm just thinking there should not cause any name conflict. so.
thanks

[rachana88]

@ahsan856jalal
this is what i have in cifar.data.
classes=4
train = data/cifar/train.list
valid = data/cifar/test.list
labels = data/cifar/labels.txt
backup = backup/
top=2
labels.txt
capnomaskno
capnomaskyes
capyesmaskno
capyesmaskyes
cifar is in cfg folder and this labels.txt in cifar folder along with test.list and train.list

Ya the naming scheme is u have explained.
thanks

@ahsan856jalal
this is what i have in cifar.data.
classes=4
train = data/cifar/train.list
valid = data/cifar/test.list
labels = data/cifar/labels.txt
backup = backup/
top=2
labels.txt
capnomaskno
capnomaskyes
capyesmaskno
capyesmaskyes
cifar is in cfg folder and this labels.txt in cifar folder along with test.list and train.list

Ya the naming scheme is u have explained.
thanks

[rachana88]

@LolikaPadmanbhan are your labels custom in labels.list? Also can you let me know how your image set look like ? I am assuming it will be different person with cap mask ,no cap , no mask etc?

[rachana88]

@LolikaPadmanbhan can you please share how your label.list and train.list looks like? I am trying to do exactly what you were trying to do , I have my own dataset with own classes !!Thank you in advance

[estebanvel]

I will write the steps again , may be you forgot to do anyone of them
1: change the filtes to 4 at the end of cifar_small.cfg file instead of 10 as your classes are 10.
2: make a new labels list e.g test1.list and the contents will the your class names ( try to have one word label for each class , may be space is adding problems)
3: Copy all image files in one directory like cifar10 train folder and name them in the same manner as train folder of cifar 10 i.e 1_dog.png, 2_cat.png, 1_car.png, ....
4: Make a list train_img.list and copy absolute paths of these training images ( e.g /home/ahsan/custom_data/train/1_cat.png )
5: Your data file will looks like
classes=4
train = data/train.list
valid = data/test.list
labels = data/labels.list
backup = backup
top=2
6: Now do training
./darknet classifier train data/your_data.data cfg/your_cfg.cfg

Hi ahsan856jalal, I follow the steps in the https://pjreddie.com/darknet/train-cifar/ page, that are the same as you posted, but as sson I try to train the network I have following error
PS C:\Users\user236\darknet> ./darknet classifier train cfg/cifar.data cfg/cifar_small.cfg
cifar_small
1
First section must be [net] or [network]: No error
assertion "0" failed: file "./src/utils.c", line 256, function: error
2 [main] darknet 13628 cygwin_exception::open_stackdumpfile: Dumping stack trace to darknet.exe.stackdump

goinginto the "./src/utils.c", line 256 i found this function
void error(const char *s)
{
perror(s);
assert(0); //This is the line 256
exit(-1);
}

Can you help me to solve the issue, can you check in your utils.c file and check if is different from mine.
Thanks

[Sudhakar17]

How to get the validation/test accuracy for the cifar dataset/own dataset?
I used the following command:

./darknet classifier valid data/your_data.data cfg/your_cfg.cfg Pretrained_weights.weights

It shows the top 1 and top 2 results. Not the overall accuracy of the model

@ahsan856jalal @LolikaPadmanbhan

[LolikaPadmanbhan]

@Sudhakar17
Sorry for the late reply, u need to change the top value in the data file. if u want to see for all the classes.
thanks

[LolikaPadmanbhan]

@rachana88
hi,
this is my lable.list and make sure ur lables and folder names are not same.
labels.txt
capnomaskno
capnomaskyes
capyesmaskno
capyesmaskyes
And train.list would be a path of the images that are the traing folder
eg: /home/lolika/darknet/data/train/capyesmaskyes/0000_capyesmaskyes.jpg
Hope this is what u wear looking for .. ?
thanks

[Sadegh-Saberian]

Hi,

Is there any function that allows integration of the classification model into python code? Something similar to performDetect function for detection.

thanks

[riktimmondal]

I followed all the above mentioned instruction and tried to run a 12 class classification using the network of darknet19.cfg file but my accuracy is very bad after training it for 5000 iteration and it predicts only one class for every image with confidence of below 40%(same for every different image belonging to different class).
How can I fixed it and how to train the model on pretrained weights (provided in the official webpage like darknet 19,darknet 53 etc).
Thanks