Darknet: Loss graph during training

[abeyang00]

Is there a way to show loss graph during training like tensorflow?

[springkim]

Hi @abeyang00
Here! https://github.com/AlexeyAB/darknet.
He made loss plot for training.

[abeyang00]

@springkim can you tell me where the plot is located in his folder? is it in .c file in src??

[ahsan856jalal]

https://github.com/AlexeyAB/darknet/issues/504#issuecomment-377500006
your answer is at the bottom

[Caroline1994]

can someone tell me how to show loss graph during training when i use pjreddie's darknet

[Sikandarkhan]

Any update on this thread?

[rbarman]

I found one solution here : https://github.com/Jumabek/darknet_scripts/#how-to-plot-yolo-loss

You basically need to save the output of ./darknet detector train <> into a log file and then python plot_yolo_log.py log_file.log

Note that the plot does not show in a jupyter-notebook even with %matplotlib inline. A work around is to copy all plot related code from https://github.com/Jumabek/darknet_scripts/blob/master/plot_yolo_log.py into a new function.

[AlexeyAB]

You can use repo https://github.com/AlexeyAB/darknet that shows Loss & mAP chart during Training:

[groszste]

@AlexeyAB is this plot of the training loss or validation loss? If training loss, do you have a way of viewing the validation loss?

[AlexeyAB]

@groszste
It is Training loss and Validation mAP.
For me it isn't necessary to see Validation loss, it is much better to see Validation mAP.

[JakupGuven]

@AlexeyAB
What commands do you use to display validation mAP during training?

[krxxxxxxxanc]

@JakupGuven

From README: https://github.com/AlexeyAB/darknet/blob/master/README.md

"Or just train with -map flag:

darknet.exe detector train data/obj.data yolo-obj.cfg darknet53.conv.74 -map

So you will see mAP-chart (red-line) in the Loss-chart Window. mAP will be calculated for each 4 Epochs using valid=valid.txt file that is specified in obj.data file (1 Epoch = images_in_train_txt / batch iterations)"

[yjdeveloper]

I have followed the steps given by Mr. @AlexeyAB and got the red line but my problem is how to plot a mAP after every 100 iteration. In your documentation until 1000 iteration, but i want in every 100 iteration.

[snphnolt]

@yjdeveloper have you figured out how to downscale the mAP calculation to a shorter interval?

[fcakyon]

@yjdeveloper @snphnolt use this version with -map 0.02 for map calculation at every 0.02 epoch (starts after warmup iterations)

https://github.com/fcakyon/darknet

[nehasoni3]

mAP-chart

Where this map graph has seen?

[nehasoni3]

I have followed the steps given by Mr. @AlexeyAB and got the red line but my problem is how to plot a mAP after every 100 iteration. In your documentation until 1000 iteration, but i want in every 100 iteration.

How do you got the red line?

[ghost]

@AlexeyAB

I am using your repo to detect a custom objects using yolov3. however I have get in to trouble. The predictions.jpg image do not draw the confidence score but it draws the class id.

i traced the image.c code and I have found that in the function definition

void draw_detections_v3(image im, detection dets, int num, float thresh, char *names, image **alphabet, int classes, int ext_output)

how to resolev e the issue?

[ghost]

please, anyone, help. which function I have to use in AlexeyAB yolo repository in order to get confidence score drawings on the predictions.jpg image file???? I have get only class Id using this

!./darknet detector test data/trainer.data cfg/yolov3.cfg backup/yolov3_last.weights -thresh 0.1 -iou_thresh 0.3 data/img/tb500.jpg

[Leprechault]

You can use repo https://github.com/AlexeyAB/darknet that shows Loss & mAP chart during Training:

The command ./darknet detector demo ... -json_port 8070 -mjpeg_port 8090 works very well, but is there any way to save the image in vectorial format like eg. *pdf, *svg, *ps?

[Leprechault]

I found one solution here : https://github.com/Jumabek/darknet_scripts/#how-to-plot-yolo-loss

You basically need to save the output of ./darknet detector train <> into a log file and then python plot_yolo_log.py log_file.log

Note that the plot does not show in a jupyter-notebook even with %matplotlib inline. A work around is to copy all plot related code from https://github.com/Jumabek/darknet_scripts/blob/master/plot_yolo_log.py into a new function.

@rbarman in the log.txt output what's the information about mAP?

[ak3509311]

How to save the loss graph on drive because i run the code on colab .

[harshkc03]

I'm training Yolov3-tiny on colab using the following command-
!./darknet detector train /content/obj.data /content/yolov3-tiny-obj.cfg backup/yolov3-tiny-obj_last.weights -dont_show -mjpeg_port 8090 -map

It shows MJPEG-stream sent in the output after every iteration and i know we have to use http://ip-address:8090 format to access the chart, but I'm unable to find the ip-address of my colab notebook. I tried using addresses from !ifconfig and !curl ipecho.net/plain but still no result.
Any help would be appreciated.

[himewel]

@harshkc03 I found this quote in StackOverflow. I still not found a way to propagate the json and graph at same time, but you can try something like this to train and see your graph updating. It prints a url that you can access your loss graph with the follow commands:

!wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip
!unzip ngrok-stable-linux-amd64.zip

get_ipython().system_raw('./ngrok http 8090 &')

!curl -s http://localhost:4040/api/tunnels | python3 -c \
 "import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])"

After this, start your training:

!./darknet detector train /content/obj.data /content/yolov3-tiny-obj.cfg backup/yolov3-tiny-obj_last.weights -dont_show -mjpeg_port 8090 -map

[francismontalbo]

Is there a way to produce the loss curve and mAP from an existing weight?

[harshkc03]

@francismontalbo you can obtain mAP of the existing weight using the command-
./darknet detector map data/obj.data yolo-obj.cfg backup\yolo-obj_last.weights
but you cannot generate the loss curve of an existing weight. Loss curve generates only during training.

[francismontalbo]

@francismontalbo you can obtain mAP of the existing weight using the command-

./darknet detector map data/obj.data yolo-obj.cfg backup\yolo-obj_last.weights

but you cannot generate the loss curve of an existing weight. Loss curve generates only during training.

Yes, I've been using that. I see, thank you for the response good sir.

[wc1997]

You can use pyngrok python package to display loss graph

!pip install pyngrok
from pyngrok import ngrok# Open a HTTP tunnel on port 8090
public_url = ngrok.connect(port = '8090')

public_url

Then run your training with flags

-mjpeg_port 8090 -map

[harshkc03]

You can use pyngrok python package to display loss graph

!pip install pyngrok
from pyngrok import ngrok# Open a HTTP tunnel on port 8090
public_url = ngrok.connect(port = '8090')

public_url

Then run your training with flags

-mjpeg_port 8090 -map

Thankyou sir, it works as expected.

[himewel]

You can use pyngrok python package to display loss graph

!pip install pyngrok
from pyngrok import ngrok# Open a HTTP tunnel on port 8090
public_url = ngrok.connect(port = '8090')

public_url

Then run your training with flags

-mjpeg_port 8090 -map

Seems much more elegant than my response, ty auhdsuahsduahs

[vishnuvardhan58]

Hello , I am getting the following error while using the command "!./darknet detector train data/obj.data cfg/yolov3_custom.cfg darknet53.conv.74 -dont_show -mjpeg_port 8090 -map".I am using google colab.

The connection to http://d80c91c46410.ngrok.io was successfully tunneled to your ngrok client, but the client failed to establish a connection to the local address localhost:8090.

Make sure that a web service is running on localhost:8090 and that it is a valid address.

The error encountered was: dial tcp 127.0.0.1:8090: connect: connection refused

[sercangokturk]

I have followed the steps given by Mr. @AlexeyAB and got the red line but my problem is how to plot a mAP after every 100 iteration. In your documentation until 1000 iteration, but i want in every 100 iteration.

Did you find a solution? Thanks in advance.