Darknet: Training Steps

[hnhuang]

Hi Alexey,

In yolov3_voc.cfg, the steps = 40000,45000. However, I still get a lot of weights files in backup directory...

Best

Hongnian

[AlexeyAB]

This is normal. Sometimes you want to stop training and continue it later. So you can start from the last weights-file.
darknet.exe detector train data/voc.data cfg/yolov3-voc.cfg /backup/yolov3-voc_1000.weights

[hnhuang]

What does steps do in yolov3_voc.cfg then? I thought it is a way to control the number of weights files that it generates.

[AlexeyAB]

steps control when learning_rate will be changed according to policy and scales.

For these params in cfg-file:

learning_rate=0.001
burn_in=1000
max_batches = 50200
policy=steps
steps=40000,45000
scales=.1,.1

• burn_in=1000 - before 1000 iterations: learning_rate will vary according to this formula : https://github.com/AlexeyAB/darknet/blob/a0dc4d717ab2d95e5e90f5b7b6344e8074b81606/src/network.c#L54

• steps=40000 - at 40000 iterations: learning_rate will be multiplied by 0.1 (scales=.1): https://github.com/AlexeyAB/darknet/blob/a0dc4d717ab2d95e5e90f5b7b6344e8074b81606/src/network.c#L64

• steps=45000 - at 45000 iterations: learning_rate will be multiplied by 0.1 (scales=.1) again: https://github.com/AlexeyAB/darknet/blob/a0dc4d717ab2d95e5e90f5b7b6344e8074b81606/src/network.c#L64

[hnhuang]

Thanks. That is very helpful.

[i-chaochen]

steps control when learning_rate will be changed according to policy and scales.

For these params in cfg-file:

learning_rate=0.001
burn_in=1000
max_batches = 50200
policy=steps
steps=40000,45000
scales=.1,.1

• burn_in=1000 - before 1000 iterations: learning_rate will vary according to this formula : https://github.com/AlexeyAB/darknet/blob/a0dc4d717ab2d95e5e90f5b7b6344e8074b81606/src/network.c#L54
• steps=40000 - at 40000 iterations: learning_rate will be multiplied by 0.1 (scales=.1): https://github.com/AlexeyAB/darknet/blob/a0dc4d717ab2d95e5e90f5b7b6344e8074b81606/src/network.c#L64
• steps=45000 - at 45000 iterations: learning_rate will be multiplied by 0.1 (scales=.1) again: https://github.com/AlexeyAB/darknet/blob/a0dc4d717ab2d95e5e90f5b7b6344e8074b81606/src/network.c#L64

Is any particular reason why the first 1000 iterations learning rate is so small?

In addition, I noticed if steps and scales ared used in cfg, cfg will read a float array for two steps and scales, instead of scale and step. And scale and step is not initialised, so I assumed they will be 0?

But in upsample computation (blas.c), scale is used for computing out and in for the forward and backward, respectively.

                    if (forward) out[out_index] = scale*in[in_index];
                    else in[in_index] += scale*out[out_index];

I wonder would the scale be 0 and this upsample always 0 if used steps and scales?

I am not sure am I understood correctly. hope you can help. Thanks!
@AlexeyAB