Darknet: CUDA Error: out of memory

Created on 24 Dec 2019  路  4Comments  路  Source: AlexeyAB/darknet

C:\Users\karay\darknet>darknet detector train cfg/plate-obj.data cfg/plate-yolov3.cfg darknet53.conv.74
plate-yolov3
net.optimized_memory = 0
batch = 1, time_steps = 1, train = 1
   layer   filters  size/strd(dil)      input                output
   0 conv     32       3 x 3/ 1    416 x 416 x   3 ->  416 x 416 x  32 0.299 BF
   1 conv     64       3 x 3/ 2    416 x 416 x  32 ->  208 x 208 x  64 1.595 BF
   2 conv     32       1 x 1/ 1    208 x 208 x  64 ->  208 x 208 x  32 0.177 BF
   3 conv     64       3 x 3/ 1    208 x 208 x  32 ->  208 x 208 x  64 1.595 BF
   4 Shortcut Layer: 1
   5 conv    128       3 x 3/ 2    208 x 208 x  64 ->  104 x 104 x 128 1.595 BF
   6 conv     64       1 x 1/ 1    104 x 104 x 128 ->  104 x 104 x  64 0.177 BF
   7 conv    128       3 x 3/ 1    104 x 104 x  64 ->  104 x 104 x 128 1.595 BF
   8 Shortcut Layer: 5
   9 conv     64       1 x 1/ 1    104 x 104 x 128 ->  104 x 104 x  64 0.177 BF
  10 conv    128       3 x 3/ 1    104 x 104 x  64 ->  104 x 104 x 128 1.595 BF
  11 Shortcut Layer: 8
  12 conv    256       3 x 3/ 2    104 x 104 x 128 ->   52 x  52 x 256 1.595 BF
  13 conv    128       1 x 1/ 1     52 x  52 x 256 ->   52 x  52 x 128 0.177 BF
  14 conv    256       3 x 3/ 1     52 x  52 x 128 ->   52 x  52 x 256 1.595 BF
  15 Shortcut Layer: 12
  16 conv    128       1 x 1/ 1     52 x  52 x 256 ->   52 x  52 x 128 0.177 BF
  17 conv    256       3 x 3/ 1     52 x  52 x 128 ->   52 x  52 x 256 1.595 BF
  18 Shortcut Layer: 15
  19 conv    128       1 x 1/ 1     52 x  52 x 256 ->   52 x  52 x 128 0.177 BF
  20 conv    256       3 x 3/ 1     52 x  52 x 128 ->   52 x  52 x 256 1.595 BF
  21 Shortcut Layer: 18
  22 conv    128       1 x 1/ 1     52 x  52 x 256 ->   52 x  52 x 128 0.177 BF
  23 conv    256       3 x 3/ 1     52 x  52 x 128 ->   52 x  52 x 256 1.595 BF
  24 Shortcut Layer: 21
  25 conv    128       1 x 1/ 1     52 x  52 x 256 ->   52 x  52 x 128 0.177 BF
  26 conv    256       3 x 3/ 1     52 x  52 x 128 ->   52 x  52 x 256 1.595 BF
  27 Shortcut Layer: 24
  28 conv    128       1 x 1/ 1     52 x  52 x 256 ->   52 x  52 x 128 0.177 BF
  29 conv    256       3 x 3/ 1     52 x  52 x 128 ->   52 x  52 x 256 1.595 BF
  30 Shortcut Layer: 27
  31 conv    128       1 x 1/ 1     52 x  52 x 256 ->   52 x  52 x 128 0.177 BF
  32 conv    256       3 x 3/ 1     52 x  52 x 128 ->   52 x  52 x 256 1.595 BF
  33 Shortcut Layer: 30
  34 conv    128       1 x 1/ 1     52 x  52 x 256 ->   52 x  52 x 128 0.177 BF
  35 conv    256       3 x 3/ 1     52 x  52 x 128 ->   52 x  52 x 256 1.595 BF
  36 Shortcut Layer: 33
  37 conv    512       3 x 3/ 2     52 x  52 x 256 ->   26 x  26 x 512 1.595 BF
  38 conv    256       1 x 1/ 1     26 x  26 x 512 ->   26 x  26 x 256 0.177 BF
  39 conv    512       3 x 3/ 1     26 x  26 x 256 ->   26 x  26 x 512 1.595 BF
  40 Shortcut Layer: 37
  41 conv    256       1 x 1/ 1     26 x  26 x 512 ->   26 x  26 x 256 0.177 BF
  42 conv    512       3 x 3/ 1     26 x  26 x 256 ->   26 x  26 x 512 1.595 BF
  43 Shortcut Layer: 40
  44 conv    256       1 x 1/ 1     26 x  26 x 512 ->   26 x  26 x 256 0.177 BF
  45 conv    512       3 x 3/ 1     26 x  26 x 256 ->   26 x  26 x 512 1.595 BF
  46 Shortcut Layer: 43
  47 conv    256       1 x 1/ 1     26 x  26 x 512 ->   26 x  26 x 256 0.177 BF
  48 conv    512       3 x 3/ 1     26 x  26 x 256 ->   26 x  26 x 512 1.595 BF
  49 Shortcut Layer: 46
  50 conv    256       1 x 1/ 1     26 x  26 x 512 ->   26 x  26 x 256 0.177 BF
  51 conv    512       3 x 3/ 1     26 x  26 x 256 ->   26 x  26 x 512 1.595 BF
  52 Shortcut Layer: 49
  53 conv    256       1 x 1/ 1     26 x  26 x 512 ->   26 x  26 x 256 0.177 BF
  54 conv    512       3 x 3/ 1     26 x  26 x 256 ->   26 x  26 x 512 1.595 BF
  55 Shortcut Layer: 52
  56 conv    256       1 x 1/ 1     26 x  26 x 512 ->   26 x  26 x 256 0.177 BF
  57 conv    512       3 x 3/ 1     26 x  26 x 256 ->   26 x  26 x 512 1.595 BF
  58 Shortcut Layer: 55
  59 conv    256       1 x 1/ 1     26 x  26 x 512 ->   26 x  26 x 256 0.177 BF
  60 conv    512       3 x 3/ 1     26 x  26 x 256 ->   26 x  26 x 512 1.595 BF
  61 Shortcut Layer: 58
  62 conv   1024       3 x 3/ 2     26 x  26 x 512 ->   13 x  13 x1024 1.595 BF
  63 conv    512       1 x 1/ 1     13 x  13 x1024 ->   13 x  13 x 512 0.177 BF
  64 conv   1024       3 x 3/ 1     13 x  13 x 512 ->   13 x  13 x1024 1.595 BF
  65 Shortcut Layer: 62
  66 conv    512       1 x 1/ 1     13 x  13 x1024 ->   13 x  13 x 512 0.177 BF
  67 conv   1024       3 x 3/ 1     13 x  13 x 512 ->   13 x  13 x1024 1.595 BF
  68 Shortcut Layer: 65
  69 conv    512       1 x 1/ 1     13 x  13 x1024 ->   13 x  13 x 512 0.177 BF
  70 conv   1024       3 x 3/ 1     13 x  13 x 512 ->   13 x  13 x1024 1.595 BF
  71 Shortcut Layer: 68
  72 conv    512       1 x 1/ 1     13 x  13 x1024 ->   13 x  13 x 512 0.177 BF
  73 conv   1024       3 x 3/ 1     13 x  13 x 512 ->   13 x  13 x1024 1.595 BF
  74 Shortcut Layer: 71
  75 conv    512       1 x 1/ 1     13 x  13 x1024 ->   13 x  13 x 512 0.177 BF
  76 conv   1024       3 x 3/ 1     13 x  13 x 512 ->   13 x  13 x1024 1.595 BF
  77 conv    512       1 x 1/ 1     13 x  13 x1024 ->   13 x  13 x 512 0.177 BF
  78 conv   1024       3 x 3/ 1     13 x  13 x 512 ->   13 x  13 x1024 1.595 BF
  79 conv    512       1 x 1/ 1     13 x  13 x1024 ->   13 x  13 x 512 0.177 BF
  80 conv   1024       3 x 3/ 1     13 x  13 x 512 ->   13 x  13 x1024 1.595 BF
  81 conv     18       1 x 1/ 1     13 x  13 x1024 ->   13 x  13 x  18 0.006 BF
  82 yolo
[yolo] params: iou loss: mse (2), iou_norm: 0.75, cls_norm: 1.00, scale_x_y: 1.00
  83 route  79                                     ->   13 x  13 x 512
  84 conv    256       1 x 1/ 1     13 x  13 x 512 ->   13 x  13 x 256 0.044 BF
  85 upsample                 2x    13 x  13 x 256 ->   26 x  26 x 256
  86 route  85 61                                  ->   26 x  26 x 768
  87 conv    256       1 x 1/ 1     26 x  26 x 768 ->   26 x  26 x 256 0.266 BF
  88 conv    512       3 x 3/ 1     26 x  26 x 256 ->   26 x  26 x 512 1.595 BF
  89 conv    256       1 x 1/ 1     26 x  26 x 512 ->   26 x  26 x 256 0.177 BF
  90 conv    512       3 x 3/ 1     26 x  26 x 256 ->   26 x  26 x 512 1.595 BF
  91 conv    256       1 x 1/ 1     26 x  26 x 512 ->   26 x  26 x 256 0.177 BF
  92 conv    512       3 x 3/ 1     26 x  26 x 256 ->   26 x  26 x 512 1.595 BF
  93 conv     18       1 x 1/ 1     26 x  26 x 512 ->   26 x  26 x  18 0.012 BF
  94 yolo
[yolo] params: iou loss: mse (2), iou_norm: 0.75, cls_norm: 1.00, scale_x_y: 1.00
  95 route  91                                     ->   26 x  26 x 256
  96 conv    128       1 x 1/ 1     26 x  26 x 256 ->   26 x  26 x 128 0.044 BF
  97 upsample                 2x    26 x  26 x 128 ->   52 x  52 x 128
  98 route  97 36                                  ->   52 x  52 x 384
  99 conv    128       1 x 1/ 1     52 x  52 x 384 ->   52 x  52 x 128 0.266 BF
 100 conv    256       3 x 3/ 1     52 x  52 x 128 ->   52 x  52 x 256 1.595 BF
 101 conv    128       1 x 1/ 1     52 x  52 x 256 ->   52 x  52 x 128 0.177 BF
 102 conv    256       3 x 3/ 1     52 x  52 x 128 ->   52 x  52 x 256 1.595 BF
 103 conv    128       1 x 1/ 1     52 x  52 x 256 ->   52 x  52 x 128 0.177 BF
 104 conv    256       3 x 3/ 1     52 x  52 x 128 ->   52 x  52 x 256 1.595 BF
 105 conv     18       1 x 1/ 1     52 x  52 x 256 ->   52 x  52 x  18 0.025 BF
 106 yolo
[yolo] params: iou loss: mse (2), iou_norm: 0.75, cls_norm: 1.00, scale_x_y: 1.00
Total BFLOPS 65.290
 Allocate additional workspace_size = 12.46 MB
Loading weights from darknet53.conv.74...
 seen 64
Done! Loaded 75 layers from weights-file
Learning Rate: 0.001, Momentum: 0.9, Decay: 0.0005
 If error occurs - run training with flag: -dont_show
Resizing
608 x 608
 used slow CUDNN algo without Workspace! Need memory: 6653952, available: 6229196
 CUDNN-slow  Try to set subdivisions=64 in your cfg-file.
CUDA status Error: file: C:\Users\karay\darknet\src\dark_cuda.c : cuda_make_array() : line: 362 : build time: Dec 24 2019 - 22:34:53
CUDA Error: out of memory

My plate-yolov3.cfg file is like below

[net]
# Testing
batch=64
subdivisions=64
# Training
# batch=64
# subdivisions=64
width=416
height=416
channels=3
momentum=0.9
decay=0.0005
angle=0
saturation = 1.5
exposure = 1.5
hue=.1

learning_rate=0.001
burn_in=1000
max_batches = 4000
policy=steps
steps=4800,5400
scales=.1,.1

[convolutional]
batch_normalize=1
filters=32
size=3
stride=1
pad=1
activation=leaky

# Downsample

[convolutional]
batch_normalize=1
filters=64
size=3
stride=2
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=32
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=64
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

# Downsample

[convolutional]
batch_normalize=1
filters=128
size=3
stride=2
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=64
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=128
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=64
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=128
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

# Downsample

[convolutional]
batch_normalize=1
filters=256
size=3
stride=2
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear


[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

# Downsample

[convolutional]
batch_normalize=1
filters=512
size=3
stride=2
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear


[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear


[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear


[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear


[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear


[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

# Downsample

[convolutional]
batch_normalize=1
filters=1024
size=3
stride=2
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=512
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=1024
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=512
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=1024
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=512
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=1024
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=512
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=1024
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

######################

[convolutional]
batch_normalize=1
filters=512
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
size=3
stride=1
pad=1
filters=1024
activation=leaky

[convolutional]
batch_normalize=1
filters=512
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
size=3
stride=1
pad=1
filters=1024
activation=leaky

[convolutional]
batch_normalize=1
filters=512
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
size=3
stride=1
pad=1
filters=1024
activation=leaky

[convolutional]
size=1
stride=1
pad=1
filters=18
activation=linear


[yolo]
mask = 6,7,8
anchors = 10,13,  16,30,  33,23,  30,61,  62,45,  59,119,  116,90,  156,198,  373,326
classes=1
num=9
jitter=.3
ignore_thresh = .7
truth_thresh = 1
random=1


[route]
layers = -4

[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[upsample]
stride=2

[route]
layers = -1, 61



[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
size=3
stride=1
pad=1
filters=512
activation=leaky

[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
size=3
stride=1
pad=1
filters=512
activation=leaky

[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
size=3
stride=1
pad=1
filters=512
activation=leaky

[convolutional]
size=1
stride=1
pad=1
filters=18
activation=linear


[yolo]
mask = 3,4,5
anchors = 10,13,  16,30,  33,23,  30,61,  62,45,  59,119,  116,90,  156,198,  373,326
classes=1
num=9
jitter=.3
ignore_thresh = .7
truth_thresh = 1
random=1



[route]
layers = -4

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[upsample]
stride=2

[route]
layers = -1, 36



[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
size=3
stride=1
pad=1
filters=256
activation=leaky

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
size=3
stride=1
pad=1
filters=256
activation=leaky

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
size=3
stride=1
pad=1
filters=256
activation=leaky

[convolutional]
size=1
stride=1
pad=1
filters=18
activation=linear


[yolo]
mask = 0,1,2
anchors = 10,13,  16,30,  33,23,  30,61,  62,45,  59,119,  116,90,  156,198,  373,326
classes=1
num=9
jitter=.3
ignore_thresh = .7
truth_thresh = 1
random=1


Most helpful comment

Set random=0 in the last [yolo] layer in cfg-file
If it doesn't help, also set width=320 height=320

All 4 comments

Set random=0 in the last [yolo] layer in cfg-file
If it doesn't help, also set width=320 height=320

Is this alright

C:\Users\karay\darknet>darknet.exe detector train cfg/plate-obj.data cfg/plate-yolov3.cfg darknet53.conv.74
plate-yolov3
net.optimized_memory = 0
batch = 1, time_steps = 1, train = 1
layer filters size/strd(dil) input output
0 conv 32 3 x 3/ 1 320 x 320 x 3 -> 320 x 320 x 32 0.177 BF
1 conv 64 3 x 3/ 2 320 x 320 x 32 -> 160 x 160 x 64 0.944 BF
2 conv 32 1 x 1/ 1 160 x 160 x 64 -> 160 x 160 x 32 0.105 BF
3 conv 64 3 x 3/ 1 160 x 160 x 32 -> 160 x 160 x 64 0.944 BF
4 Shortcut Layer: 1
5 conv 128 3 x 3/ 2 160 x 160 x 64 -> 80 x 80 x 128 0.944 BF
6 conv 64 1 x 1/ 1 80 x 80 x 128 -> 80 x 80 x 64 0.105 BF
7 conv 128 3 x 3/ 1 80 x 80 x 64 -> 80 x 80 x 128 0.944 BF
8 Shortcut Layer: 5
9 conv 64 1 x 1/ 1 80 x 80 x 128 -> 80 x 80 x 64 0.105 BF
10 conv 128 3 x 3/ 1 80 x 80 x 64 -> 80 x 80 x 128 0.944 BF
11 Shortcut Layer: 8
12 conv 256 3 x 3/ 2 80 x 80 x 128 -> 40 x 40 x 256 0.944 BF
13 conv 128 1 x 1/ 1 40 x 40 x 256 -> 40 x 40 x 128 0.105 BF
14 conv 256 3 x 3/ 1 40 x 40 x 128 -> 40 x 40 x 256 0.944 BF
15 Shortcut Layer: 12
16 conv 128 1 x 1/ 1 40 x 40 x 256 -> 40 x 40 x 128 0.105 BF
17 conv 256 3 x 3/ 1 40 x 40 x 128 -> 40 x 40 x 256 0.944 BF
18 Shortcut Layer: 15
19 conv 128 1 x 1/ 1 40 x 40 x 256 -> 40 x 40 x 128 0.105 BF
20 conv 256 3 x 3/ 1 40 x 40 x 128 -> 40 x 40 x 256 0.944 BF
21 Shortcut Layer: 18
22 conv 128 1 x 1/ 1 40 x 40 x 256 -> 40 x 40 x 128 0.105 BF
23 conv 256 3 x 3/ 1 40 x 40 x 128 -> 40 x 40 x 256 0.944 BF
24 Shortcut Layer: 21
25 conv 128 1 x 1/ 1 40 x 40 x 256 -> 40 x 40 x 128 0.105 BF
26 conv 256 3 x 3/ 1 40 x 40 x 128 -> 40 x 40 x 256 0.944 BF
27 Shortcut Layer: 24
28 conv 128 1 x 1/ 1 40 x 40 x 256 -> 40 x 40 x 128 0.105 BF
29 conv 256 3 x 3/ 1 40 x 40 x 128 -> 40 x 40 x 256 0.944 BF
30 Shortcut Layer: 27
31 conv 128 1 x 1/ 1 40 x 40 x 256 -> 40 x 40 x 128 0.105 BF
32 conv 256 3 x 3/ 1 40 x 40 x 128 -> 40 x 40 x 256 0.944 BF
33 Shortcut Layer: 30
34 conv 128 1 x 1/ 1 40 x 40 x 256 -> 40 x 40 x 128 0.105 BF
35 conv 256 3 x 3/ 1 40 x 40 x 128 -> 40 x 40 x 256 0.944 BF
36 Shortcut Layer: 33
37 conv 512 3 x 3/ 2 40 x 40 x 256 -> 20 x 20 x 512 0.944 BF
38 conv 256 1 x 1/ 1 20 x 20 x 512 -> 20 x 20 x 256 0.105 BF
39 conv 512 3 x 3/ 1 20 x 20 x 256 -> 20 x 20 x 512 0.944 BF
40 Shortcut Layer: 37
41 conv 256 1 x 1/ 1 20 x 20 x 512 -> 20 x 20 x 256 0.105 BF
42 conv 512 3 x 3/ 1 20 x 20 x 256 -> 20 x 20 x 512 0.944 BF
43 Shortcut Layer: 40
44 conv 256 1 x 1/ 1 20 x 20 x 512 -> 20 x 20 x 256 0.105 BF
45 conv 512 3 x 3/ 1 20 x 20 x 256 -> 20 x 20 x 512 0.944 BF
46 Shortcut Layer: 43
47 conv 256 1 x 1/ 1 20 x 20 x 512 -> 20 x 20 x 256 0.105 BF
48 conv 512 3 x 3/ 1 20 x 20 x 256 -> 20 x 20 x 512 0.944 BF
49 Shortcut Layer: 46
50 conv 256 1 x 1/ 1 20 x 20 x 512 -> 20 x 20 x 256 0.105 BF
51 conv 512 3 x 3/ 1 20 x 20 x 256 -> 20 x 20 x 512 0.944 BF
52 Shortcut Layer: 49
53 conv 256 1 x 1/ 1 20 x 20 x 512 -> 20 x 20 x 256 0.105 BF
54 conv 512 3 x 3/ 1 20 x 20 x 256 -> 20 x 20 x 512 0.944 BF
55 Shortcut Layer: 52
56 conv 256 1 x 1/ 1 20 x 20 x 512 -> 20 x 20 x 256 0.105 BF
57 conv 512 3 x 3/ 1 20 x 20 x 256 -> 20 x 20 x 512 0.944 BF
58 Shortcut Layer: 55
59 conv 256 1 x 1/ 1 20 x 20 x 512 -> 20 x 20 x 256 0.105 BF
60 conv 512 3 x 3/ 1 20 x 20 x 256 -> 20 x 20 x 512 0.944 BF
61 Shortcut Layer: 58
62 conv 1024 3 x 3/ 2 20 x 20 x 512 -> 10 x 10 x1024 0.944 BF
63 conv 512 1 x 1/ 1 10 x 10 x1024 -> 10 x 10 x 512 0.105 BF
64 conv 1024 3 x 3/ 1 10 x 10 x 512 -> 10 x 10 x1024 0.944 BF
65 Shortcut Layer: 62
66 conv 512 1 x 1/ 1 10 x 10 x1024 -> 10 x 10 x 512 0.105 BF
67 conv 1024 3 x 3/ 1 10 x 10 x 512 -> 10 x 10 x1024 0.944 BF
68 Shortcut Layer: 65
69 conv 512 1 x 1/ 1 10 x 10 x1024 -> 10 x 10 x 512 0.105 BF
70 conv 1024 3 x 3/ 1 10 x 10 x 512 -> 10 x 10 x1024 0.944 BF
71 Shortcut Layer: 68
72 conv 512 1 x 1/ 1 10 x 10 x1024 -> 10 x 10 x 512 0.105 BF
73 conv 1024 3 x 3/ 1 10 x 10 x 512 -> 10 x 10 x1024 0.944 BF
74 Shortcut Layer: 71
75 conv 512 1 x 1/ 1 10 x 10 x1024 -> 10 x 10 x 512 0.105 BF
76 conv 1024 3 x 3/ 1 10 x 10 x 512 -> 10 x 10 x1024 0.944 BF
77 conv 512 1 x 1/ 1 10 x 10 x1024 -> 10 x 10 x 512 0.105 BF
78 conv 1024 3 x 3/ 1 10 x 10 x 512 -> 10 x 10 x1024 0.944 BF
79 conv 512 1 x 1/ 1 10 x 10 x1024 -> 10 x 10 x 512 0.105 BF
80 conv 1024 3 x 3/ 1 10 x 10 x 512 -> 10 x 10 x1024 0.944 BF
81 conv 18 1 x 1/ 1 10 x 10 x1024 -> 10 x 10 x 18 0.004 BF
82 yolo
[yolo] params: iou loss: mse (2), iou_norm: 0.75, cls_norm: 1.00, scale_x_y: 1.00
83 route 79 -> 10 x 10 x 512
84 conv 256 1 x 1/ 1 10 x 10 x 512 -> 10 x 10 x 256 0.026 BF
85 upsample 2x 10 x 10 x 256 -> 20 x 20 x 256
86 route 85 61 -> 20 x 20 x 768
87 conv 256 1 x 1/ 1 20 x 20 x 768 -> 20 x 20 x 256 0.157 BF
88 conv 512 3 x 3/ 1 20 x 20 x 256 -> 20 x 20 x 512 0.944 BF
89 conv 256 1 x 1/ 1 20 x 20 x 512 -> 20 x 20 x 256 0.105 BF
90 conv 512 3 x 3/ 1 20 x 20 x 256 -> 20 x 20 x 512 0.944 BF
91 conv 256 1 x 1/ 1 20 x 20 x 512 -> 20 x 20 x 256 0.105 BF
92 conv 512 3 x 3/ 1 20 x 20 x 256 -> 20 x 20 x 512 0.944 BF
93 conv 18 1 x 1/ 1 20 x 20 x 512 -> 20 x 20 x 18 0.007 BF
94 yolo
[yolo] params: iou loss: mse (2), iou_norm: 0.75, cls_norm: 1.00, scale_x_y: 1.00
95 route 91 -> 20 x 20 x 256
96 conv 128 1 x 1/ 1 20 x 20 x 256 -> 20 x 20 x 128 0.026 BF
97 upsample 2x 20 x 20 x 128 -> 40 x 40 x 128
98 route 97 36 -> 40 x 40 x 384
99 conv 128 1 x 1/ 1 40 x 40 x 384 -> 40 x 40 x 128 0.157 BF
100 conv 256 3 x 3/ 1 40 x 40 x 128 -> 40 x 40 x 256 0.944 BF
101 conv 128 1 x 1/ 1 40 x 40 x 256 -> 40 x 40 x 128 0.105 BF
102 conv 256 3 x 3/ 1 40 x 40 x 128 -> 40 x 40 x 256 0.944 BF
103 conv 128 1 x 1/ 1 40 x 40 x 256 -> 40 x 40 x 128 0.105 BF
104 conv 256 3 x 3/ 1 40 x 40 x 128 -> 40 x 40 x 256 0.944 BF
105 conv 18 1 x 1/ 1 40 x 40 x 256 -> 40 x 40 x 18 0.015 BF
106 yolo
[yolo] params: iou loss: mse (2), iou_norm: 0.75, cls_norm: 1.00, scale_x_y: 1.00
Total BFLOPS 38.633
Allocate additional workspace_size = 7.37 MB
Loading weights from darknet53.conv.74...
seen 64
Done! Loaded 75 layers from weights-file
Learning Rate: 0.001, Momentum: 0.9, Decay: 0.0005
If error occurs - run training with flag: -dont_show

C:\Users\karay\darknet>darknet53.convdarknet.exe detector train cfg/plate-obj.data cfg/plate-yolov3.cfg darknet53.conv.74 -dont_show
'darknet53.convdarknet.exe' is not recognized as an internal or external command,
operable program or batch file.

C:\Users\karay\darknet>darknet.exe detector train cfg/plate-obj.data cfg/plate-yolov3.cfg darknet53.conv.74
plate-yolov3
net.optimized_memory = 0
batch = 1, time_steps = 1, train = 1
layer filters size/strd(dil) input output
0 conv 32 3 x 3/ 1 320 x 320 x 3 -> 320 x 320 x 32 0.177 BF
1 conv 64 3 x 3/ 2 320 x 320 x 32 -> 160 x 160 x 64 0.944 BF
2 conv 32 1 x 1/ 1 160 x 160 x 64 -> 160 x 160 x 32 0.105 BF
3 conv 64 3 x 3/ 1 160 x 160 x 32 -> 160 x 160 x 64 0.944 BF
4 Shortcut Layer: 1
5 conv 128 3 x 3/ 2 160 x 160 x 64 -> 80 x 80 x 128 0.944 BF
6 conv 64 1 x 1/ 1 80 x 80 x 128 -> 80 x 80 x 64 0.105 BF
7 conv 128 3 x 3/ 1 80 x 80 x 64 -> 80 x 80 x 128 0.944 BF
8 Shortcut Layer: 5
9 conv 64 1 x 1/ 1 80 x 80 x 128 -> 80 x 80 x 64 0.105 BF
10 conv 128 3 x 3/ 1 80 x 80 x 64 -> 80 x 80 x 128 0.944 BF
11 Shortcut Layer: 8
12 conv 256 3 x 3/ 2 80 x 80 x 128 -> 40 x 40 x 256 0.944 BF
13 conv 128 1 x 1/ 1 40 x 40 x 256 -> 40 x 40 x 128 0.105 BF
14 conv 256 3 x 3/ 1 40 x 40 x 128 -> 40 x 40 x 256 0.944 BF
15 Shortcut Layer: 12
16 conv 128 1 x 1/ 1 40 x 40 x 256 -> 40 x 40 x 128 0.105 BF
17 conv 256 3 x 3/ 1 40 x 40 x 128 -> 40 x 40 x 256 0.944 BF
18 Shortcut Layer: 15
19 conv 128 1 x 1/ 1 40 x 40 x 256 -> 40 x 40 x 128 0.105 BF
20 conv 256 3 x 3/ 1 40 x 40 x 128 -> 40 x 40 x 256 0.944 BF
21 Shortcut Layer: 18
22 conv 128 1 x 1/ 1 40 x 40 x 256 -> 40 x 40 x 128 0.105 BF
23 conv 256 3 x 3/ 1 40 x 40 x 128 -> 40 x 40 x 256 0.944 BF
24 Shortcut Layer: 21
25 conv 128 1 x 1/ 1 40 x 40 x 256 -> 40 x 40 x 128 0.105 BF
26 conv 256 3 x 3/ 1 40 x 40 x 128 -> 40 x 40 x 256 0.944 BF
27 Shortcut Layer: 24
28 conv 128 1 x 1/ 1 40 x 40 x 256 -> 40 x 40 x 128 0.105 BF
29 conv 256 3 x 3/ 1 40 x 40 x 128 -> 40 x 40 x 256 0.944 BF
30 Shortcut Layer: 27
31 conv 128 1 x 1/ 1 40 x 40 x 256 -> 40 x 40 x 128 0.105 BF
32 conv 256 3 x 3/ 1 40 x 40 x 128 -> 40 x 40 x 256 0.944 BF
33 Shortcut Layer: 30
34 conv 128 1 x 1/ 1 40 x 40 x 256 -> 40 x 40 x 128 0.105 BF
35 conv 256 3 x 3/ 1 40 x 40 x 128 -> 40 x 40 x 256 0.944 BF
36 Shortcut Layer: 33
37 conv 512 3 x 3/ 2 40 x 40 x 256 -> 20 x 20 x 512 0.944 BF
38 conv 256 1 x 1/ 1 20 x 20 x 512 -> 20 x 20 x 256 0.105 BF
39 conv 512 3 x 3/ 1 20 x 20 x 256 -> 20 x 20 x 512 0.944 BF
40 Shortcut Layer: 37
41 conv 256 1 x 1/ 1 20 x 20 x 512 -> 20 x 20 x 256 0.105 BF
42 conv 512 3 x 3/ 1 20 x 20 x 256 -> 20 x 20 x 512 0.944 BF
43 Shortcut Layer: 40
44 conv 256 1 x 1/ 1 20 x 20 x 512 -> 20 x 20 x 256 0.105 BF
45 conv 512 3 x 3/ 1 20 x 20 x 256 -> 20 x 20 x 512 0.944 BF
46 Shortcut Layer: 43
47 conv 256 1 x 1/ 1 20 x 20 x 512 -> 20 x 20 x 256 0.105 BF
48 conv 512 3 x 3/ 1 20 x 20 x 256 -> 20 x 20 x 512 0.944 BF
49 Shortcut Layer: 46
50 conv 256 1 x 1/ 1 20 x 20 x 512 -> 20 x 20 x 256 0.105 BF
51 conv 512 3 x 3/ 1 20 x 20 x 256 -> 20 x 20 x 512 0.944 BF
52 Shortcut Layer: 49
53 conv 256 1 x 1/ 1 20 x 20 x 512 -> 20 x 20 x 256 0.105 BF
54 conv 512 3 x 3/ 1 20 x 20 x 256 -> 20 x 20 x 512 0.944 BF
55 Shortcut Layer: 52
56 conv 256 1 x 1/ 1 20 x 20 x 512 -> 20 x 20 x 256 0.105 BF
57 conv 512 3 x 3/ 1 20 x 20 x 256 -> 20 x 20 x 512 0.944 BF
58 Shortcut Layer: 55
59 conv 256 1 x 1/ 1 20 x 20 x 512 -> 20 x 20 x 256 0.105 BF
60 conv 512 3 x 3/ 1 20 x 20 x 256 -> 20 x 20 x 512 0.944 BF
61 Shortcut Layer: 58
62 conv 1024 3 x 3/ 2 20 x 20 x 512 -> 10 x 10 x1024 0.944 BF
63 conv 512 1 x 1/ 1 10 x 10 x1024 -> 10 x 10 x 512 0.105 BF
64 conv 1024 3 x 3/ 1 10 x 10 x 512 -> 10 x 10 x1024 0.944 BF
65 Shortcut Layer: 62
66 conv 512 1 x 1/ 1 10 x 10 x1024 -> 10 x 10 x 512 0.105 BF
67 conv 1024 3 x 3/ 1 10 x 10 x 512 -> 10 x 10 x1024 0.944 BF
68 Shortcut Layer: 65
69 conv 512 1 x 1/ 1 10 x 10 x1024 -> 10 x 10 x 512 0.105 BF
70 conv 1024 3 x 3/ 1 10 x 10 x 512 -> 10 x 10 x1024 0.944 BF
71 Shortcut Layer: 68
72 conv 512 1 x 1/ 1 10 x 10 x1024 -> 10 x 10 x 512 0.105 BF
73 conv 1024 3 x 3/ 1 10 x 10 x 512 -> 10 x 10 x1024 0.944 BF
74 Shortcut Layer: 71
75 conv 512 1 x 1/ 1 10 x 10 x1024 -> 10 x 10 x 512 0.105 BF
76 conv 1024 3 x 3/ 1 10 x 10 x 512 -> 10 x 10 x1024 0.944 BF
77 conv 512 1 x 1/ 1 10 x 10 x1024 -> 10 x 10 x 512 0.105 BF
78 conv 1024 3 x 3/ 1 10 x 10 x 512 -> 10 x 10 x1024 0.944 BF
79 conv 512 1 x 1/ 1 10 x 10 x1024 -> 10 x 10 x 512 0.105 BF
80 conv 1024 3 x 3/ 1 10 x 10 x 512 -> 10 x 10 x1024 0.944 BF
81 conv 18 1 x 1/ 1 10 x 10 x1024 -> 10 x 10 x 18 0.004 BF
82 yolo
[yolo] params: iou loss: mse (2), iou_norm: 0.75, cls_norm: 1.00, scale_x_y: 1.00
83 route 79 -> 10 x 10 x 512
84 conv 256 1 x 1/ 1 10 x 10 x 512 -> 10 x 10 x 256 0.026 BF
85 upsample 2x 10 x 10 x 256 -> 20 x 20 x 256
86 route 85 61 -> 20 x 20 x 768
87 conv 256 1 x 1/ 1 20 x 20 x 768 -> 20 x 20 x 256 0.157 BF
88 conv 512 3 x 3/ 1 20 x 20 x 256 -> 20 x 20 x 512 0.944 BF
89 conv 256 1 x 1/ 1 20 x 20 x 512 -> 20 x 20 x 256 0.105 BF
90 conv 512 3 x 3/ 1 20 x 20 x 256 -> 20 x 20 x 512 0.944 BF
91 conv 256 1 x 1/ 1 20 x 20 x 512 -> 20 x 20 x 256 0.105 BF
92 conv 512 3 x 3/ 1 20 x 20 x 256 -> 20 x 20 x 512 0.944 BF
93 conv 18 1 x 1/ 1 20 x 20 x 512 -> 20 x 20 x 18 0.007 BF
94 yolo
[yolo] params: iou loss: mse (2), iou_norm: 0.75, cls_norm: 1.00, scale_x_y: 1.00
95 route 91 -> 20 x 20 x 256
96 conv 128 1 x 1/ 1 20 x 20 x 256 -> 20 x 20 x 128 0.026 BF
97 upsample 2x 20 x 20 x 128 -> 40 x 40 x 128
98 route 97 36 -> 40 x 40 x 384
99 conv 128 1 x 1/ 1 40 x 40 x 384 -> 40 x 40 x 128 0.157 BF
100 conv 256 3 x 3/ 1 40 x 40 x 128 -> 40 x 40 x 256 0.944 BF
101 conv 128 1 x 1/ 1 40 x 40 x 256 -> 40 x 40 x 128 0.105 BF
102 conv 256 3 x 3/ 1 40 x 40 x 128 -> 40 x 40 x 256 0.944 BF
103 conv 128 1 x 1/ 1 40 x 40 x 256 -> 40 x 40 x 128 0.105 BF
104 conv 256 3 x 3/ 1 40 x 40 x 128 -> 40 x 40 x 256 0.944 BF
105 conv 18 1 x 1/ 1 40 x 40 x 256 -> 40 x 40 x 18 0.015 BF
106 yolo
[yolo] params: iou loss: mse (2), iou_norm: 0.75, cls_norm: 1.00, scale_x_y: 1.00
Total BFLOPS 38.633
Allocate additional workspace_size = 7.37 MB
Loading weights from darknet53.conv.74...
seen 64
Done! Loaded 75 layers from weights-file
Learning Rate: 0.001, Momentum: 0.9, Decay: 0.0005
If error occurs - run training with flag: -dont_show

batch / subdivisions is the amount of VRAM darknet will use when training

I had this same issue. Try to modify batch and subdivisions params first. If that does not help, change the width and height params.

My current config file is this, after tweaking multiple times to not get a CUDA Out of Memory error.

Testing

batch=8
subdivisions=64

Training

batch=8
subdivisions=64
width=320
height=320

Was this page helpful?
0 / 5 - 0 ratings

Related issues

yongcong1415 picture yongcong1415  路  3Comments

off99555 picture off99555  路  3Comments

zihaozhang9 picture zihaozhang9  路  3Comments

siddharth2395 picture siddharth2395  路  3Comments

louisondumont picture louisondumont  路  3Comments