Pytorch-yolov3: Out of Memory and machine freezes

[kracekumar]

I'm running the custom training on 25 images with the steps specified in the readme. To train the network, I'm using paperspace service P4000, 8GB GPU RAM and 30 GB CPU RAM. After running three epochs, the code freezes and the machine refuses to accept the SSH connection. I guess, the machine runs out of memory. Can you suggest a minimum CPU and GPU RAM size?

Here is the sample log while running.

---- [Epoch 3/100, Batch 0/1] ----
+------------+--------------+--------------+--------------+
| Metrics    | YOLO Layer 0 | YOLO Layer 1 | YOLO Layer 2 |
+------------+--------------+--------------+--------------+
| grid_size  | 10           | 20           | 40           |
| loss       | 66.810303    | 58.259697    | 69.881714    |
| x          | 0.014225     | 0.149338     | 0.000203     |
| y          | 0.023167     | 0.414138     | 0.504312     |
| w          | 0.891367     | 0.332123     | 0.385917     |
| h          | 4.693002     | 0.286055     | 3.041605     |
| conf       | 60.534138    | 56.576481    | 65.782593    |
| cls        | 0.654405     | 0.501562     | 0.167081     |
| cls_acc    | 100.00%      | 100.00%      | 100.00%      |
| recall50   | 0.000000     | 0.000000     | 0.000000     |
| recall75   | 0.000000     | 0.000000     | 0.000000     |
| precision  | 0.000000     | 0.000000     | 0.000000     |
| conf_obj   | 0.450175     | 0.519545     | 0.710236     |
| conf_noobj | 0.436896     | 0.418163     | 0.477449     |
+------------+--------------+--------------+--------------+
Total loss 194.95172119140625
---- ETA 0:00:00

---- Evaluating Model ----
Detecting objects:   0%|                                                                                                                                    | 0/1 [00:00<?, ?it/s]

[shihanyu]

same problem

[kracekumar]

செவ்., 10 மார்., 2020, பிற்பகல் 2:19 அன்று shihanyu <
[email protected]> எழுதியது:

same problem

After adding an extra line at the end of the class names file, the problem
disappeared.

I don’t know if this is the solution.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/eriklindernoren/PyTorch-YOLOv3/issues/433?email_source=notifications&email_token=AACME6P4TOHETI5CIKIYP43RGX5HZA5CNFSM4LBR33LKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOKQU6A#issuecomment-596970104,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACME6MRZHBUVA5VCHSABXTRGX5HZANCNFSM4LBR33LA
.

>

Sent from Gmail Mobile

[shihanyu]

I solved the problem by adding more data, and I found that the lines of train.txt and valid.txt should be times of 8.

[tand826]

For me, train.py often stopped at the calculation of IoU in bbox_iou func, so I changed non_max_suppression in utils/utils.py as below

import torchvision
def non_max_suppression(prediction, conf_thres=0.5, nms_thres=0.4):
    """
    Removes detections with lower object confidence score than 'conf_thres' and performs
    Non-Maximum Suppression to further filter detections.
    Returns detections with shape:
        (x1, y1, x2, y2, object_conf, class_score, class_pred)
    """

    # From (center x, center y, width, height) to (x1, y1, x2, y2)
    prediction[..., :4] = xywh2xyxy(prediction[..., :4])
    output = [None for _ in range(len(prediction))]
    for image_i, image_pred in enumerate(prediction):
        # Filter out confidence scores below threshold
        image_pred = image_pred[image_pred[:, 4] >= conf_thres]
        # If none are remaining => process next image
        if not image_pred.size(0):
            continue
        # Object confidence times class confidence
        boxes = image_pred[:, :4]
        scores = image_pred[:, 4] * image_pred[:, 5:].max(1)[0]
        iou_threshold = nms_thres
        class_confs, class_preds = image_pred[:, 5:].max(1, keepdim=True)
        keep = torchvision.ops.nms(boxes, scores, iou_threshold)

        boxes = boxes[keep]
        class_confs = class_confs[keep]
        scores = scores[keep].view_as(class_confs)
        class_preds = class_preds[keep].type(dtype=torch.float)
        output[image_i] = torch.cat([boxes, class_confs, scores, class_preds], dim=1)

    return output

[buptlj]

the same problem.
`

Perform non-maximum suppression

    keep_boxes = []
    while detections.size(0):
        large_overlap = bbox_iou(detections[0, :4].unsqueeze(0), detections[:, :4]) > nms_thres
        label_match = detections[0, -1] == detections[:, -1]
        # Indices of boxes with lower confidence scores, large IOUs and matching labels
        invalid = large_overlap & label_match
        weights = detections[invalid, 4:5]
        # Merge overlapping bboxes by order of confidence
        detections[0, :4] = (weights * detections[invalid, :4]).sum(0) / weights.sum()
        keep_boxes += [detections[0]]
        detections = detections[~invalid]
    if keep_boxes:
        output[image_i] = torch.stack(keep_boxes)

`
Maybe there is a bug in the while loop. If 'invalid' is all 0, the while loop will not stop.

[ashuta03]

For me, train.py often stopped at the calculation of IoU in bbox_iou func, so I changed non_max_suppression in utils/utils.py as below

import torchvision
def non_max_suppression(prediction, conf_thres=0.5, nms_thres=0.4):
    """
    Removes detections with lower object confidence score than 'conf_thres' and performs
    Non-Maximum Suppression to further filter detections.
    Returns detections with shape:
        (x1, y1, x2, y2, object_conf, class_score, class_pred)
    """

    # From (center x, center y, width, height) to (x1, y1, x2, y2)
    prediction[..., :4] = xywh2xyxy(prediction[..., :4])
    output = [None for _ in range(len(prediction))]
    for image_i, image_pred in enumerate(prediction):
        # Filter out confidence scores below threshold
        image_pred = image_pred[image_pred[:, 4] >= conf_thres]
        # If none are remaining => process next image
        if not image_pred.size(0):
            continue
        # Object confidence times class confidence
        boxes = image_pred[:, :4]
        scores = image_pred[:, 4] * image_pred[:, 5:].max(1)[0]
        iou_threshold = nms_thres
        class_confs, class_preds = image_pred[:, 5:].max(1, keepdim=True)
        keep = torchvision.ops.nms(boxes, scores, iou_threshold)

        boxes = boxes[keep]
        class_confs = class_confs[keep]
        scores = scores[keep].view_as(class_confs)
        class_preds = class_preds[keep].type(dtype=torch.float)
        output[image_i] = torch.cat([boxes, class_confs, scores, class_preds], dim=1)

    return output

Worked like a charm. Thanks!