Yolact: FPS

[Rm1n90]

Hey @dbolya,

I have trained the yolact++ with ResNet101 Backbone and max_size=700. The maximum FPS that I can achieve is 12.64 on full HD video (1920x1080).

In my project, I just need to apply the pretrained model to a part of the frame, not the whole frame so I have changed the get_next_frame function from

    def get_next_frame(vid):
        frames = []
        for idx in range(args.video_multiframe):
            frame = vid.read()[1]
            if frame is None:
                return frames
            frames.append(frame)
        return frames

to the below function to consider only part of the input video

    def get_next_frame(vid):
        frames = []
        for idx in range(args.video_multiframe):
            _, frame = vid.read()
            frame = frame [500:900, 500:1088]
            if frame is None:
                return frames
            frames.append(frame)
        return frames

But instead of getting higher FPS (since I'm feeding a part of the video), I'm getting lower FPS (11.18) than feeding the whole frame to the model. Do you have any idea why I'm getting lower FPS when feeding part of the video?

Thanks in advance!

[dbolya]

Yup, that's because you're doing the "cropping" on the CPU. You want to avoid any CPU operation like the plague because they take so incredibly long. Instead, you should do the cropping after the image has been moved to the GPU:

    def transform_frame(frames):
        with torch.no_grad():
            frames = [torch.from_numpy(frame).cuda().float()[500:900, 500:1088] for frame in frames]
            return frames, transform(torch.stack(frames, 0))

Side note: the masks will be resized back to the original 1080p image size, so perhaps this is not exactly what you want.