Yolact: how to blur the bAckgorund in image/video

[sureshgorakala]

Hi Daniel @dbolya , first of all thanks for the great work.
I'm trying to use the model to achieve background blur and just apply mask on the detected objects.
Since, I'm very new to this field.Can you please guide where and how the changes needs to be done?

[jasonkena]

Hey @sureshgorakala, I'm not sure I understand what you are trying to achieve. So, given a normal input image, you are trying to blur the background pixels and leave the pixels with objects intact?

[sureshgorakala]

Yes @dbolya, you are right. I want to blur background pixels and leave the pixels with object as it is.

[Auth0rM0rgan]

@sureshgorakala, This is what you have to do, if I understand correctly.

[Auth0rM0rgan]

@sureshgorakala, forgot to mention the repo https://github.com/jagin/detectron2-pipeline/blob/master/pipeline/separate_background.py

[sureshgorakala]

Thanks for the response. @Auth0rM0rgan Morgan, will check

[sureshgorakala]

Hi @Auth0rM0rgan , Morgan, I have made below changes in def prep_display() to achieve background blur as suggested from Detectron2-pipeline.

mask = masks.cpu().sum(0) >= 1
mask = mask.numpy().astype("uint8")*255
mask = np.stack([mask, mask, mask], axis=2)
foreground = (img_gpu * 255).byte().cpu().numpy()
background = cv2.GaussianBlur(foreground,(21,21), 0)
foreground = foreground.astype(float)
background = background.astype(float)
mask = mask.astype(float)/255.0
foreground2 = cv2.multiply(foreground, mask)
background2 = cv2.multiply(background, 1.0 - mask)
img_dist = cv2.add(foreground2, background2)
img_numpy = img_dist.astype("uint8")

It seems working with image & Video as well. The final FPS seems to be (less than) < 10FPS when processing a video, (though much better than Mask-RCNN). Is there any way to make this run fast? Ultimately I want to run the background blur in realtime live video processing. Any suggestions please. (I run the model in Google Colab)

[jasonkena]

It seems that you are performing these masking operations with CPU.
You should be able to get a significant improvement by performing all the calculations on GPU.

For the Gaussian blur, I recommend you to use Kornia's Gaussian Blur, since it has a PyTorch backend.
You can then replace all the OpenCV and Numpy operations to PyTorch operations, and reserve OpenCV for the final image displaying.

[sureshgorakala]

Hi @jasonkena thanks for the suggestion, I made changes as mentioned. It worked. there is improvement in the FPS on google colab from 9FPS to 19FPS. Is there any additional suggestions to improve the FPS further. Below are the changes made:

foreground = img_gpu.permute(2, 0, 1).unsqueeze(0)
gauss = kornia.filters.GaussianBlur2d((11, 11), (10.5, 10.5))
background = gauss((foreground* 255).float())
mask = mask.astype(float)/255.0
foreground2 = torch.mul(foreground.squeeze(0).permute(1,2,0).float()*255 ,torch.Tensor(mask).cuda())
background2 = torch.mul(background.squeeze(0).permute(1,2,0).float() ,1-torch.Tensor(mask).cuda())
img_dist = torch.add(foreground2, background2)
img_numpy = img_dist.byte().cpu().numpy().astype("uint8")

[jasonkena]

1. If you change foreground = img_gpu.permute(2, 0, 1).unsqueeze(0) into foreground = img_gpu.permute(2, 0, 1).unsqueeze(0).float(), you can remove all the .float() casts in your code. Saving you from some redundant casts.
2. If mask is already a PyTorch tensor, you don't have to call torch.Tensor(mask), just use mask, saving you some constructor calls. If it isn't already a PyTorch tensor, you can just replace mask = mask.astype(float)/255.0 with mask = torch.Tensor(mask,dtype=torch.float).cuda()/255.0

Although these changes won't lead to drastic FPS improvements.

[sureshgorakala]

All the suggested changes are made. Thanks for the wonderful support.

[Kodekat84]

All the suggested changes are made. Thanks for the wonderful support.

Are you sharing your results anywhere? I'd love to walk through your process.

T.I.A.