Pytorch3d: Silhouette rendering with an unexpected scale problem

[Eckert-ZJB]

When I follow the silhouette rendering offered in tutorials fit_textured_mesh.ipynb. I found that the silhouette renderer can produce a silhouette of an object, while the silhouette does not completely coincide with the edge of the rendered color image. It seems larger than the rendered color image.

1. the rendered color image:
   

1. the 4-th channel of the rendered color image:
   

2. the result produced by silhouette renderer:
   

I assume this is because the blur_radius > 0 in the silhouette renderer:

sigma = 1e-4
raster_settings_soft = RasterizationSettings(
image_size=128,
blur_radius=np.log(1. / 1e-4 - 1.)*sigma,
faces_per_pixel=50,
)
renderer_silhouette = MeshRenderer(
rasterizer=MeshRasterizer(
cameras=camera,
raster_settings=raster_settings_soft
),
shader=SoftSilhouetteShader()
)

But, how can I obtain a rendered silhouette with an accurate scale?

[nikhilaravi]

If you set blur radius to 0.0 then you should get the same scale - if the blur is different between the color and silhouette renderer settings, then the image sizes will be different.

[Eckert-ZJB]

@nikhilaravi But if I set blur radius to 0.0, the pixel value of the rendered silhouette image is not 0 or 1, most of them are around 0.5. What is mean of these value? I assume that the value of the silhouette image should be 0 or 1.

[Eckert-ZJB]

@nikhilaravi is there any solutions to get a rendered silhouette image where the pixel value is 0 or 1?

[wangsen1312]

@Eckert-ZJB Do you have solution for your question, I use post-processing but the gradient easily get to 0.

[towardsautonomy]

Was this problem ever resolved?

[MengXinChengXuYuan]

Well, does anyone know how to get a 0 1 silhouette ? :(

[wangsen1312]

@MengXinChengXuYuan renderer = MeshRenderer(
rasterizer=rasterizer,
shader=SoftSilhouetteShader(blend_params=BlendParams(sigma=0, gamma=0))
)

Change the sigma and gamma to zero will get the 0/1 silhouette. but this will not differentialable.

[gkioxari]

Folks,

The output of the silhouette renderer is in [0, 1] and not in {0, 1}. Remember that we want the silhouette renderer to be differentiable so the blending produces probabilities for each pixel that determine the "occupancy" in image space. This is computed here

https://github.com/facebookresearch/pytorch3d/blob/6fa66f55341fe1fa8c84c2611a6ec57c1d83b4fb/pytorch3d/renderer/blending.py#L88-L109

The sigma value determines the sharpness of the peak in the normal distribution used in the blending function. You can set sigma to something very small and the silhouette will become shaper though still in [0, 1]. If you want a {0, 1} silhouette then you can threshold the alpha channel, but then you have lost differentiability.