Pytorch3d: Camera Position Optimization with phong_renderer

[Jiayi-Wang-MPI]

I replaced the silhouette_renderer with the defined phong_renderer in the Camera Position Optimization example. Essentially I replaced the following line.

#model = Model(meshes=teapot_mesh, renderer=silhouette_renderer, image_ref=image_ref).to(device)
model = Model(meshes=teapot_mesh, renderer=phong_renderer, image_ref=image_ref).to(device)

With this modification, the optimization provides zero gradient and no longer updates the camera position or loss. Is this expected behavior?

[nikhilaravi]

@Jiayi-Wang-MPI The phong_renderer only uses 1 face for each pixel and applies hard blending i.e uses the only the color of the closest face for each pixel. If you look at the raster_settings you can see the option faces_per_pixel is set to 1. You'll need to make the following changes.

1) Set the rasterization settings to the same values as for the SilhouetteShader

raster_settings = RasterizationSettings(
    image_size=256, 
    blur_radius=np.log(1. / 1e-4 - 1.) * blend_params.sigma, 
    faces_per_pixel=100, 
    bin_size=0
)

2) Define a new Phong shader which uses the softmax_rgb_blend function:

class SoftPhongShader(nn.Module):
    def __init__(
        self,
        device="cpu",
        cameras=None,
        lights=None,
        materials=None,
        blend_params=None,
    ):
        super().__init__()
        self.lights = (
            lights if lights is not None else PointLights(device=device)
        )
        self.materials = (
            materials if materials is not None else Materials(device=device)
        )
        self.cameras = (
            cameras
            if cameras is not None
            else OpenGLPerspectiveCameras(device=device)
        )
        self.blend_params = (
            blend_params if blend_params is not None else BlendParams()
        )

    def forward(self, fragments, meshes, **kwargs) -> torch.Tensor:
        texels = interpolate_vertex_colors(fragments, meshes)
        cameras = kwargs.get("cameras", self.cameras)
        lights = kwargs.get("lights", self.lights)
        materials = kwargs.get("materials", self.materials)
        colors = phong_shading(
            meshes=meshes,
            fragments=fragments,
            texels=texels,
            lights=lights,
            cameras=cameras,
            materials=materials,
        )
        images = softmax_rgb_blend(colors, fragments, self.blend_params)
        return images

3) Initialize the phong_renderer as follows:

phong_renderer = MeshRenderer(
    rasterizer=MeshRasterizer(
        cameras=cameras, 
        raster_settings=raster_settings
    ),
    shader= SoftPhongShader(device=device, lights=lights, blend_params=blend_params)
)

We are making some updates to the Shading API so the above renderer will be available soon to import directly.

[nikhilaravi]

@Jiayi-Wang-MPI the SoftPhongShader is now available to import directly from pytorch3d so you don't need to define it yourself.

[nikhilaravi]

@Jiayi-Wang-MPI were you able to get this working with the new SoftPhongShader or the code above? Can I close this issue?

[Jiayi-Wang-MPI]

@nikhilaravi the new SoftPhongShader works for me. Thanks!