Pytorch3d: How to propagate from renderer into meshes?

[umyta]

❓ Questions on how to use PyTorch3D

Hi, for the differentiable renderer, it seems like we can propagate error from images back to camera positions through R and T transformations when calling self.renderer(meshes_world=meshes, R=R, T=T). I was wondering if there is a way to propagate to Meshes?

I did a small experiment that is built on top of the deform tutorial and the differentiable rendering tutorial, but it wasn't successful.

In the forward function, I pass the new_meshes to the renderer:

deform_verts = torch.full(src_mesh.verts_packed().shape, 0.0, device=device, requires_grad=True)
...
new_src_mesh = src_mesh.offset_verts(deform_verts)
image = self.renderer(meshes_world=new_src_mesh, R=R, T=T)

After building the model, I could not find the deform_verts in the params list.

I'm relatively new to differentiable rendering. Any suggestions would be much appreciated. Thanks!

[gkioxari]

Hi @umyta! Nice to see you on this repo :)

There should be information back propagated into the deform_verts. Check deform_verts.grad after you do call backward. I think your confusion stems from the fact that we don't have a model that we are optimizing here. The optimizer accepts deform_verts as the only tensor to be updated after the optimization steps and that happens via deform_verts.grad which contain the gradients wrt gradient_verts.

[umyta]

Hi @gkioxari ! Thanks for the explanation and for helping me out last time. I realize that I did not clarify the loss function I used. In this small experiment, I want to see if we can get any information backpropagated into deform_verts from the rendered image loss alone, so I didn't use any loss from the deform tutorial, only this loss function:

loss = torch.sum((image[..., 3] - self.image_ref) ** 2)

And the optimizer is the one in the differentiable renderer

optimizer = torch.optim.Adam(model.parameters(), lr=0.05)

I was looking for deform_verts in the model.parameters() list, and didn't find it. That's why I was wondering if any of the loss from the image is getting backpropagated to the deform_verts.

I do see that deform_verts.grad is changing. I just want to verify if I understand it correctly. Even it's not explicitly sent to the optimizer, there is still information back-propagated from the image to the deform_verts? I'm a little confused because if I just want to optimize deform_verts through images and set camera positions to be fixed position, the optimizer would have nothing to optimize against in this case. Or do you mean I would need to put deform_verts into the list explicitly like this?

optimizer = torch.optim.Adam(model.deform_verts, lr=0.05)

updated: Oh, wait. This works! Thank you!

[magicknight]

Hi @umyta,

I am also trying to back-propagating to deform the mesh with one single image, did you get it work? My loss keep goes up and to nan, just wondering if you have the same experiecnce ?

Thanks, :D

[umyta]

Hi @umyta,

I am also trying to back-propagating to deform the mesh with one single image, did you get it work? My loss keep goes up and to nan, just wondering if you have the same experiecnce ?

Thanks, :D

For my experiments with one sample image, it did work. I have not yet tested on a batch.

[magicknight]

Hi @umyta,
I am also trying to back-propagating to deform the mesh with one single image, did you get it work? My loss keep goes up and to nan, just wondering if you have the same experiecnce ?
Thanks, :D

For my experiments with one sample image, it did work. I have not yet tested on a batch.

Thank you for your reply!

Yes I am using only one single image too. However the loss keeps going up.

I am writing my model and optimizer based on the deform tutorial and the differentiable rendering tutorial too:

class Model(nn.Module):
    def __init__(self, mesh, tracer, image_ref):
        super().__init__()
        self.mesh = mesh
        self.new_mesh = mesh
        self.device = tracer.device
        self.tracer = tracer

        # Get the reference image
        image_ref = defect_image
        self.register_buffer('image_ref', image_ref)

        # Create an optimizable parameter mesh vertices
#         self.vertices = nn.Parameter(self.mesh.verts_packed()).to(self.device)

        # We will learn to deform the source mesh by offsetting its vertices
        # The shape of the deform parameters is equal to the total number of vertices in src_mesh
        self.deform_verts = torch.full(self.mesh.verts_packed().shape, 0.0, device=self.device, requires_grad=True)

    def forward(self):
        self.new_mesh = self.mesh.offset_verts(self.deform_verts)

        # Based on the new position of the vertices we update mesh, then make new projection using the updated mesh      
        image = renderer(self.new_mesh, lights=lights, materials=materials, cameras=cameras)

        # Calculate the loss
        loss = torch.sum((image - self.image_ref) ** 2)
        return loss, image


# Initialize a model using the renderer, mesh and reference image
model = Model(mesh=cow_mesh, tracer=tracer, image_ref=reference_proj).to(device)

# Create an optimizer. Here we are using Adam and we pass in the vertices of the model
optimizer = torch.optim.Adam([model.deform_verts], lr=0.05)