Pytorch3d: Code for Experiments: Unsupervised mesh prediction

[vikashranjan]

Is the code for Experiments with Sphere FC, Sphere GCN, and Voxel GCN mentioned in the paper "Accelerating 3D Deep Learning with PyTorch3D" available?

As I am new to 3d deep learning I would like to go through the code to understand Unsupervised mesh prediction.

[gkioxari]

Our plan is to have this soon! In the mean time we have a tutorial on fitting a mesh from image views using differentiable rendering that could be helpful and that don't deviate much from the learning setting (mesh verts are predicted via a neural network instead of being treated as a variable tensor):

https://github.com/facebookresearch/pytorch3d/blob/master/docs/tutorials/fit_textured_mesh.ipynb

[vikashranjan]

Thanks, looking forward to the code release!

[sjcv]

Hi @gkioxari , just wanted to check on the timeline for this. Thanks again for providing it!

[nikhilaravi]

@sjcv we are planning to open sourcing this as a part of a new projects section in PyTorch3D. We will comment on this task when the code is available.

[JobAtom]

@sjcv we are planning to open sourcing this as a part of a new projects section in PyTorch3D. We will comment on this task when the code is available.

In SphereGCN, 3xGraphConv(512 + 3, 512) + ReLU layer, I am wondering what is 512 + 3 dimension input? Is the 3 means the x, y, z position of input vertex? Thank you.

[gkioxari]

Yes!

[stephentandjiria]

Hi, I'm super interested in unsupervised 3D prediction from a 2D image.
However, I am confused as to how to train the following NN

in a way that minimizes all loss functions (rgb, silhouette, edge, normal, laplacian) inside the loop in fit_textured_mesh.ipynb

I noticed that the only two ways in comparing prediction to ground truth are silhouette and RGB.

    for j in np.random.permutation(num_views).tolist()[:num_views_per_iteration]:
        images_predicted = renderer_textured(new_src_mesh, cameras=target_cameras[j], lights=lights)

        # Squared L2 distance between the predicted silhouette and the target 
        # silhouette from our dataset
        predicted_silhouette = images_predicted[..., 3]
        loss_silhouette = ((predicted_silhouette - target_silhouette[j]) ** 2).mean()
        loss["silhouette"] += loss_silhouette / num_views_per_iteration

        # Squared L2 distance between the predicted RGB image and the target 
        # image from our dataset
        predicted_rgb = images_predicted[..., :3]
        loss_rgb = ((predicted_rgb - target_rgb[j]) ** 2).mean()
        loss["rgb"] += loss_rgb / num_views_per_iteration

but I am confused as to how to pull out the silhouette and RGB as outputs from the NN.

or maybe I should pull out the NN verts output and put it into src_mesh? replacing the following line:
src_mesh = ico_sphere(4, device)
and then just continue the deformation loop?