Pytorch3d: How can I do a faster rendering with a big mesh?

Created on 3 Oct 2020 · 3Comments · Source: facebookresearch/pytorch3d

We try to change the renderer in the paper 3D Photography using Context-aware Layered Depth Inpainting(another project of facebook) to pytorch3d.
The mesh contains about 1 million verts and 4 million faces. It takes about one second to render an image. However It may take less than 0.1 second with Vispy. Is it because of my program settings? The relevant procedures are as follows：

def render_try():

class myShader(torch.nn.Module):

    def __init__(
        self, device="cpu", cameras=None, blend_params=None
    ):
        super().__init__()
        self.cameras = cameras
        self.blend_params = blend_params if blend_params is not None else BlendParams()

    def forward(self, fragments, meshes, **kwargs) -> torch.Tensor:
        cameras = kwargs.get("cameras", self.cameras)
        if cameras is None:
            msg = "Cameras must be specified either at initialization \
                or in the forward pass of TexturedSoftPhongShader"
            raise ValueError(msg)
        # get renderer output
        blend_params = kwargs.get("blend_params", self.blend_params)
        texels = meshes.sample_textures(fragments)
        images = softmax_rgb_blend(texels, fragments, blend_params)

        return images

R, T = look_at_view_transform(0.01, 180, 0)
R1 = torch.eye(3).unsqueeze(0)
R1[0,1,1] =-1
T1 = torch.zeros_like(T)
verts = np.load("./mesh/4029_verts.npy")
colors = np.load("./mesh/4029_colors.npy")
faces  = np.load("./mesh/4029_faces.npy")

verts = torch.tensor(verts.astype(np.float32))
colors = torch.tensor(colors[:,:-1].astype(np.float32)).unsqueeze(0)
faces = torch.tensor(faces)
textures = TexturesVertex(verts_features=colors.to(device))
hFov = 2 * np.arctan((1. / 2.) * (480 / 640))
vFov = 2 * np.arctan((1. / 2.) * (640 / 640))
fov_in_rad = max(vFov, hFov)
fov = (fov_in_rad * 180 / np.pi)
img_h_len = 640
img_w_len = 480
cur_H = 640
cur_W = 640
mesh = Meshes(
    verts = [verts.to(device)],
    faces = [faces.to(device)],
    textures = textures
)

cameras = FoVPerspectiveCameras(device=device,fov=fov)
#blend_params = BlendParams(sigma=1e-4, gamma=1e-4)
raster_settings = RasterizationSettings(
    image_size=640,
    blur_radius=0.0,
    faces_per_pixel=1,
)

renderer = MeshRenderer(
    rasterizer=MeshRasterizer(
        cameras=cameras,
        raster_settings=raster_settings
    ),
    shader=myShader(
        device=device,
        cameras=cameras,
         )
)
R1 = R1.to(device)
T1 = T1.to(device)
a = time.time()
img = renderer(mesh,R=R1,T=T1)
print(time.time()-a)

how to

Source

TSKongLingwei

All 3 comments

I don't see anything obvious in your settings that would give easy speedup. You might try explicitly setting bin_size in your RasterizationSettings; if it's not specified we set it heuristically based on the size of the image, but you might get a small speedup by tuning it for your task.

We did not do much testing with meshes of this size while developing our renderer, so it's very likely that there are CUDA-level improvements we can make to the rasterizer for meshes of this size. However no matter how much additional optimization we do, I don't expect we will be able to match the speed of Vispy. The reason is that Vispy relies on OpenGL under the hood, which is able to make use of the specialized rasterization hardware on the GPU (for e.g. edge and plane equations, clipping, etc); in contrast PyTorch3D implements a rasterizer purely in software, which runs on the generic CUDA cores of the GPU. For this reason we should always expect a significant performance gap between OpenGL rendering and pure software rendering. A 2011 paper from NVIDIA found a ~2x-8x performance gap between their software rasterizer (which is likely more well-optimized than the PyTorch3D rasterizer) and OpenGL.

In short, if you care only about raw rendering speed, you are probably better off sticking with Vispy. The main advantages of PyTorch3D would be (1) computing gradients; (2) running in contexts where you can't use OpenGL.