Pytorch3d: Big Runtime Error with some particular BlendParams

[michele-arrival]

🐛 Bugs / Unexpected behaviors

Rendering a simple triangle with SilhouetteShader and BlendParams as follows

blend_params = BlendParams(sigma=1e-4, gamma=1e-4)
raster_settings = RasterizationSettings(
    image_size=frame_size,
    blur_radius=np.log(1.0 / 1e-4 - 1.0) * blend_params.sigma,
    faces_per_pixel=2,
    bin_size=1
)

gives a big runtime error. However if bin_size=0, no errors arise.

Instructions To Reproduce the Issue:

1. My environment:

• pytorch version: from source revision 234658901ae1891c71f6aed2dcf617ab1964851d (latest as of today) installed as pip install git+git://github.com/facebookresearch/pytorch3d.git@234658901ae1891c71f6aed2dcf617ab1964851d
• Ubuntu 18.04 kernel 5.3.0-28-generic
• gcc (Ubuntu 7.4.0-1ubuntu1\~18.04.1) 7.4.0
• g++ (Ubuntu 7.4.0-1ubuntu1\~18.04.1) 7.4.0
• Python version: 3.7.5 (3.6.9 tried too) - environment as python3.7 -m venv env
• GPU: RTX 2070, Driver Version: 440.33.01 CUDA Version: 10.2
• output of pip freeze:

fvcore==0.1.dev200218
numpy==1.18.1
Pillow==7.0.0
pkg-resources==0.0.0
portalocker==1.5.2
pytorch3d==0.1
PyYAML==5.3
six==1.14.0
termcolor==1.1.0
torch==1.4.0
torchvision==0.5.0
tqdm==4.43.0
yacs==0.1.6

1. code snippet:

import numpy as np
import torch
from pytorch3d.renderer import (
    look_at_view_transform,
    OpenGLPerspectiveCameras, RasterizationSettings, BlendParams,
    MeshRenderer, MeshRasterizer, SilhouetteShader
)
from pytorch3d.structures.meshes import Meshes

device = torch.device("cuda:0")

vertices = np.array([
    [-0.4, 0, 0],
    [ 0.2, 0, 0],
    [ 0.0, 0.5, 0]], dtype=np.float32)
faces = np.array([[0, 1, 2]], dtype=np.float32)

frame_size = 256

mesh = Meshes(verts=[torch.tensor(vertices, dtype=torch.float32, device=device)],
              faces=[torch.tensor(faces, dtype=torch.float32, device=device)])

R, T = look_at_view_transform(2.7, 45.0, 0.0)

cameras = OpenGLPerspectiveCameras(device=device, R=R, T=T)

blend_params = BlendParams(sigma=1e-4, gamma=1e-4)
raster_settings = RasterizationSettings(
    image_size=frame_size,
    blur_radius=np.log(1.0 / 1e-4 - 1.0) * blend_params.sigma,
    faces_per_pixel=2,
    bin_size=1
)

renderer = MeshRenderer(
    rasterizer=MeshRasterizer(
        cameras=cameras, raster_settings=raster_settings
    ),
    shader=SilhouetteShader(blend_params=blend_params),
)

images = renderer(meshes_world=mesh)

1. Log:

Traceback (most recent call last):
  File "/tmp/diff_renderers/standalone/run_pytorch3d.py", line 45, in <module>
    images = renderer(meshes_world=mesh)
  File "/tmp/diff_renderers/standalone/env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/tmp/diff_renderers/standalone/env/lib/python3.7/site-packages/pytorch3d/renderer/mesh/renderer.py", line 37, in forward
    fragments = self.rasterizer(meshes_world, **kwargs)
  File "/tmp/diff_renderers/standalone/env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/tmp/diff_renderers/standalone/env/lib/python3.7/site-packages/pytorch3d/renderer/mesh/rasterizer.py", line 124, in forward
    perspective_correct=raster_settings.perspective_correct,
  File "/tmp/diff_renderers/standalone/env/lib/python3.7/site-packages/pytorch3d/renderer/mesh/rasterize_meshes.py", line 120, in rasterize_meshes
    perspective_correct,
  File "/tmp/diff_renderers/standalone/env/lib/python3.7/site-packages/pytorch3d/renderer/mesh/rasterize_meshes.py", line 169, in forward
    perspective_correct,
RuntimeError: Got 256; that's too many! (RasterizeMeshesCoarseCuda at /tmp/pip-req-build-ih7k1k4e/pytorch3d/csrc/rasterize_meshes/rasterize_meshes.cu:627)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7f903310a193 in /tmp/diff_renderers/standalone/env/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: RasterizeMeshesCoarseCuda(at::Tensor const&, at::Tensor const&, at::Tensor const&, int, float, int, int) + 0x3c8 (0x7f902b8c6471 in /tmp/diff_renderers/standalone/env/lib/python3.7/site-packages/pytorch3d/_C.cpython-37m-x86_64-linux-gnu.so)
frame #2: RasterizeMeshesCoarse(at::Tensor const&, at::Tensor const&, at::Tensor const&, int, float, int, int) + 0xb2 (0x7f902b87aed2 in /tmp/diff_renderers/standalone/env/lib/python3.7/site-packages/pytorch3d/_C.cpython-37m-x86_64-linux-gnu.so)
frame #3: RasterizeMeshes(at::Tensor const&, at::Tensor const&, at::Tensor const&, int, float, int, int, int, bool) + 0x62 (0x7f902b87b352 in /tmp/diff_renderers/standalone/env/lib/python3.7/site-packages/pytorch3d/_C.cpython-37m-x86_64-linux-gnu.so)
frame #4: <unknown function> + 0x3cc10 (0x7f902b88bc10 in /tmp/diff_renderers/standalone/env/lib/python3.7/site-packages/pytorch3d/_C.cpython-37m-x86_64-linux-gnu.so)
frame #5: <unknown function> + 0x39207 (0x7f902b888207 in /tmp/diff_renderers/standalone/env/lib/python3.7/site-packages/pytorch3d/_C.cpython-37m-x86_64-linux-gnu.so)
frame #6: _PyMethodDef_RawFastCallKeywords + 0x224 (0x5d7f64 in /tmp/diff_renderers/standalone/env/bin/python)
frame #7: /tmp/diff_renderers/standalone/env/bin/python() [0x54a9c0]
frame #8: _PyEval_EvalFrameDefault + 0x43f8 (0x551c08 in /tmp/diff_renderers/standalone/env/bin/python)
frame #9: _PyEval_EvalCodeWithName + 0x252 (0x54b302 in /tmp/diff_renderers/standalone/env/bin/python)
frame #10: _PyFunction_FastCallDict + 0x34e (0x5d9dbe in /tmp/diff_renderers/standalone/env/bin/python)
frame #11: THPFunction_apply(_object*, _object*) + 0xa7f (0x7f903a50c3ef in /tmp/diff_renderers/standalone/env/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #12: _PyMethodDef_RawFastCallKeywords + 0x132 (0x5d7e72 in /tmp/diff_renderers/standalone/env/bin/python)
frame #13: /tmp/diff_renderers/standalone/env/bin/python() [0x54a9c0]
frame #14: _PyEval_EvalFrameDefault + 0x43f8 (0x551c08 in /tmp/diff_renderers/standalone/env/bin/python)
frame #15: _PyEval_EvalCodeWithName + 0x252 (0x54b302 in /tmp/diff_renderers/standalone/env/bin/python)
frame #16: _PyFunction_FastCallKeywords + 0x482 (0x5d8bd2 in /tmp/diff_renderers/standalone/env/bin/python)
frame #17: /tmp/diff_renderers/standalone/env/bin/python() [0x54a880]
frame #18: _PyEval_EvalFrameDefault + 0x13ad (0x54ebbd in /tmp/diff_renderers/standalone/env/bin/python)
frame #19: _PyEval_EvalCodeWithName + 0x252 (0x54b302 in /tmp/diff_renderers/standalone/env/bin/python)
frame #20: _PyFunction_FastCallDict + 0x34e (0x5d9dbe in /tmp/diff_renderers/standalone/env/bin/python)
frame #21: /tmp/diff_renderers/standalone/env/bin/python() [0x4d8102]
frame #22: PyObject_Call + 0x56 (0x5dbbc6 in /tmp/diff_renderers/standalone/env/bin/python)
frame #23: _PyEval_EvalFrameDefault + 0x18d4 (0x54f0e4 in /tmp/diff_renderers/standalone/env/bin/python)
frame #24: _PyEval_EvalCodeWithName + 0x252 (0x54b302 in /tmp/diff_renderers/standalone/env/bin/python)
frame #25: _PyFunction_FastCallDict + 0x34e (0x5d9dbe in /tmp/diff_renderers/standalone/env/bin/python)
frame #26: /tmp/diff_renderers/standalone/env/bin/python() [0x59412b]
frame #27: PyObject_Call + 0x56 (0x5dbbc6 in /tmp/diff_renderers/standalone/env/bin/python)
frame #28: _PyEval_EvalFrameDefault + 0x18d4 (0x54f0e4 in /tmp/diff_renderers/standalone/env/bin/python)
frame #29: _PyEval_EvalCodeWithName + 0x252 (0x54b302 in /tmp/diff_renderers/standalone/env/bin/python)
frame #30: _PyFunction_FastCallDict + 0x34e (0x5d9dbe in /tmp/diff_renderers/standalone/env/bin/python)
frame #31: /tmp/diff_renderers/standalone/env/bin/python() [0x4d8102]
frame #32: PyObject_Call + 0x56 (0x5dbbc6 in /tmp/diff_renderers/standalone/env/bin/python)
frame #33: _PyEval_EvalFrameDefault + 0x18d4 (0x54f0e4 in /tmp/diff_renderers/standalone/env/bin/python)
frame #34: _PyEval_EvalCodeWithName + 0x252 (0x54b302 in /tmp/diff_renderers/standalone/env/bin/python)
frame #35: _PyFunction_FastCallDict + 0x34e (0x5d9dbe in /tmp/diff_renderers/standalone/env/bin/python)
frame #36: /tmp/diff_renderers/standalone/env/bin/python() [0x59412b]
frame #37: _PyObject_FastCallKeywords + 0x4eb (0x5d96db in /tmp/diff_renderers/standalone/env/bin/python)
frame #38: /tmp/diff_renderers/standalone/env/bin/python() [0x54aa51]
frame #39: _PyEval_EvalFrameDefault + 0x13ad (0x54ebbd in /tmp/diff_renderers/standalone/env/bin/python)
frame #40: _PyEval_EvalCodeWithName + 0x252 (0x54b302 in /tmp/diff_renderers/standalone/env/bin/python)
frame #41: PyEval_EvalCode + 0x23 (0x54d803 in /tmp/diff_renderers/standalone/env/bin/python)
frame #42: /tmp/diff_renderers/standalone/env/bin/python() [0x6308e2]
frame #43: PyRun_FileExFlags + 0x97 (0x630997 in /tmp/diff_renderers/standalone/env/bin/python)
frame #44: PyRun_SimpleFileExFlags + 0x17f (0x63160f in /tmp/diff_renderers/standalone/env/bin/python)
frame #45: /tmp/diff_renderers/standalone/env/bin/python() [0x65450e]
frame #46: _Py_UnixMain + 0x2e (0x65486e in /tmp/diff_renderers/standalone/env/bin/python)
frame #47: __libc_start_main + 0xe7 (0x7f903f22db97 in /lib/x86_64-linux-gnu/libc.so.6)
frame #48: _start + 0x2a (0x5df80a in /tmp/diff_renderers/standalone/env/bin/python)


Process finished with exit code 1

Maybe i'm the proverbial idiot of murphy's laws - "If you make something idiot-proof, someone will just make a better idiot.". But I think if the input parameters are invalid, it should be captured before crashing and burning :D

[nikhilaravi]

@michele-arrival this is not because of the blend params but the rasterization settings. Currently your values are:

raster_settings = RasterizationSettings(
    image_size=frame_size,
    blur_radius=np.log(1.0 / 1e-4 - 1.0) * blend_params.sigma,
    faces_per_pixel=2,
    bin_size=1
)

There are two implementations of rasterization in pytorch3d and the approach that is used is determined by the value of the bin_size setting.

The first naive approach (when bin_size = 0) loops through all the faces in the mesh for each pixel. The second approach (when bin_size > 0) involves two steps the first of which is a coarse step where the image is split into a rough grid (or bins) and each face in the mesh is assigned to a grid cell (or bin). The second step loops through each pixel and only looks at the faces in the bin in which the pixel lies. Refer to the docs on the renderer for more details on this 'coarse-to-fine' approach: https://pytorch3d.org/docs/renderer. Also refer to the PyTorch3D API docs for the rasterizer here

For coarse-to-fine rasterization the bin_size setting determines the number of bins and in this case you have set it to 1. There is an upper bound on the number of faces which can be allocated to each bin and as the error mentions Got 256; that's too many! (for reference with, the error is from RasterizeMeshesCoarseCuda to help you debug such issues in future).

If you want to enable the coarse-to-fine rasterization you can just leave the bin_size empty and it will be set based on heuristics. If you want to use the naive rasterization approach you can set bin_size = 0.

[michele-arrival]

Yes sorry of course I meant RasterizationSettings.

Thanks for the explanation, I understand now.
I guess a clearer error message wouldn't be too bad though. For one thing it looks like a low level exception, not a user error, even thought it is. Also It's not immediately clear what *that* in that's too many is.

As a side note, when the image is big (> 1024) leaving bin_size out of the settings means another kind of exception incompatible function arguments later on.
see here where bin_size remains None:
https://github.com/facebookresearch/pytorch3d/blob/dbf06b504b525c7f6680c5709b63df6413616d2e/pytorch3d/renderer/mesh/rasterize_meshes.py#L93-L121

Cheers!

[nikhilaravi]

@michele-arrival, yes we could definitely make the error message clearer. Thanks for pointing out the error for when image size is > 1024. We assumed that most use cases involving batched rendering of meshes would require image sizes < 1024, but we can fix this so that rendering is enabled with any image size. Feel free to submit a PR if you would like to try and fix this yourself!

[michele-arrival]

90

[nikhilaravi]

@michele-arrival your PR has been merged so I am closing this issue.