Pytorch3d: Render objects consisting of multiple parts with different textures

[Chr1k0]

❓ Questions on how to use PyTorch3d

How to render objects consisting of multiple parts with different textures? The example given: https://github.com/facebookresearch/pytorch3d/blob/master/docs/tutorials/render_textured_meshes.ipynb is quite simple. Only one mesh with one texture map. Can someone please give an concrete example with an object from ShapeNetCore.V2?

http://shapenet.cs.stanford.edu/shapenet/obj-zip/ShapeNetCore.v2/02958343/

Thanks!

[nikhilaravi]

@Chr1k0, currently we do not support loading multiple texture maps per mesh so Shapenet meshes cannot be rendered with texture. We are adding support for initializing textures as a texture atlas instead of a texture image and uv map - after this is added we will provide a demo with shapenet.

[Chr1k0]

Thanks for the info, just wanted to make sure I didn’t missed it.

Am 20.02.2020 um 18:30 schrieb Nikhila Ravi [e-mail]:


@Chr1k0, currently we do not support loading multiple texture maps per mesh so Shapenet meshes cannot be rendered with texture. We are adding support for initializing textures as a texture atlas instead of a texture image and uv map - after this is added we will provide a demo with shapenet.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[nikhilaravi]

I will leave this issue open until we add this feature!

[kondela]

@nikhilaravi Sounds great, is there any time estimate please?

[ryanfarrell]

@nikhilaravi What about simple a different color per part -- is this feasible??

[nikhilaravi]

@kondela apologies for the late reply! We are working on this feature now.

@ryanfarrell you can assign a color per vertex? Would that be sufficient? Or do you want a color per face?

[ryanfarrell]

Hi @nikhilaravi - I think that either would be fine (whatever is easier to implement). We would just like the ability to get a part-wise colored segmentation rather than a figure/ground segmentation. This would, for example, allow left/right disambiguation for symmetric objects (e.g. you could tell the difference between looking at a person facing you vs. looking away by which side the left arm color vs. the right arm color is on). Does that make sense?

I think color-per-face would be ideal of us as we don't need distinct colors for the corners of a given face. But we can probably work with vertices if you find it's easier to implement color-per-vertex instead.

[hanseungwook]

@nikhilaravi How would you go about rendering a ShapeNetCore.V2 object without texture at all then? I've tried arbitrarily setting the texture_image (in the tutorial) to all 1's, but that doesn't work either.

[nikhilaravi]

@ryanfarrell we already have support for vertex colors. You can initialize an array of vertex rgb colors and use this with the Textures class when initializing a mesh e.g.

from pytorch3d.utils import ico_sphere
from pytorch3d.structures import Textures, Meshes

# Initialize an ico sphere
mesh = ico_sphere(3)  
verts, faces = mesh.get_mesh_verts_faces(0)

# Initalize the textures as an RGB color per vertex
textures = Textures(verts_rgb=torch.ones_like(verts))

# Create a Meshes object with textures
mesh = Meshes(verts=[verts], faces=[faces], textures=textures)

We are working on updating the texturing API so we can try to support per face colors as well if possible! :)

@hanseungwook you can initialize vertex rgb colors to ones as shown above to create a white surface for an object loaded from ShapeNetCore.V2. When loading the mesh with load_obj you can set load_textures=False so that the textures from the obj file are not loaded.

[hanseungwook]

Thanks for following up so closely on this!

[hanseungwook]

By the way it seems that you to do the following to make conform verts_rgb into the correct shape of (1, V, 3):

verts_rgb_src = torch.ones_like(verts)[None]

[charleschiu2012]

Looking forward to the new feature!
My scenario is to apply high-resolution pictures as textures on a particular face on different meshes and combine meshes into one. (i.e. Make 6 meshes to form a ball, then apply textures on the outer face of each valve.)
Thanks for your hard work!

[StphTphsn]

Any update on loading ShapeNet models? I tried different things using functions from https://github.com/facebookresearch/pytorch3d/commit/c9267ab7af0f72217a1ee1a0b37941a5c8fdb325 and code from https://github.com/facebookresearch/pytorch3d/issues/79 but no success. Excuse my impatience for this feature and thanks in advance!

[nikhilaravi]

Hi @StphTphsn here is a complete example of how to load the textures as an atlas, create a mesh and and render it: https://github.com/facebookresearch/pytorch3d/blob/master/tests/test_render_meshes.py#L468.

I will close this issue but feel free to reopen if it you have further questions.