Pytorch3d: Choice of texture interpolation method

[Dragoo417]

Choice of texture interpolation method

Motivation

It would be useful to be able to choose the interpolation method used to sample the texture of a mesh. I would like that the visible texture on the mesh matches the texture as closely as possible but it is not possible for now. I believe it is fixed as bilinear for now, which yields blurry patches of colors for low resolution textures.

[nikhilaravi]

@Dragoo417 You could actually implement this yourself. e.g. if you are using the UV textures class you could subclass this to create a custom class and override the sample_textures method to sample the textures in any way you like.

https://github.com/facebookresearch/pytorch3d/blob/cc08c6b28839a096b9488e17f809c1339f66bd72/pytorch3d/renderer/mesh/textures.py#L880-L957

[Dragoo417]

Oh, I didn't think of that. Thank you for pointing it out !

[Dragoo417]

I finally got around to try this and encoutered a problem: I subclassed TexturesUV with MyTexturesUV and overrode the sample_textures() method. I then use an instance of MyTexturesUV to instantiate a Mesh instead of TexturesUV (as the textures parameter in the Mesh constructor) but when rendering sample_textures() from MyTexturesUV does not seem to even get called. Any hint on what might be wrong ?

[bottler]

Can you paste the relevant parts of your code?

[Dragoo417]

My custom class:

class NoInterpolationTexturesUV(TexturesUV):
    def sample_textures(self, fragments, **kwargs) -> torch.Tensor:
        print("hi")
        # same code as in base class

And I use it as such:

tex = NoInterpolationTexturesUV(
    verts_uvs=[verts_uvs],    # Comes from loading an .obj file
    faces_uvs=[faces_uvs],   # Comes from loading an .obj file
    maps=self._adversarial_texture_tensor
)
myMesh = Meshes(
    verts=[verts.to(self._device)], # Comes from loading an .obj file
    faces=[faces.verts_idx.to(self._device)], # Comes from loading an .obj file
    textures=tex
)

Thank you for your swift reply. Let me know if anthing more is needed.

[bottler]

That should work. If you render myMesh you should see "hi".

If you ever call extend() it won't work (this can be fixed). I can't think of any other reason it wouldn't work.

[Dragoo417]

I do actually call extend() later in the pipeline but I didn't think it could affect this. What would be the way to go to fix it in this case ?

Edit: I looked at the code and it looks like I would have to override the extend() method in my custom texture

[bottler]

The easiest thing for now would be to define extend in your own class

class NoInterpolationTexturesUV(TexturesUV):
    def sample_textures(self, fragments, **kwargs)
         ....
    def extend(self,  N: int):
        new_props = self._extend(
            N,
            [
                "maps_padded",
                "verts_uvs_padded",
                "faces_uvs_padded",
                "_num_faces_per_mesh",
            ],
        )
        new_tex = NoInterpolationTexturesUV(
            maps=new_props["maps_padded"],
            faces_uvs=new_props["faces_uvs_padded"],
            verts_uvs=new_props["verts_uvs_padded"],
            padding_mode=self.padding_mode,
            align_corners=self.align_corners,
        )

        new_tex._num_faces_per_mesh = new_props["_num_faces_per_mesh"]
        return new_tex

This is the same as the extend in TexturesUV except I replace TexturesUV( with NoInterpolationTexturesUV(. Ideally we would use something like self.__class__( in TexturesUV to avoid this problem.

[Dragoo417]

This works perfectly, thanks a lot for your time and have a nice day !