Blenderproc: Rotation and translation matrices are not align with ground truth.

[neixlo]

Problem:
When I take the 3D points (.ply file) of an object, rotate and translate them with the given R, t and K of BopWriter and project them onto the image, the projected points are not aligning with the ground truth.

Expected behavior:
The rotated and translated 3D points should align exactly with the ground truth when projected onto the image plane.

Example:

Steps to reproduce:
I used the R, t and K from the BopWriter, the "colors" image from the hdf5Writer and the 3D points from the LineMod .ply files. To project the 3d points to the image plane I used functions similar to the following.

def project_3d_to_2d(3d_points, R, t, K):
        # projects the 3d points onto the 2d image plane
        2d_points = np.matmul(3d_points, R.T) + t.T
        2d_points = np.matmul(3d_points, K.T)
        2d_points = 2d_points[:, :2] / 2d_points[:, 2]
        2d_points = np.array(2d_points.astype(np.float32))
        return 2d_points

def visualize_pose(img, 2d_points):
        # visualizes the 2d projection in the image
        plt.imshow(img)
        plt.plot(pts_2d[:, 0], pts_2d[:, 1], '.')
        plt.show()

The config.yaml is a slightly modified version from an old example of bop_object_physics_positioning and looks like this.

# Args: <path_to_bop_data> <bop_datset_name> <bop_toolkit_path> <path_to_cc_textures> <output_dir>
{
  "version": 3,
  "setup": {
    "blender_install_path": "./blender/",
    "pip": [
      "h5py",
      "scikit-image",
      "pypng==0.0.20",
      "scipy==1.2.2"
    ]
  },
  "modules": [
    {
      "module": "main.Initializer",
      "config": {
        "global": {
          "output_dir": "<args:4>",
          "sys_paths": ["<args:2>"]
        }
      }
    },
    {
      "module": "loader.BopLoader",
      "config": {
        "bop_dataset_path": "<args:0>/lm",
        "model_type": "",
        "mm2m": True,
        "sample_objects": True,
        "num_of_objs_to_sample": 3,
        "obj_instances_limit": 1,
        "add_properties": {
          "cp_physics": True
        },
        "cf_set_shading": "SMOOTH"
      }
    },
    {
      "module": "materials.MaterialManipulator",
      "config": {
        "selector": {
          "provider": "getter.Material",
          "conditions": [
          {
            "name": "bop_lm_vertex_col_material.*"
          }
          ]
        },
        "cf_set_specular": {
          "provider": "sampler.Value",
          "type": "float",
          "min": 0.0,
          "max": 1.0
        },
        "cf_set_roughness": {
          "provider": "sampler.Value",
          "type": "float",
          "min": 0.0,
          "max": 1.0
        }
      }
    },
    {
      "module": "constructor.BasicMeshInitializer",
      "config": {
        "meshes_to_add": [
        {
          "type": "plane",
          "name": "ground_plane0",
          "scale": [2, 2, 1]
        },
        {
          "type": "plane",
          "name": "ground_plane1",
          "scale": [2, 2, 1],
          "location": [0, -2, 2],
          "rotation": [-1.570796, 0, 0] # switch the sign to turn the normals to the outside
        },
        {
          "type": "plane",
          "name": "ground_plane2",
          "scale": [2, 2, 1],
          "location": [0, 2, 2],
          "rotation": [1.570796, 0, 0]
        },
        {
          "type": "plane",
          "name": "ground_plane4",
          "scale": [2, 2, 1],
          "location": [2, 0, 2],
          "rotation": [0, -1.570796, 0]
        },
        {
          "type": "plane",
          "name": "ground_plane5",
          "scale": [2, 2, 1],
          "location": [-2, 0, 2],
          "rotation": [0, 1.570796, 0]
        },
        {
          "type": "plane",
          "name": "light_plane",
          "location": [0, 0, 10],
          "scale": [3, 3, 1]
        }
        ]
      }
    },
    {
      "module": "manipulators.EntityManipulator",
      "config": {
        "selector": {
          "provider": "getter.Entity",
          "conditions": {
            "name": '.*plane.*'
          }
        },
        "cp_physics": False,
        "cp_category_id": 333
      }
    },
    {
      "module": "materials.MaterialManipulator",
      "config": {
        "selector": {
          "provider": "getter.Material",
          "conditions": {
            "name": "light_plane_material"
          }
        },
        "cf_switch_to_emission_shader": {
          "color": {
            "provider": "sampler.Color",
            "min": [0.5, 0.5, 0.5, 1.0],
            "max": [1.0, 1.0, 1.0, 1.0]
          },
          "strength": {
            "provider": "sampler.Value",
            "type": "float",
            "min": 3,
            "max": 6
          }
        }
      }
    },
    {
      "module": "loader.CCMaterialLoader",
      "config": {
        "folder_path": "<args:3>"
      }
    },
    {
      "module": "materials.MaterialRandomizer",
      "config": {
        "randomization_level": 1,
        "mode": "once_for_all",
        "manipulated_objects": {
          "provider": "getter.Entity",
          "conditions": {
            "name": "ground_plane.*"
          }
        },
        "materials_to_replace_with": {
          "provider": "getter.Material",
          "conditions": {
            "cp_is_cc_texture": True
          }
        }
      }
    },
    {
      "module": "object.ObjectPoseSampler",
      "config": {
        "objects_to_sample": {
          "provider": "getter.Entity",
          "conditions": {
            "cp_physics": True
          }
        },
        "pos_sampler": {
          "provider":"sampler.Uniform3d",
          "min": {
            "provider": "sampler.Uniform3d",
            "min": [-0.3, -0.3, 0.2], # [-0.3, -0.3, 0.0]
            "max": [-0.2, -0.2, 0.2] # [-0.2, -0.2, 0.0]
          },
          "max": {
            "provider": "sampler.Uniform3d",
            "min": [0.2, 0.2, 0.4],
            "max": [0.3, 0.3, 0.6]
          }
        },
        "rot_sampler":{
          "provider":"sampler.UniformSO3",
          "around_x": True,
          "around_y": True,
          "around_z": True,
        }
      }
    },
    {
      "module": "object.PhysicsPositioning",
      "config": {
        "min_simulation_time": 3,
        "max_simulation_time": 10,
        "check_object_interval": 1,
        "solver_iters": 25,
        "steps_per_sec": 100,
        "friction": 100.0,
        "linear_damping": 0.99,
        "angular_damping": 0.99,
        "objs_with_box_collision_shape": {
          "provider": "getter.Entity",
          "conditions": {
            "name": "ground_plane.*"
          }
        }
      }
    },
    {
      "module": "lighting.LightSampler",
      "config": {
        "lights": [
        {
          "location": {
            "provider": "sampler.Shell",
            "center": [0, 0, 0],
            "radius_min": 1,
            "radius_max": 1.5,
            "elevation_min": 5,
            "elevation_max": 89,
            "uniform_elevation": True
          },
          "color": {
            "provider": "sampler.Color",
            "min": [0.5, 0.5, 0.5, 1.0],
            "max": [1.0, 1.0, 1.0, 1.0]
          },
          "type": "POINT",
          "energy": 200
        }
        ]
      }
    },
    {
      "module": "camera.CameraSampler",
      "config": {
        "cam_poses": [
        {
          "proximity_checks": {
            "min": 0.3
          },
          "excluded_objs_in_proximity_check":  {
            "provider": "getter.Entity",
            "conditions": {
              "name": "ground_plane.*",
              "type": "MESH"
            }
          },
          "special_objects": ["obj_000006"],
          "resolution_x": 640,
          "resolution_y": 480,
          "cam_K": [572.4114, 0, 325.2611, 0, 573.57043, 242.04899, 0, 0, 1],
          "number_of_samples": 10,
          "location": {
            "provider": "sampler.Shell",
            "center": [0, 0, 0],
            "radius_min": 0.41,
            "radius_max": 1.24,
            "elevation_min": 5,
            "elevation_max": 89,
            "uniform_elevation": True
          },
          "rotation": {
            "format": "look_at",
            "value": {
              "provider": "getter.POI",
              "selector": {
                "provider": "getter.Entity",
                "conditions": {
                  "type": "MESH",
                  "cp_bop_dataset_name": "<args:1>",
                },
                "random_samples": 3
              }
            },
            "inplane_rot": {
              "provider": "sampler.Value",
              "type": "float",
              "min": -0.7854,
              "max": 0.7854
            }
          }
        }
        ]
      }
    },
    {
      "module": "renderer.RgbRenderer",
      "config": {
        "samples": 3,
        "image_type": "JPEG",
        #"render_normals": True,
        #"normal_output_key": "normals",
        "render_depth": True,
        "depth_output_key": "depth"
      }
    },
    {
      "module": "renderer.SegMapRenderer",
      "config": {
        "map_by": "instance"
      }
    },
    {
      "module": "writer.BopWriter_custom",
      "config": {
        "append_to_existing_output": True
        }    
    },
    {
      "module": "writer.BopWriter",
      "config": {
        "append_to_existing_output": False
        }    
    },
    {
      "module": "writer.Hdf5Writer",
      "config": {
        "append_to_existing_output": True
        }
    } 
  ]
}

Some more examples:

[thodan]

Hi @neixlo. Your python code looks good. Are you using the 3D object models downloaded from the BOP website? I've double-checked the ground-truth poses generated using the latest example and everything looks fine -- see below. I would recommend starting with that example.

[MartinSmeyer]

Yes, following your config you do not seem to use the latest BlenderProc version. We added a mandatory dataset parameter to the BopWriter 21 days ago which your config file does not have. Your issue #27 related to camera intrinsics overwriting with the CameraSampler was solved 18 days ago #29. So maybe you are still using an old version of the code where this was not fixed? Just pull again, adapt your configs and try again please. Also to make sure it is not your projection, you can try with the bop_toolkit:

https://github.com/thodan/bop_toolkit/blob/master/scripts/calc_gt_masks.py

[neixlo]

Thanks, I'll try your suggestions.

[neixlo]

Hi @thodan, hi @MartinSmeyer,
thanks for the support! I've now tried your suggestions and here are my findings:

1. I've re-downloaded the lm models from the BOP website with the command line arguments as described.
   --> There was no difference compared to the models I had (just double checked). But a very good hint, because there quite a few different versions of the lm dataset in the www.

2. I was aware of #27 and #29 and had the changes in my code. However, it's right that I mixed the old config.yaml with the new parameters. So I did a new clean clone of the BlenderProc and BOPtoolkit repo yesterday.
   --> The latest example is working for me, with my visualization and the ones of the bop toolkit.

I modified the config_lm_upright.yaml of the latest example to not beeing in need to download the ycbv
and tyol dataset. So I deleted them form loader.BopLoader and materials.MaterialManipulator in the config.yaml. I also changed the "number_of_samples" to 10 in the camera.CameraSampler.

Here the bop toolkit visualization is looking good.

So far, so good.
Now i did some more testing to get to the point where my code failed.

I tested different version of the modified config_lm_upright.yaml, which differs in the object positioning.

With the modification of rotation parameters, everything looks fine:

      "module": "object.OnSurfaceSampler",
      "config": {
        ...
        "rot_sampler": {
          "provider": "sampler.Uniform3d",
          "min": [0, 0, 0],
          "max": [6.28, 6.28, 6.28]
        }
      }

When I add an PhysicsPositioning module, the ground truth start to differ.

      "module": "object.PhysicsPositioning",
      "config": {
        "min_simulation_time": 3,
        "max_simulation_time": 10,
        "check_object_interval": 1,
        "solver_iters": 25,
        "steps_per_sec": 300,
        "collision_margin": 0.0005,
        "friction": 100.0,
        "linear_damping": 0.99,
        "angular_damping": 0.99,
        "objs_with_box_collision_shape": {
          "provider": "getter.Entity",
          "conditions": {
            "name": "ground_plane.*"
          }
        }
      }

One should maybe not use the OnSurfaceSampler with and PhysicsPositioning afterwards.
So I deleted that one and replaced it through an ObjectPoseSampler and and PhysicsPositioning module with the same parameters like in the /bop_challenge/config_tless_random.yaml.
Unfortunately with the same slight misalignment of the ground truth.

{
      "module": "object.ObjectPoseSampler",
      "config": {
        "objects_to_sample": {
          "provider": "getter.Entity",
          "conditions": {
            "cp_physics": True
          }
        },
        "pos_sampler": {
          "provider":"sampler.Uniform3d",
          "min": {
            "provider": "sampler.Uniform3d",
            "min": [-0.3, -0.3, 0.0],
            "max": [-0.2, -0.2, 0.0]
          },
          "max": {
            "provider": "sampler.Uniform3d",
            "min": [0.2, 0.2, 0.4],
            "max": [0.3, 0.3, 0.6]
          }
        },
        "rot_sampler":{
          "provider":"sampler.UniformSO3"
        }
      }
    },
    {
      "module": "object.PhysicsPositioning",
      "config": {
        "min_simulation_time": 3,
        "max_simulation_time": 10,
        "check_object_interval": 1,
        "solver_iters": 25,
        "steps_per_sec": 100,
        "friction": 100.0,
        "linear_damping": 0.99,
        "angular_damping": 0.99,
        "objs_with_box_collision_shape": {
          "provider": "getter.Entity",
          "conditions": {
            "name": "ground_plane.*"
          }
        }
      }
    },

Running the ObjectPoseSampler without the PhysicsPositioning is working good.
So I think there might be a bug in the PhysicsPositioning module.

      "module": "object.ObjectPoseSampler",
      "config": {
        "objects_to_sample": {
          "provider": "getter.Entity",
          "conditions": {
            "cp_physics": True
          }
        },
        "pos_sampler": {
          "provider":"sampler.Uniform3d",
          "min": {
            "provider": "sampler.Uniform3d",
            "min": [-0.3, -0.3, 0.0],
            "max": [-0.2, -0.2, 0.0]
          },
          "max": {
            "provider": "sampler.Uniform3d",
            "min": [0.2, 0.2, 0.4],
            "max": [0.3, 0.3, 0.6]
          }
        },
        "rot_sampler":{
          "provider":"sampler.UniformSO3"
        }
      }

@MartinSmeyer, @thodan:
Do you have any ideas on what's going on?
I would like to do a random positioning with a physics simulation.

[neixlo]

Another way to reproduce this, is just running the bop_object_physics_positioning example.

I just changed the "image_type": "JPEG" to PNG in the renderer to have the images in the right format for the bop toolkit visualization.

What I also saw a lot during testing is that some object ground truth is clearly misaligned by a certain amount and some seem to be really good. In the example above the iron, hole puncher and bowl ground truth are aligned from all image perspectives, while all other have misalignments.

[MartinSmeyer]

Thank you @neixlo, we will investigate.

[MartinSmeyer]

Hi @neixlo,
we have fixed the issue of slight misalignments, thanks again for finding it. Unfortunately, Blender physics requires to mess with the object origins to work properly, and the reassignment of the original origins contained a bug. We will update the prerendered YCBV dataset and inform challenge participants in the next two days.