Blenderproc: Background issue (cut & place) and how to use this in a practical scenario?

[jingweirobot]

Dear Authors.
I have a few questions to consult you about YCB-Video dataset.

1. In RgbRenderer.py and RendererInterface.py, we cannot find a function introducing how to load the background (cctexture files) so that blender renders these objects (ycb-video dataset) and scenarios? 
2. In resources/cctextures, there are many sub-folders including some pictures, does the blenders randomly choose a sub-folder to be a background? How many rendered images for one sub-folder in resources/cctextures, how to set such a value?
3. We are trying to use this method proposed by you to real robot applications. Do we need to use the real camera intrinsic parameters to replace that camera intrinsic parameters in the blenderproc when creating the datasets?
4. To implement this method in real robotic applications, what do we need to do such as using cctextures as backgrounds, replacing the camera intrinsic parameters as well as other configurations, ? 


Thanks a lot in advance.

[themasterlink]

Hey,

first the YCB-V dataset is supported via the bop dataset, which includes the YCB-V dataset.

In RgbRenderer.py and RendererInterface.py, we cannot find a function introducing how to load the background (cctexture files) so that blender renders these objects (ycb-video dataset) and scenarios?

Right now, we do not offer a script for the cut & placing of content on random images. But, for such scenarios we usually recommend not doing this, as the results are better if instead of cut & place you render a full scene, like in this example: https://github.com/DLR-RM/BlenderProc/blob/master/examples/bop_challenge/config_ycbv_random.yaml

If you have questions on how to render with an alpha background see #73.

1. In resources/cctextures, there are many sub-folders including some pictures, does the blenders randomly choose a sub-folder to be a background?

Blender does not choose randomly BlenderProc does. But that depends on how you configure your EntityManipulator:

https://github.com/DLR-RM/BlenderProc/blob/1e5fd9f85facbbd91f5514d918f8a5eeb0b1b0e5/examples/bop_challenge/config_ycbv_random.yaml#L223-L242

As you can see here all cctextures are used, if you filter them further by name you can only get the ones with wood as done here:
https://github.com/DLR-RM/BlenderProc/blob/1e5fd9f85facbbd91f5514d918f8a5eeb0b1b0e5/examples/suncg_with_improved_mat/config.yaml#L116-L124

How many rendered images for one sub-folder in resources/cctextures, how to set such a value?

As many as you want, there is probably no upper limit to these, as the positions of the objects will be random. But the more the better and at one point you will run out of disc space ;)

1. We are trying to use this method proposed by you to real robot applications. Do we need to use the real camera intrinsic parameters to replace that camera intrinsic parameters in the blenderproc when creating the datasets?

Yes, you should do that. If you have a K matrix please use it.

1. To implement this method in real robotic applications, what do we need to do such as using cctextures as backgrounds, replacing the camera intrinsic parameters as well as other configurations, ?

As always to bridge the gap to real world, you have to randomize as hard core as possible, of course during training I would also advise to use strong augmentations, so that the network has it easier to generalize to the real world.

Best,
Max

[jingweirobot]

@themasterlink @ideas-man

Thanks a lot for your reply.

First question: I mean I used the default configuration from yours. We want to see how it works about getting backgrounds of resources/cctextures.
Second question: how many images generated by blenderproc for each sub-folder in resources/cctextures? How to set this value?

Another question: Actually, we generated 3000 images(three sub folder 000000,000001,000002) using blenderproc. But found that the blenderproc system used same background from cctextures. We think if the blenderproc does not choose background randomly from the cctextures, it may be difficult to be generalized to practical scenarios. Actually, this issue is related to the above-mentioned questions.

The desired way is that the blenderproc system is able to choose randomly any background from cctextures folder. For example, for each background pictures, it generates a certain number of images. We can set this value. Something like this. I am not sure whether you capture my idea or not.

Thanks in advance.

[jingweirobot]

@themasterlink @ideas-man
Could you give me some guides about these questions? Thanks a lot.

[themasterlink]

Hey,

I mean I used the default configuration from yours. We want to see how it works about getting backgrounds of resources/cctextures.

Which one is the default config? You should use this config: https://github.com/DLR-RM/BlenderProc/blob/1e5fd9f85facbbd91f5514d918f8a5eeb0b1b0e5/examples/bop_challenge/config_ycbv_random.yaml

And remove all the BopLoader parts, then use your own Loader to load your object and then generate as many images as possible.

how many images generated by blenderproc for each sub-folder in resources/cctextures? How to set this value?

There is no such value, generate as many as you can store and use during training. Maybe a 100.000 depending on how hard the problem is, maybe less or maybe more. 3000 is probably not enough, but I haven't seen the model or the task, so I can't tell you. You have to try it out.

But found that the blenderproc system used same background from cctextures.

Please share your config file, so that we can check what happened here.

Best,
Max

[jingweirobot]

@themasterlink Thanks a lot for your reply. The following is my yaml file.
I ran this blendproc several times. For each time, it just used the backgrounds in the same sub-folder in cctextures, regardless of how many rendered images. I think, normally, for each rendered image, the blenderproc should choose background images randomly from any sub-folder in cctextures. But when I ran this again, the system chose background images in another sub-folder in cctextures. So I want to check the principle of loading the background images to see why.

Thanks in advance.

# Args: <path_to_bop_data> <bop_datset_name> <bop_toolkit_path> <path_to_cc_textures> <output_dir>
{
  "version": 3,
  "setup": {
    "blender_install_path": "/home_local/<env:USER>/blender/",
    "pip": [
      "h5py",
      "scikit-image",
      "pypng==0.0.20",
      "scipy==1.2.2",
      "matplotlib==2.2.4"
    ]
  },
  "modules": [

    {
      "module": "main.Initializer",
      "config": {
        "global": {
          "output_dir": "<args:4>",    ### /home/jwei/Downloads/bop_ycb_dataset/gen
          "sys_paths": ["<args:2>"]    ### /home/jwei/bop_toolkit 
        }
      }
    },


    {
      "module": "loader.BopLoader",
      "config": {
        "bop_dataset_path": "<args:0>/<args:1>", ### ###ycbv  tless    /home/jwei/Downloads/bop_ycb_dataset
        "model_type": "",
        "mm2m": True,
        "sample_objects": True,
        "num_of_objs_to_sample": 21,
        "obj_instances_limit": 1,   ### original 1
        "add_properties": {
          "cp_physics": True
        },
        "cf_set_shading": "SMOOTH"
      }
    },
    {
      "module": "manipulators.MaterialManipulator",
      "config": {
        "selector": {
          "provider": "getter.Material",
          "conditions": [
          {
            "name": "bop_ycbv_vertex_col_material.*"     ### "name": "bop_tless_vertex_col_material.*"
          }
          ]
        },
        "cf_set_base_color": {
          "provider": "sampler.Color",
          "grey": True,
          "min": [0.1, 0.1, 0.1, 1.0],
          "max": [0.9, 0.9, 0.9, 1.0]
        }
      }
    },
    {
      "module": "manipulators.MaterialManipulator",
      "config": {
        "selector": {
          "provider": "getter.Material",
          "conditions": [
          {
            "name": "bop_ycbv_vertex_col_material.*"   ### "name": "bop_tless_vertex_col_material.*"
          },
          {
            "name": "bop_<args:1>_vertex_col_material.*"
          }
          ]
        },
        "cf_set_specular": {
          "provider": "sampler.Value",
          "type": "float",
          "min": 0.0,
          "max": 1.0
        },
        "cf_set_roughness": {
          "provider": "sampler.Value",
          "type": "float",
          "min": 0.0,
          "max": 1.0
        }
      }
    },
    {
      "module": "constructor.BasicMeshInitializer",
      "config": {
        "meshes_to_add": [
        {
          "type": "plane",
          "name": "ground_plane0",
          "scale": [2, 2, 1]
        },
        {
          "type": "plane",
          "name": "ground_plane1",
          "scale": [2, 2, 1],
          "location": [0, -2, 2],
          "rotation": [-1.570796, 0, 0] # switch the sign to turn the normals to the outside
        },
        {
          "type": "plane",
          "name": "ground_plane2",
          "scale": [2, 2, 1],
          "location": [0, 2, 2],
          "rotation": [1.570796, 0, 0]
        },
        {
          "type": "plane",
          "name": "ground_plane4",
          "scale": [2, 2, 1],
          "location": [2, 0, 2],
          "rotation": [0, -1.570796, 0]
        },
        {
          "type": "plane",
          "name": "ground_plane5",
          "scale": [2, 2, 1],
          "location": [-2, 0, 2],
          "rotation": [0, 1.570796, 0]
        },
        {
          "type": "plane",
          "name": "light_plane",
          "location": [0, 0, 10],
          "scale": [3, 3, 1]
        }
        ]
      }
    },
    {
      "module": "manipulators.EntityManipulator",
      "config": {
        "selector": {
          "provider": "getter.Entity",
          "conditions": {
            "name": '.*plane.*'
          }
        },
        "cp_physics": False,
        "cp_category_id": 333
      }
    },
    {
      "module": "manipulators.MaterialManipulator",
      "config": {
        "selector": {
          "provider": "getter.Material",
          "conditions": {
            "name": "light_plane_material"
          }
        },
        "cf_switch_to_emission_shader": {
          "color": {
            "provider": "sampler.Color",
            "min": [0.5, 0.5, 0.5, 1.0],
            "max": [1.0, 1.0, 1.0, 1.0]
          },
          "strength": {
            "provider": "sampler.Value",
            "type": "float",
            "min": 3,
            "max": 6
          }
        }
      }
    },
    {
      "module": "loader.CCMaterialLoader",
      "config": {
        "folder_path": "<args:3>"
      }
    },
    {
      "module": "manipulators.EntityManipulator",
      "config": {
        "selector": {
          "provider": "getter.Entity",
          "conditions": {
            "name": "ground_plane.*"
          }
        },
        "mode": "once_for_all",
        "cf_randomize_materials": {
          "randomization_level": 1,
          "materials_to_replace_with": {
            "provider": "getter.Material",
            "random_samples": 1,
            "conditions": {
              "cp_is_cc_texture": True
            }
          }
        }
      }
    },
    {
      "module": "object.ObjectPoseSampler",
      "config": {
        "objects_to_sample": {
          "provider": "getter.Entity",
          "conditions": {
            "cp_physics": True
          }
        },
        "pos_sampler": {
          "provider":"sampler.Uniform3d",
          "min": {
            "provider": "sampler.Uniform3d",
            "min": [-0.3, -0.3, 0.0],
            "max": [-0.2, -0.2, 0.0]
          },
          "max": {
            "provider": "sampler.Uniform3d",
            "min": [0.2, 0.2, 0.4],
            "max": [0.3, 0.3, 0.6]
          }
        },
        "rot_sampler":{
          "provider":"sampler.UniformSO3"
        }
      }
    },
    {
      "module": "object.PhysicsPositioning",
      "config": {
        "min_simulation_time": 3,
        "max_simulation_time": 10,
        "check_object_interval": 1,
        "solver_iters": 25,
        "substeps_per_frame": 20,
        "friction": 100.0,
        "linear_damping": 0.99,
        "angular_damping": 0.99,
        "objs_with_box_collision_shape": {
          "provider": "getter.Entity",
          "conditions": {
            "name": "ground_plane.*"
          }
        }
      }
    },
    {
      "module": "lighting.LightSampler",
      "config": {
        "lights": [
        {
          "location": {
            "provider": "sampler.Shell",
            "center": [0, 0, 0],
            "radius_min": 1, # now depends on the bottom area of the box
            "radius_max": 1.5, # this one too
            "elevation_min": 5,
            "elevation_max": 89,
            "uniform_elevation": True
          },
          "color": {
            "provider": "sampler.Color",
            "min": [0.5, 0.5, 0.5, 1.0],
            "max": [1.0, 1.0, 1.0, 1.0]
          },
          "type": "POINT",
          "energy": 200
        }
        ]
      }
    },
    {
      "module": "camera.CameraSampler",
      "config": {
        "cam_poses": [
        {
          "proximity_checks": {
            "min": 0.3
          },
          "excluded_objs_in_proximity_check":  {
            "provider": "getter.Entity",
            "conditions": {
              "name": "ground_plane.*",
              "type": "MESH"
            }
          },
          "number_of_samples": 30,     #### original 25
          "location": {
            "provider": "sampler.Shell",
            "center": [0, 0, 0],
            "radius_min": 0.61,          ###  0.61
            "radius_max": 1.2,          ###  1.24
            "elevation_min": 60,       ### 5
            "elevation_max": 89,      ### 89
            "uniform_elevation": True
          },
          "rotation": {
            "format": "look_at",
            "value": {
              "provider": "getter.POI",
              "selector": {
                "provider": "getter.Entity",
                "conditions": {
                  "type": "MESH",
                  "cp_bop_dataset_name": "<args:1>",
                },
                "random_samples": 15
              }
            },
            "inplane_rot": {
              "provider": "sampler.Value",
              "type": "float",
              "min": -3.14159,
              "max": 3.14159
            }
          }
        }
        ]
      }
    },

    {
      "module": "renderer.RgbRenderer",   ### render rgb and depth images
      "config": {
        "samples": 50,
        "render_distance": True,   ### True
        "image_type": "PNG"                                     #### JPEG ---- png
      }
    },
        ### changing file name and format.
    {   #### set the number of images in each sub-folder  the current value 1000
      "module": "writer.BopWriter",                        ### save everything  image, depth, json files
      "config": {
        "dataset": "<args:1>",    ### <bop_datset_name>
        "append_to_existing_output": True,
        "postprocessing_modules": {    
          "distance": [
            {"module": "postprocessing.Dist2Depth"}
          ]
        }
      }
    }
  ]
}

[themasterlink]

Hey,

I hope I understand you correctly. If I do please let me remind you that BlenderProc generates one static scene in each run and in this static scene you sample different camera poses and then you run it again giving you a new scene, with different textures. And you repeat this process until you have enough training images.

That means that if you sample 30 camera poses, all of them will use the same background. I would suggest only sampling 3-10 images and then rerun BlenderProc more often.

Best,
Max

[jingweirobot]

@themasterlink Thanks a lot. I see.
Which script allows to do this? I mean the blenderproc can be ran many times. Do I need to write a shell script or use other method?

[themasterlink]

We don't provide a script for doing this, but we probably should. As you are not the first to ask for this.

Here is a tiny example of it:

for i in range(100):
    cmd = ["python", "run.py", "examples/basic/config.yaml"]
    cmd.append("examples/basic/camera_positions")
    cmd.append("examples/basic/scene.obj")
    cmd.append("examples/basic/output_{}".format(i))
    subprocess.call(cmd)

[themasterlink]

This should close this issue, if have another question, please open a new issue, to make it easier for others to find it.

[jingweirobot]

thanks a lot for your information