Blenderproc: how to use some custom backgrounds and set the number with more than 25 for each background image

[jingweirobot]

Dear Author

1. How could we use some custom backgrounds (under bop challenge)? just replacing images in cctextures is not available.
2. how to set the number of generated images? Such as more than 25 for each scene?

Thanks a lot.

[ideas-man]

@jingweirobot Hi,

1. It will require slight modification of the base BOP examples, but environment maps is what you need.
2. I would suggest making sure you are familiar with camera_sampling example. number_of_samples parameter is what you are looking for.

[themasterlink]

Hey,

if that doesn't answer your question. You ask again, with your config file attached, so that it is easier for us to directly tell you what to change.

Best,
Max

[themasterlink]

I assume this resolved your problem.

[jingweirobot]

@ideas-man @themasterlink
Dear Authors. For the first question, I still can not find a solution. My target is to use a picture of desk top surface as the background. I want to render these ycb-v objects on the desk top surface so that the robot is able to grasp them by a method. Something like your demo video. So could you help me how to change the background using a simple background. I do not capture your ' It will require slight modification of the base BOP examples, but environment maps is what you need.

Thanks a lot in advance.

[themasterlink]

Hey,

what you are referring to is cut & place. Where you render an object and place it on a random background (in your case a desktop surface). However, this will lead to a problem later on transferring your model from simulation to the real world. It would better rendering your model in simulated environments, so that the lighting of the model matches the background and that the shading of your environment matches your model. These details are crucial in generating data, which can easily be used for sim2real transfer.

So what I would recommend is that you either, use random textures as background as done here:
https://github.com/DLR-RM/BlenderProc/tree/master/examples/bop_object_on_surface_sampling

Or that you use full 3D scenes for even better realism:
https://github.com/DLR-RM/BlenderProc/tree/master/examples/shapenet_with_scenenet

Does this help? If you insist on using a fixed background then render the images with an alpha channel and place them randomly on your backgrounds after rendering. That is faster and allows you to randomly change them during training if you wish.

Best,
Max

[jingweirobot]

@themasterlink You are so kind and smart. I will try to use the method you recommended. After that, I will go back to you and also share my experience or problem. Thanks a lot again.

[themasterlink]

No problem ;)

Recommending BlenderProc to your friends is thanks enough ;)

[jingweirobot]

@themasterlink Sure. I believe more and more researcher will use this excellent work. By the way, how can we re-implement the system including the desk, robot and grasp like your demo video? I want to do this in the real work as well. Thanks in advance.

[themasterlink]

Hey,

the scene is from SUNCG, the objects are from BOP and the robot is from us, which we can't make open source. So if you have your robot model, you can use that and just rework the shapenet scenenet example so that the BOP objects are above the table and then let them fall with some physics.

PS: If you don't have the SUNCG set it also works with 3D Front or SceneNet.

[ideas-man]

@jingweirobot thank you for nice words!

[jingweirobot]

@themasterlink @ideas-man
Dear Authors.
I have a few questions to consult you.

1. In RgbRenderer.py and RendererInterface.py, we cannot find a function introducing how to load the background (cctexture files) so that blender renders these objects (ycb-video dataset) and scenarios? 
2. In resources/cctextures, there are many sub-folders including some pictures, does the blenders randomly choose a sub-folder to be a background? How many rendered images for one sub-folder in resources/cctextures, how to set such a value?
3. We are trying to use this method proposed by you to real robot applications. Do we need to use the real camera intrinsic parameters to replace that camera intrinsic parameters in the blenderproc when creating the datasets?
4. To implement this method in real robotic applications, what do we need to do such as using cctextures as backgrounds, replacing the camera intrinsic parameters as well as other configurations, ? 


Thanks a lot in advance.

[themasterlink]

Hey,

could you please move this into a new issue, questions in closed issues are easily overlooked and it makes it easier for others to find your issue.

Best,
Max