Blenderproc: Best way for achieving dynamic domain randomisation?

[eugval]

Hello,

First, thank you for putting blenderProc together, it is a great repo!

I would like to use it for the following usecase, I would appreciate any advice on the best way to proceed:

REPEAT (say 10 000 times):

1. Create a scene
2. For say 10 keyframes .
   
   
   
   1. Change the camera pose
   
   
   2. Change poses of certain objects of the scene and do domain randomisation (lights, materials, textures), while keeping other aspects constant
   
   
   3. Render and save the rendering
   
   

From issues #154 and #52 (and also from working through the examples), my understanding is that this is not currently natively supported by BlenderProc, the issue being the "dynamic" domain randomisation on each of the keyframes in step ii.

My question is: 

Would you recommend to :

A. Modify the blenderProc modules slightly to make them rely on independent random generators and then on the external script play set the seeds for these generators so that on each call of blenderProc the generated scenes keep some aspects constant and some aspects randomised?

B. Make a module (or modify existing ones) to save different randomised states (object poses, materials, lights) to keyframes, the same way that camera poses are being saved for different keyframes (Would that even be possible?)?

C. Any better recommended way to proceed for that usecase?

Thanks again, I appreciate any help/tips on this!

[themasterlink]

Hey,

first of all, are you sure you need this? Creating a wider variety of camera poses and scene randomizations usually helps with the training of CNNs.

If you are sure that you need to fix a certain part, because you want to study how certain randomizations affect the performance, then I would suggest going with B. It is super easy to key frame any kind of attribute of an object, here for example, we key frame the location:

https://github.com/DLR-RM/BlenderProc/blob/fe5ea258acb9e3ec11e0d4c6f380321e01c4e751/src/utility/CameraUtility.py#L29

Materials and textures on the other hand are harder to key frame, as blender doesn't support the keyframing of the material assignment. You could still do this by using a mix shader to blend between different materials, however, this is really complicated and I would rather suggest you generate a random collection of scenes, save them as blend and then randomize the things you want to randomize on top of that (light, material, ...).

Best,
Max

[eugval]

Oh great, thank you for the answer!

I essentially need to change the objects loaded in the scene (randomly) every 10 steps, but then do domain randomisation and take renderings from different camera poses at every step.

So, to summarise:

• I can change the location or other attributes of objects (cameras, lights , meshes ) by keyframing as you showed.
• Changing materials is not natively supported by blender, so better to steer clear until fairly familiar with the APIs.
• The best way to proceed for the usecase I set out is probably as you recommended :
  a. Create 10k scenes by running BlenderProc 10k times and saving on disc.
  b. Re-run blenderProc loading each scene and running blenderProc 10 times on it with the needed randomisations to do the rendering.

I think that works, thanks again for the help! :)

[themasterlink]

Hey,

that sounds like a plan. As always if you have finished a certain module, we are always looking for contributions!

I will close this for now.

Best,
Max