BlenderProc 🚀 - How to import URDF robot model and manipulate it?

Hey @b-yogesh,

I think there are two ways to get BlenderProc to work with Phobos:

There seems to be ongoing effort to make Phobos work with blender 2.9 (https://github.com/dfki-ric/phobos/issues/194#issuecomment-705082430). So maybe you can already make use of their current development and use it together with blender proc
You try to get the current BlenderProc version to work with version 2.79. However, there were quite some API changes when blender 2.8 has been introduced, so you probably need to make small adjustments on a number of places.

For controlling your robot I would advice you to write your own module (its basically just a python class). In this way its easier to implement more advanced behaviour. Also, it seems to me that the loaded robots are controlled via bones, whose manipulation is not yet supported by the entity manipulator.

cornerfarmer on 9 Nov 2020

Hey @cornerfarmer ,

Thank you very much for your prompt reply. I was trying your first suggestion yesterday (Phobos work with blender 2.9). So, I was able to link Phobos + Blender 2.90 + BlenderProc with a few changes in the script of Phobos. I am able to load the URDF file and manipulate using Rotate function of Blender GUI. It all seemed fine until I ran the basic example of BlenderProc.

This is the image from the Blender RenderImage:
blenderrender

And this is the one that BlenderProc generates:

Based on ablation study of the URDF model, when I removed the links of the robot, it gave me the exact image as the BlenderProc.
This is the URDF robot structure:

Check out these logs:

Here, it does not load any Links into BlenderProc which causes all the meshes to collapse to the origin, which I suspect is the issue. Can you please help me out here as how do I make this work?

b-yogesh on 10 Nov 2020

Hey @b-yogesh,

thanks for the detailed description.

The log you added only contains the rgb renderer, I am not sure if links are explicitly listed here.
However, you are probably right, that something in the loading of your robot goes wrong.
Can you please explain how you load your URDF model using BlenderProc. Did you write your own module? Which functions from the Phobos extension do you call?

cornerfarmer on 10 Nov 2020

Hello,

Phobos has a built-in functionality to import URDF models into it. I used that.

Have I made any mistake? Because, while importing from phobos, the links are imported and I am able to manipulate them using rotation. But, the links do not appear in the BlenderProc output 😞

Here are the complete logs:

_Blender 2.90.0 (hash 0330d1af29c0 built 2020-08-31 10:00:13)
Read prefs: C:UsersYogesh KumarAppDataRoamingBlender FoundationBlender2.90configuserpref.blend
found bundled python: J:SaarlandUniversityMasterThesisBlenderProcblender-2.90.0-windows642.90python
Importing phobos
IMPORT: phobos.defs
Parsing definitions from: C:UsersYogesh KumarAppDataRoamingphobos/definitions
defaultControllers.yml
C:UsersYogesh KumarAppDataRoamingBlender FoundationBlender2.90scriptsaddonsphobosdefs.py:201: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
tmpyaml = yaml.load(__evaluateString(tmpstring))
defaultJoints.yml
defaultMaterials.yml
defaultModel.yml
defaultMotors.yml
defaultSensors.yml
defaultSubmechanisms.yml
defaultSubmodels.yml
Creating new definition type: joints
IMPORT: phobos.display
IMPORT: phobos.io
Registered mesh plugin: ['obj', 'stl', 'dae']
ERROR in scenes/__init__: "mars.py" has no valid scene plugin interface.
Registered scene plugin: ['sdf world']
Registered scene plugin: ['smurfs']
Registered entity plugin: ['generic']
ERROR in entities/__init__: "heightmap.py" has no valid entity plugin interface.
Registered entity plugin: ['light']
Registered entity plugin: ['primitive']
Registered entity plugin: ['sdf']
Registered entity plugin: ['smurf']
Registered entity plugin: ['srdf']
Registered entity plugin: ['submechanisms']
Registered entity plugin: ['thumbnails']
Registered entity plugin: ['urdf']
Registered entity plugin: ['yaml']
Registered mesh plugin: ['obj', 'stl', 'dae']
ERROR in scenes/__init__: "mars.py" has no valid scene plugin interface.
Registered scene plugin: ['sdf world']
Registered scene plugin: ['smurfs']
RELOAD: phobos.model
IMPORT: phobos.operators
IMPORT: phobos.phobosgui
RELOAD: phobos.phoboslog
RELOAD: phobos.phobossystem
RELOAD: phobos.utils
Registering operators.selection...
Registering operators.io...
Registering operators.editing...
Registering operators.naming...
Registering operators.misc...

Registering phobosgui...
... successful.
Info: Deleted 3 object(s)
Info: Deleted 3 object(s)

Start - Initializing module main.Initializer

Finished - Initializing module main.Initializer (took 0.014 seconds)

Start - Initializing module loader.BlendLoader

Finished - Initializing module loader.BlendLoader (took 0.004 seconds)

Start - Initializing module lighting.LightLoader

Finished - Initializing module lighting.LightLoader (took 0.011 seconds)

Start - Initializing module camera.CameraLoader

Finished - Initializing module camera.CameraLoader (took 0.003 seconds)

Start - Initializing module renderer.RgbRenderer

Finished - Initializing module renderer.RgbRenderer (took 0.006 seconds)

Start - Initializing module writer.Hdf5Writer

Start - Initializing module postprocessing.TrimRedundantChannels

Finished - Initializing module postprocessing.TrimRedundantChannels (took 0.002 seconds)

Finished - Initializing module writer.Hdf5Writer (took 0.249 seconds)

Start - Running blender pipeline

Start - Running module Initializer

Device GeForce 940MX (Display) of type OPTIX found and used.

Finished - Running module Initializer (took 0.093 seconds)

Start - Running module BlendLoader

Finished - Running module BlendLoader (took 0.591 seconds)

Start - Running module LightLoader

Finished - Running module LightLoader (took 0.000 seconds)

Start - Running module CameraLoader

Finished - Running module CameraLoader (took 0.003 seconds)

Start - Running module RgbRenderer

Finished - Running module RgbRenderer (took 26.805 seconds)

Start - Running module Hdf5Writer

Merging data for frame 0 into J:SaarlandUniversityMasterThesisBlenderProcBlenderProcexamples/basic/output0.hdf5
Key: colors - shape: (720, 1280, 3) - dtype: uint8 - path: C:UsersYOGESH~1AppDataLocalTempblender_proc_81324f8d050e499b8d041559319a78b4rgb_0000.png
Key: distance - shape: (720, 1280) - dtype: float64 - path: C:UsersYOGESH~1AppDataLocalTempblender_proc_81324f8d050e499b8d041559319a78b4distance_0000.exr
Key: normals - shape: (720, 1280, 3) - dtype: float64 - path: C:UsersYOGESH~1AppDataLocalTempblender_proc_81324f8d050e499b8d041559319a78b4normals_0000.exr

Finished - Running module Hdf5Writer (took 3.201 seconds)

Finished - Running blender pipeline (took 30.705 seconds)

Blender quit_

b-yogesh on 10 Nov 2020

So as BlenderProc currently does not know how to apply these link poses, one way would be to write a custom Manipulator that loads the link pose information using

https://github.com/dfki-ric/phobos/blob/cd5357cf3d7fe5bab50ac9b4e4c53fdcc06d7dab/phobos/model/models.py#L160

and then sets these as poses of the visual/inertial meshes.

You could orient yourself at the Entity Manipulator:
https://github.com/DLR-RM/BlenderProc/blob/master/src/manipulators/EntityManipulator.py
https://github.com/DLR-RM/BlenderProc/tree/master/examples/entity_manipulation

MartinSmeyer on 10 Nov 2020

👍2

Hey @b-yogesh,

regarding your loading problem:
thanks for the explanation, I missed the info that you save your loaded robot to a .blend file and then import it via the BlendLoader.
At the moment we are appending the blend file item-by-item which seems to destroy connections between them and which in the end leads to the results you got.
As a temporary workaround you could replace the run function in the BlendLoader (src/loader/BlendLoader.py) with the following code:

    def run(self):
        # get a path to a .blend file
        path = Utility.resolve_path(self.config.get_string("path"))

        with bpy.data.libraries.load(path) as (data_from, data_to):
            data_to.collections = [name for name in data_from.collections]

        for collection in data_to.collections:
            bpy.context.scene.collection.children.link(collection)

This should now load the whole content of your blend file without destroying any links between them. If you then want to further manipulate specific links after loading, you can take a look at what Martin wrote above.

cornerfarmer on 10 Nov 2020

👍2

Thank you very much. It worked like a charm! 😄

I just have a few more follow-up questions and would appreciate if you give me some advise.

1) How do I rotate the arms using python at random different angles for each joint (I tried extracting bpy.data.objects[obj_id] and then using the rotate one joint of the arm by some degrees in some axis, but this does not respect the constraints added via URDF. Although, while using the Rotate tool, the constraints are respected). Am I missing something here (i was told that entity manipulator cannot work with robots yet)?

2) To generate a dataset, do I need to write a loop for run.py ?

3) The output of BlenderProc is a HDF file, can I change the script so that it saves only the images (rgb, seg, depth)?

Thank you

b-yogesh on 10 Nov 2020

Hey,

these are great questions, but to make the issues better searchable, please do open new issues for your questions 2 and 3.

Best,
Max

themasterlink on 10 Nov 2020

Sure, will do!

b-yogesh on 10 Nov 2020

👍1

@b-yogesh Regarding your first question:

I think you need to change the rotation of the respective bone to make use of its constraints. Changing the rotation should look like bpy.data.objects['Back_x_Link'].pose.bones["Bone"].rotation_euler[1] = 0.1.

With Back_x_Link being the name of your armature object and Bone being the name of the bone you want to manipulate.
The rotation you set should now be clipped to the configured limit interval.

cornerfarmer on 10 Nov 2020

You can simply generate a animation in blender (frame by frame) and I hope you can achieve the random effect. Or you can simply use object pose and manipulate the arms of a robot. My recommendation, creating a animation inside blender.

shreeramsigdel77 on 11 Nov 2020

Thank you for your prompt replies and help. I got it working with this bpy.data.objects['Back_x_Link'].pose.bones["Bone"].rotation_euler[1] = 0.1.

b-yogesh on 11 Nov 2020

@b-yogesh Hi, I am trying to import the URDF file with Phobos. But I cannot import the urdf file since it throw some errors when I used the Phobos add-on. Known from your reply that you had a few changes in the script of Phobos. Would you please share more details about the revised part of the Phobos to be compatibility with Blender 2.90?

hetolin on 19 Dec 2020

@b-yogesh Hi, I am trying to import the URDF file with Phobos. But I cannot import the urdf file since it throw some errors when I used the Phobos add-on. Known from your reply that you had a few changes in the script of Phobos. Would you please share more details about the revised part of the Phobos to be compatibility with Blender 2.90?

Hi @hetolin, sorry for the late reply. Yes, so I had used this code from the Phobos older releases for defLayers

def defLayers(layerlist):
    """Returns a list of 20 elements encoding the visible layers according to layerlist
    Args:
      layerlist: 
    Returns:
    """
    if not isinstance(layerlist, list):
        layerlist = [layerlist]
    layers = 20 * [False]
    for layer in layerlist:
        layers[layer] = True
    return layers

Paste this code in the following directory phobos/utils/blender.py. Remove the installation.conf and run the phobos setup again.

Hope this helps

b-yogesh on 21 Dec 2020

@b-yogesh hi, thank you for your detailed reply. But after following the instructions as you provided. I still met a problem. The blender throws the error:
`Python: Traceback (most recent call last):
File "/home/hetolin/.config/blender/2.91/scripts/addons/phobos/operators/io.py", line 283, in execute
models.buildModelFromDictionary(model)
File "/home/hetolin/.config/blender/2.91/scripts/addons/phobos/model/models.py", line 1208, in buildModelFromDictionary
model['links'][lnk]['object'] = linkmodel.createLink(link)
File "/home/hetolin/.config/blender/2.91/scripts/addons/phobos/model/links.py", line 125, in createLink
geometrymodel.createGeometry(vis, 'visual', newlink)
File "/home/hetolin/.config/blender/2.91/scripts/addons/phobos/model/geometries.py", line 150, in createGeometry
bpy.context.scene.layers = bUtils.defLayers(defs.layerTypes[geomsrc])
AttributeError: 'Scene' object has no attribute 'layers'

location: :-1`

Is the bpy.context.scene.layers no longer used in Blender 2.9?

hetolin on 22 Dec 2020

Hi @hetolin, can you try commenting this line
bpy.context.window.view_layer = bUtils.defLayers(defs.layerTypes[geomsrc]) line (150)
in
/home/hetolin/.config/blender/2.91/scripts/addons/phobos/model/geometries.py

b-yogesh on 22 Dec 2020

@b-yogesh I tried your instruction, and it also throws the error: (I am using Blender 2.91, Maybe I need to change to Blender2.90? Have you meet such a problem when you are using phobos in Blender?)

Python: Traceback (most recent call last):
File "/home/hetolin/.config/blender/2.91/scripts/addons/phobos/operators/io.py", line 283, in execute
models.buildModelFromDictionary(model)
File "/home/hetolin/.config/blender/2.91/scripts/addons/phobos/model/models.py", line 1208, in buildModelFromDictionary
model['links'][lnk]['object'] = linkmodel.createLink(link)
File "/home/hetolin/.config/blender/2.91/scripts/addons/phobos/model/links.py", line 125, in createLink
geometrymodel.createGeometry(vis, 'visual', newlink)
File "/home/hetolin/.config/blender/2.91/scripts/addons/phobos/model/geometries.py", line 151, in createGeometry
bpy.context.window.view_layer = bUtils.defLayers(defs.layerTypes[geomsrc])
TypeError: bpy_struct: item.attr = val: Window.view_layer expected a ViewLayer type, not list
location: :-1

hetolin on 22 Dec 2020

Hello @hetolin, even I am using blender 2.91, and it works just fine. Although, I would like to ask, which branch of phobos have you installed? You should install this branch https://github.com/dfki-ric/phobos/tree/blender2.9 and not the master branch

Also, I did not see this line of code
bpy.context.window.view_layer = bUtils.defLayers(defs.layerTypes[geomsrc]) Line 151
in the geometries.py. I think you might have added it by yourself and forgot to remove it. Try removing that, and restart blender. Let me know how that goes

b-yogesh on 22 Dec 2020

Hello, @b-yogesh I have installed the blender2.9 branch of phobos. And I comment bpy.context.window.view_layer = bUtils.defLayers(defs.layerTypes[geomsrc]) Line 151. It throws new errors:

Python: Traceback (most recent call last):
File "/home/hetolin/.config/blender/2.91/scripts/addons/phobos/operators/io.py", line 283, in execute
models.buildModelFromDictionary(model)
File "/home/hetolin/.config/blender/2.91/scripts/addons/phobos/model/models.py", line 1208, in buildModelFromDictionary
model['links'][lnk]['object'] = linkmodel.createLink(link)
File "/home/hetolin/.config/blender/2.91/scripts/addons/phobos/model/links.py", line 127, in createLink
geometrymodel.createGeometry(col, 'collision', newlink)
File "/home/hetolin/.config/blender/2.91/scripts/addons/phobos/model/geometries.py", line 204, in createGeometry
newgeom.select = True
AttributeError: 'Object' object has no attribute 'select'

location: :-1

and I comment newgeom.select = True(line 204) it seems works now.

hetolin on 22 Dec 2020

Good to hear that it works now :). So, in blender, in the scene collection, you have one collection called 'Collision' or 'Collision joints'. I would recommend disabling that (click on the eye symbol).
For coloring, you can add material to each joint.
Let me know how it goes.

b-yogesh on 22 Dec 2020

Thank you for your guidance. After disabling the 'Collision' collection. It works now. thank you for your detailed instructions

hetolin on 22 Dec 2020

Glad that it worked out. :)

b-yogesh on 22 Dec 2020

@b-yogesh hello, I have successfully imported the urdf file in the blender. But the 'visual' model shown in the blender(top row picture)is not visually consistent with that shown in the Rviz(bottom row picture). Noted that I used the same urdf file. Have you ever had such a problem?
截屏2020-12-25 18 47 06
截屏2020-12-22 22 29 32

hetolin on 10 Jan 2021

Blenderproc: How to import URDF robot model and manipulate it?

Most helpful comment

All 23 comments

Start - Initializing module main.Initializer

Finished - Initializing module main.Initializer (took 0.014 seconds)

Start - Initializing module loader.BlendLoader

Finished - Initializing module loader.BlendLoader (took 0.004 seconds)

Start - Initializing module lighting.LightLoader

Finished - Initializing module lighting.LightLoader (took 0.011 seconds)

Start - Initializing module camera.CameraLoader

Finished - Initializing module camera.CameraLoader (took 0.003 seconds)

Start - Initializing module renderer.RgbRenderer

Finished - Initializing module renderer.RgbRenderer (took 0.006 seconds)

Start - Initializing module writer.Hdf5Writer

Start - Initializing module postprocessing.TrimRedundantChannels

Finished - Initializing module postprocessing.TrimRedundantChannels (took 0.002 seconds)

Finished - Initializing module writer.Hdf5Writer (took 0.249 seconds)

Start - Running blender pipeline

Start - Running module Initializer

Finished - Running module Initializer (took 0.093 seconds)

Start - Running module BlendLoader

Finished - Running module BlendLoader (took 0.591 seconds)

Start - Running module LightLoader

Finished - Running module LightLoader (took 0.000 seconds)

Start - Running module CameraLoader

Finished - Running module CameraLoader (took 0.003 seconds)

Start - Running module RgbRenderer

Finished - Running module RgbRenderer (took 26.805 seconds)

Start - Running module Hdf5Writer

Finished - Running module Hdf5Writer (took 3.201 seconds)

Finished - Running blender pipeline (took 30.705 seconds)

Related issues