Airsim: Surface normals having a weird bias applied when requesting via AirSim

[LSBOSS]

I'm working with surface normals requested via AirSim and observed a strange behavior. There seems to be a small non-linear biased applied to the surface normals' color values when requesting via airsim.ImageRequest and type airsim.ImageType.SurfaceNormals.

I created a test scene with 9 axis aligned cubes and a camera looking down the positive x-axis. I then request a surface normal image via AirSim and let Unreal Engine 4 produce one be utilizing the highres screen shot function with "Include Buffer Visualization Targets" enabled. Here's the result:

Exported via UE4:

Exported via AirSim:

The camera facing side of a the cube (the cyan one) should have a world normal of (-1, 0, 0) which according to the responsible material in the UE4 AirSim plugin (Plugins\AirSim\Content\HUDAssets\NormalsMaterial.uasset) should result in a color of (0, 0.5, 0.5) in float space or (0, 127, 127) in RGB space:

But as you can clearly see, the color of said cube side on the AirSim export is (0, 187, 187) which results in a normal of (-0.87, 0.39, 0.39) after normalization. There's clearly a bias applied and I didn't found a workaround yet.

settings.json:

{
    "SettingsVersion": 1.2,
    "SimMode": "ComputerVision",
    "Vehicles": {
        "MyVehicle": {
            "VehicleType": "ComputerVision",
            "Cameras": {
                "my_camera": {
                    "CaptureSettings": [
                        {
                            "ImageType": 6,
                            "FOV_Degrees": 60,
                            "Width": 1024,
                            "Height": 1024
                        }
                    ],
                    "X": 0.0,
                    "Y": 0.0,
                    "Z": 0.0,
                    "Pitch": 0.0,
                    "Roll": 0.0,
                    "Yaw": 0.0
                }
            }
        }
    }
}

Minimal working Python code:

import setup_path
import airsim

client = airsim.VehicleClient(timeout_value = 7200)
client.confirmConnection()

responses = client.simGetImages([airsim.ImageRequest("my_camera", airsim.ImageType.SurfaceNormals)])
airsim.write_file("normal.png", responses[0].image_data_uint8)

UE4 Map file
BrokenSurfaceNormalsUE4Map.zip

Version infos

• Unreal Engine: 4.24.2
• AirSim Plugin build from commit: Mon Jun 1 10:30:23 2020 +0530
• AirSim Python library: 1.2.8
• Python: 3.7.2 32bit

[marcociara379]

I confirm the issue,

• UE4.18
• AirSim 1.2

[dnychennnn]

I also confirm the issue to exist,
my env:

• UE4.18
• AirSim 1.2

[LSBOSS]

HA! I knew it has something to do with gamma because of the non-linear nature of the bias.

I spent the last two days deep down in AirSim and Unreal rendering code and finally found the line which is responsible for the issue:

In function APIPCamera::updateCaptureComponentSetting the following line is creating render target textures with disabled linear gamma for rendering request for Scene, Segmentation, SurfaceNormals and Infrared:
render_target->InitCustomFormat(setting.width, setting.height, pixel_format, false);

Changing the last flag to true results in the cyan surface from above screenshot has the right color (0,127,127) instead of (0,187,187) and thus also the right world normal of (-1, 0, 0).

Please stay tuned for a PR :)

PS This flag could also be the reason why non of the colors of the segmentation request outputs is in the list of https://microsoft.github.io/AirSim/seg_rgbs.txt. I'll take a look in that as well.