Filament: support blend shapes more than 8

[j6lim]

Is your feature request related to a problem? Please describe.

Our company HyprSense is making an avatar animation technology (like iPhone X Animoji). We were looking for rendering engines to showcase 3D characters, and they usually have 40~50 blend shapes rigged in.

With Unity / Unreal Engine / Apple SceneKit, we had no problem because their vertex shader supports that number of blend shapes for a skinned mesh renderer. We are having difficulties with Sceneform, where the number of blend shapes is limited to 8 (or 4 with blending normals).

We want you to ask if you have a plan to implement a feature to support 40+ blend shapes.

Describe the solution you'd like
You might want to store the blend shape vertex and normal deformation information in a data texture, and load from the vertex shader to apply the deformation. There are similar threads discussing about this topic in the three.js repository, and they came up with the approach I have mentioned.

https://github.com/mrdoob/three.js/issues/15556
https://github.com/mrdoob/three.js/issues/14441

OS and backend
All OSes and backends

[prideout]

Thanks for the feature request.

For now, you might want to consider animating the verts on the CPU, which is what we do for ImGui. Also note that Filament allows you to modify the vertex shader (see our point_sprites sample app).

We would be interested in more details regarding Unity / Unreal Engine / Apple SceneKit.

Notes to self:

• See "Programmable Vertex Pulling" in OpenGL Insights, although the author only tested his approach on NVIDIA and AMD.
• OP's first link has a vertex shader loop with 136 iterations, making a texture lookup in each iteration.

[j6lim]

Unreal Engine has a feature to blend vertex morphs and tangent morphs in a computer shader.

https://github.com/EpicGames/UnrealEngine/blob/release/Engine/Shaders/Private/GpuSkinCacheComputeShader.usf

Unity / SceneKit is closed source software, but they might take advantage of compute shaders.

Since Filament is targeting OpenGL ES 3.0 which has no compute shader, there would be a couple of workarounds.

1. Blend in CPU
   This must be very slow, but better have a feature for this than none. If we are going to use filament with WebGL, it would be better to do morph in WebAseembly, not in Javascript side.

2. Blend in Vertex Shader
   You can create a data texture containing vertex morph deltas and tangent morph deltas, and read the corresponding morph data from the each vertex shader call to blend.

[j6lim]

You might want to adopt the way ViroCore supports: https://developer.viromedia.com/virocore/reference/com/viro/core/Object3D.MorphMode.html

ViroCore (which is the only renderer on Android, as far as we know, which implements morph targets and skinned mesh renderers good enough to support animated characters) supports both CPU blending / GPU blending for morph target vertices.

[erolgobel17]

Is your feature request related to a problem? Please describe.

Our company HyprSense is making an avatar animation technology (like iPhone X Animoji). We were looking for rendering engines to showcase 3D characters, and they usually have 40~50 blend shapes rigged in.

With Unity / Unreal Engine / Apple SceneKit, we had no problem because their vertex shader supports that number of blend shapes for a skinned mesh renderer. We are having difficulties with Sceneform, where the number of blend shapes is limited to 8 (or 4 with blending normals).

We want you to ask if you have a plan to implement a feature to support 40+ blend shapes.

Describe the solution you'd like
You might want to store the blend shape vertex and normal deformation information in a data texture, and load from the vertex shader to apply the deformation. There are similar threads discussing about this topic in the three.js repository, and they came up with the approach I have mentioned.

mrdoob/three.js#15556
mrdoob/three.js#14441

OS and backend
All OSes and backends

[prideout]

Yes, storing the morph targets in a data texture is a possibility that we will likely pursue (sometimes this is called "programmable vertex pulling"). However we do not have a timeline for the implementation.

One thing that might be helpful would be a real-world example that does this, ideally in the form of a glTF file. This would allows us to compare performance between CPU-based and GPU-based approaches.