Three.js: Attach object to bone (Skeletal animation)

[trevex]

Previously I calculated a point on the mesh, I attached a box to, by using the morphtarget vertex data. Obviously this is quite straight forward, but how would I go about and do the same with a skeletal animated model?

As far as I know the animation-system was developed by @alteredq , but I couldn't fully figure it out, especially the bone texture. How is the current position of a bone calculated?

[crobi]

As a questions, this would probably be the wrong place to post it, but since I already extracted this information for my project and it took me some time, I'll post it here in case anyone wants to continue the development of the skinning:

The update of the skinning matrices is implemented in SkinnedMesh.updateMatrixWorld.

The data is stored in following places:

// A Matrix4 transforming from the local bone space to global space
// The index i is here the index of the bone
SkinnedMesh.bones[i].skinMatrix

// A Matrix4 transforming from global space to the local bone space
// The index i is here the index of the bone
// Note: This should have been a property of SkinnedMesh.bones[i] in my opinion,
//       as some file formats (COLLADA) store this explicitely and in a way that is
//       incompatible with the way three.js automatically computes these matrices.
SkinnedMesh.boneInverses[i]

// A huge array of floats that stores the products of the above matrices,
// concatenated into a flat array.
SkinnedMesh.boneMatrices

// A texture containing the data of SkinnedMesh.boneMatrices
// This is just a way of passing all the matrices to a vertex shader
SkinnedMesh.boneTexture

The skinning equation for vertex i in three.js is as follows (written for clarity, not performance):

// The indices of bones influencing vertex i are stored as a Vector4 in the geometry
// Note: some indices may be invalid if a vertex is influenced by less than 4 bones
// Note: three.js does not support more than four bones per vertex
bones = []
bone[0] = mesh.geometry.skinIndices[i].x
bone[1] = mesh.geometry.skinIndices[i].y
bone[2] = mesh.geometry.skinIndices[i].z
bone[3] = mesh.geometry.skinIndices[i].w
// The weights of those four bones are stored as another Vector4 in the geometry
weights = []
weight[0] = mesh.geometry.skinWeights[i].x
weight[1] = mesh.geometry.skinWeights[i].y
weight[2] = mesh.geometry.skinWeights[i].z
weight[3] = mesh.geometry.skinWeights[i].w
// The inverse matrix transforms the static geometry into the local space of the bone
// These matrices are static and stored in the mesh
inverses = []
inverses[0] = mesh.boneInverses[ bone[0] ]
inverses[1] = mesh.boneInverses[ bone[1] ]
inverses[2] = mesh.boneInverses[ bone[2] ]
inverses[3] = mesh.boneInverses[ bone[3] ]
// The skin matrix transforms from the local space of the bone to the global space
// These matrices are animated and stored in the mesh
skinMatrices = []
skinMatrices[0] = mesh.bones[ bone[0] ].skinMatrix
skinMatrices[1] = mesh.bones[ bone[1] ].skinMatrix
skinMatrices[2] = mesh.bones[ bone[2] ].skinMatrix
skinMatrices[3] = mesh.bones[ bone[3] ].skinMatrix
// Finally, the skinning equation in pseudocode
// Note: the product of the skin matrix and inverse matrix is computed on the CPU
//       in SkinnedMesh.updateMatrixWorld
// Note: the actual skinning equation is implemented in a vertex shader
position = (0,0,0,0)
position += weights[0] * skinMatrices[0] * inverses[0] * mesh.geometry.vertices[i]
position += weights[1] * skinMatrices[1] * inverses[1] * mesh.geometry.vertices[i]
position += weights[2] * skinMatrices[2] * inverses[2] * mesh.geometry.vertices[i]
position += weights[3] * skinMatrices[3] * inverses[3] * mesh.geometry.vertices[i]

[mohamed--abdel-maksoud]

Hi Crobi!
I'm stuck with the same problem and your answer seems to be the only comprehensive one out there, thanks for sharing!
it's just the 2 notes in the skinning equation that confuse me: As far as I understood, to do skeletal animation one has to update the position of every vertex using the equations you listed on the CPU side, what is the "actual skinning" which is implemented in a vertex shader then?

sorry if the question is too primitive, I'm new to 3d animation!
regards,
-Mohamed

[crobi]

The positions of vertices are not updated on the CPU. Three.js implements hardware skinning. I did not implement it myself, but this is what I found:

• For each frame, the animation system updates the transformations of the skeleton bones (on the CPU).
• The transformation matrices of all skeleton bones are packed into a single flat array, which is then uploaded into a texture.
  
  
  
  • The texture is created in SkinnedMesh.js.
  
  
  • The array is updated in SkinnedMesh.js
  
  
  • The texture is used in WebGLRenderer.js
  
  
  • If the GPU does not support vertex shader textures, the matrices are set using shader uniforms (less ideal, since the number of uniforms is limited?).
  
  
• The vertex shader uses the texture to evaluate the skinning equation on the GPU.
  
  
  
  • Each vertex contains (in addition to the usual data like vertex position and normal) one 4D vector which stores the indices of up to four bones that affect the vertex, as well as one 4D vector with weights for those bones.
  
  
  • Vertex shader code is in WebGLShaders.js.
  
  

[makc]

This is an old thread to bump, but I've figured it is better then creating new one. I have this code to mirror skin vertices transformation:

    var transformedSkinVertex = function (skin, index) {
        var skinIndices = (new THREE.Vector4 ()).fromAttribute (skin.geometry.getAttribute ('skinIndex'), index);
        var skinWeights = (new THREE.Vector4 ()).fromAttribute (skin.geometry.getAttribute ('skinWeight'), index);
        var skinVertex = (new THREE.Vector3 ()).fromAttribute (skin.geometry.getAttribute ('position'), index).applyMatrix4 (skin.bindMatrix);
        var result = new THREE.Vector3 (), temp = new THREE.Vector3 (), tempMatrix = new THREE.Matrix4 (); properties = ['x', 'y', 'z', 'w'];
        for (var i = 0; i < 4; i++) {
            var boneIndex = skinIndices[properties[i]];
            tempMatrix.multiplyMatrices (skin.skeleton.bones[boneIndex].matrixWorld, skin.skeleton.boneInverses[boneIndex]);
            result.add (temp.copy (skinVertex).multiplyScalar (skinWeights[properties[i]]).applyMatrix4 (tempMatrix));
        }
        return result.applyMatrix4 (skin.bindMatrixInverse);
    };

This works perfectly for T pose:

But with arms lowered some parts explode into angel-like shape:

Here is arms placed slightly differently:

My current theory is that this happens when there are > 2 bones. But... why?

[titansoftime]

Unless I am misunderstanding the OP... Since the original post date this has been made very straight-forward. The bones are now an Object3D so you can simply add a mesh to the bone in a mesh.skeleton.bones[someindex].add(mesh2).

This is how I add weapon objects to my character mesh in my app.

[makc]

true, but I need to know where the vertices are, but since

your answer seems to be the only comprehensive one out there

means I've found it on google, I'm here. perhaps I should echo this to stackoverflow.

edit: added to stackoverflow