Three.js: OrbitControls breaks if camera's parent is moving (lookAt() issue)

[lguminski]

Description of the problem

OrbitControls (and TrackballControls) break if the camera's parent is moving.

 movingObject.add(camera);
 controls = new THREE.OrbitControls(camera);

The expected behaviour: jsfiddle
The actual behaviour: jsfiddle

The problem is that OrbitControls operates in world's space (due to Object3D.lookAt() bound to world's space), but the camera is bound to its moving parent. As a result their is a conflict between camera world position set by mouse movements, and by camera's parent movement.

The problem can be solved by introducing an additional camera (Forcing OrbitControls to navigate around a moving object (stackoverflow) ), but this is merely a workaround.

I believe that the real root cause of the problem a little design flaw in the Object3D.lookAt() method used by OrbitControls. The method is a low-level function that takes a point (x, y,, z) argument without any information in the point's reference system. So it makes an assumption that the point is in world's coordinates, and the assumption is sometimes right, sometimes wrong (the OrbitControls case).

Saying simply, I believe that the low-level method tries to be too smart.

Suggested solution:

1. Making Object3D.lookAt( x, y, z ) really simple by using parent's coordinates (this.matrix) instead of world coordinates (this.matrixWorld).
   
   
   
   • after the modificationOrbitControls starts respecting camera.parent.
   
   
   • this change is backwards compatible (except for r98 - more about it later), because for objects in world space this.matrix equals this.matrixWorld. Object's having a parent were not really supported anyway, and that was explicitly stated ("This method does not support objects with rotated and/or translated parent(s).")
   
   
2. Introducing a new high-level method Object3D.lookAtObject3D( object ), which would have all information on target's coordinate system provided, because its argument is THREE.Object3D. Thus this method can deal with all complexities caused by parents' translations and rotations.

The code would look like this

    lookAt: function () {
        // This method operates relatively to object's parent
        var q1 = new Quaternion();
        var m1 = new Matrix4();
        var target = new Vector3();
        var position = new Vector3();
        return function lookAt( x, y, z ) {
            if ( x.isVector3 ) {
                target.copy( x );
            } else {
                target.set( x, y, z );
            }
            this.updateMatrix();
            position.setFromMatrixPosition( this.matrix );
            if ( this.isCamera ) {
                m1.lookAt( position, target, this.up );
            } else {
                m1.lookAt( target, position, this.up );
            }
            this.quaternion.setFromRotationMatrix( m1 );
        };
    }(),
    lookAtObject3D: function () {
        var m1 = new Matrix4();
        var position = new Vector3();
        var parentMatrix;
        return function lookAtObject3D( object ) {
            object.updateWorldMatrix( true, false );
            if (this.parent) {
                this.parent.updateWorldMatrix( true, false );
                parentMatrix = this.parent.matrixWorld;
            } else {
                parentMatrix = m1.identity();
            }
            m1
                .getInverse( parentMatrix )
                .multiply( object.matrixWorld );
            position.setFromMatrixPosition( m1 );
            this.lookAt( position );
        };
    }(),

The only problem I can see is that the latest version (r98) started supporting rotated parents. But I don't think people started using it (perhaps except for @greggman ), and the advantage of Object3D.lookAtObject3D( object ) over r98's lookAt() is that it supports any coordinate system (no matter how object's parents are rotated).

Here are 2 examples showing how it would work:

• solar system (jsfiddle) showing OrbitControls working with a moving object, and also using the new Object3D.lookAtObject3D( object ) method
• a targetting tank (jsfiddle) - this is the original example by @greggman from #14496 after my modification

An initial PR follows.

Three.js version

• [x] r98

Browser

• [x] All of them

OS

• [x] All of them

Hardware Requirements (graphics card, VR Device, ...)

EDIT: code reformatted and added support for parentless objects (detached camera)

[WestLangley]

OrbitControls ... breaks if the camera's parent is moving.

Actually, it has nothing to do with a parent moving. The camera must either have no parent, or if it has a parent, the parent's world position must be zero.

You can't use the controls and simultaneously expect the camera's position to be controlled by a different method.

OrbitControls. The method is a low-level function that takes a point (x, y,, z) argument without any information in the point's reference system.

Actually, that is not true. The Object3D docs state lookAt()

Rotates the object to face a point in world space

[lguminski]

The camera must either have no parent, or if it has a parent, the parent's world position must be zero.

I just presented a way of dropping this limitation.

You can't use the controls and simultaneously expect the camera's position to be controlled by a different method.

@WestLangley with all the respect, why not? The use case of looking at the moving moon is a good example. And at the moment you are looking at it, it doesn't stop rotating. It keep doing that. This is what I am modelling.

Actually, that is not true. The Object3D docs state lookAt()

Rotates the object to face a point in world space.

Documentation states it clearly. But what I am saying is that the x, y, z point, which Object3D.lookAt() receives as an argument in this use case, is influenced by the movement of object's parent. So I understand what the documentation says, but this is not what happens in this use case, and I am explaining why. And this is a valid use case.

And I believe that by the modification I am suggesting, you will get a more robust library. Personally I can live with the workaround I came up with (the "second camera" solution), so I am not relying on this PR. Nevertheless, I think that having a universal method that is able to rotate an object toward any other object in the scene (no matter how they are situated in objects' hierarchy or rotated), would help many people.

[EliasHasle]

A candidate solution for the original problem is PR #16374.

[titansoftime]

Personally I hope something like this gets implemented as r98's lookAt broke my control setup as well. In the meantime I'm just overwriting the lookAt function with an older one.

[WestLangley]

Closing. See discussion in #15268 and #16374.

[trusktr]

I believe it is intuitively expected that OrbitControls should work relative to the camera's parent. This would make it possible to simply attach a camera to any object, and orbit it without thinking about the implementation.

I spent a whole day yesterday figuring how to do this properly, and the best approach (a hack) is the fakeCamera approach, then copying that camera to the actual camera that is child of some object. Just like the answer in this StackOverflow answer: https://stackoverflow.com/a/53298655/454780

[trusktr]

Before I try to implement a solution, would it be a welcome change? I'm thinking adding an option to OrbitControls, something like

new OrbitControls(camera, domElement, { relativeToParent: true })

(having a single arg, or passing an options object, is up for debate. Maybe we just set it as a property on the instance?)

[mrdoob]

Before I try to implement a solution, would it be a welcome change?

Would you like to create a new issue with that suggestion?

[lguminski]

@trusktr It would be a change definitely welcome by me, as I also perceive the fact that camera has a parent, and moves independently from it, as counterintuitive. And despite the fakeCamera workaround comes from me, I would be very happy if everyone could forget about it.

And although I still believe that the cleanest way of solving the problem would be by changing lookAt function to use coordinates in parent's coordinate system (like in the PR which I attached to this issue), I still remember well @WestLangley 's concern:

I believe that concept is a bit too complex for most users.

So I am looking forward to see your PR, and learn how you approach this problem.

I think you should go ahead and create a new issue.

[lguminski]

@trusktr @mrdoob BTW I just realized that Babylon.js does not suffer from this problem, because their lookAt uses local coordinate system. Exactly what I was proposing in the PR enclosed here.

https://doc.babylonjs.com/api/classes/babylon.mesh#lookat

targetPoint: Vector3
the position (must be in same space as current mesh) to look at