Use Your Hand as a 3-D Mouse, or, Relative Orientation from Extended Sequences of Sparse Point and Line Correspondences Using the Affine Trifocal Tensor

Lars Bretzner and Tony Lindeberg

In: Proc. 5th European Conference on Computer Vision (H. Burkhardt and B. Neumann, eds.), vol. 1406 of Lecture Notes in Computer Science, (Freiburg, Germany), pp. 141--157, Springer Verlag, Berlin, June 1998.

Abstract

This paper addresses the problem of computing three-dimensional structure and motion from an unknown rigid configuration of point and lines viewed by an affine projection model. An algebraic structure, analogous to the trilinear tensor for three perspective cameras, is defined for configurations of three centered affine cameras. This centered affine trifocal tensor contains 12 coefficients and involves linear relations between point correspondences and trilinear relations between line correspondences It is shown how the affine trifocal tensor relates to the perspective trilinear tensor, and how three-dimensional motion can be computed from this tensor in a straightforward manner. A factorization approach is also developed to handle point features and line features simultaneously in image sequences.

This theory is applied to a specific problem of human-computer interaction of capturing three-dimensional rotations from gestures of a human hand. A qualitative model is presented, in which three fingers are represented by their position and orientation, and it is shown how three point correspondences (blobs at the finger tips) and three line correspondences (ridge features at the fingers) allow the affine trifocal tensor to be determined, from which the rotation is computed. Besides the obvious application, this test problem illustrates the usefulness of the affine trifocal tensor in a situation where sufficient information is not available to compute the perspective trilinear tensor, while the geometry requires point correspondences as well as line correspondences over at least three views.

Keywords: affine trifocal tensor, trilinear tensor, affine, perspective, projection, structure, motion, factorization, tracking, point, blob, line, ridge, scale selection, human-computer interaction, computer vision.

PDF: (271 kb)

Video illustrating how hand motions computed in this way can be used for controlling the motion of other objects:

Applications of visual control using the 3-D hand mouse are described in the following patent application (now released):
T. Lindeberg and L. Bretzner ``Method and arrangement for controlling means for three-dimensional transfer of information by motion detection'', PCT/SE1999/000402, 1999.

Background and related material: (Later version of this work with a more extensive experimental evaluation) (The feature tracker with automatic scale selection that this work builds upon)

Related project: Computer vision based human-computer interaction