A Survey of Hand Posture and Gesture Recognition Techniques and Technology
Laviola presents an excellent literature survey over hand and gesture recognition.
Included are:
Devices - Tracking and gloves
Methods:
Features - Template Matching, PCA
Learning - Neural Nets, HMMS, Instance-based method
Applications:
Sign Language Recognition
Gesture to Speech
Virtual Reality
3D Modeling
Control Systems (robots, tv, etc)
Discussion
A good starting point to find useful methods. Summarizes the state of the art at 1999. We're still using pretty much the same methods 8 years later.
Reference
Joseph J. LaViola, Jr. (1999). A Survey of Hand Posture and Gesture
Recognition Techniques and Technology, Brown University.
Saturday, May 3, 2008
Real-Time Locomotion Control by Sensing Gloves
Real-Time Locomotion Control by Sensing Gloves
Komura maps the motions of the fingers as measured by P5 glove to the motion of a humanoid figure in a real time game. The user first mimicks a reference character so that finger bend and hand orientation can be mapped to various parts of the virtual figure. This is done by matching the perioid of change in finger flex to the period of the figure's motion. Users were successfully able to control a figure in a game.
Discussion
The authors subtly switch from P5 to cyberglove for their experiments suggesting that the P5 does not have the sensitivity necessary for this system. In the experiments the user took more time to complete tasks but collided fewer times on average, suggesting that they were more careful when using the glove.
Reference
Taku Komura, Wai-Chun Lam; Real-time locomotion control by sensing gloves; Computer Animation and Virtual Worlds 17:5, 513-525, 2006
Komura maps the motions of the fingers as measured by P5 glove to the motion of a humanoid figure in a real time game. The user first mimicks a reference character so that finger bend and hand orientation can be mapped to various parts of the virtual figure. This is done by matching the perioid of change in finger flex to the period of the figure's motion. Users were successfully able to control a figure in a game.
Discussion
The authors subtly switch from P5 to cyberglove for their experiments suggesting that the P5 does not have the sensitivity necessary for this system. In the experiments the user took more time to complete tasks but collided fewer times on average, suggesting that they were more careful when using the glove.
Reference
Taku Komura, Wai-Chun Lam; Real-time locomotion control by sensing gloves; Computer Animation and Virtual Worlds 17:5, 513-525, 2006
A dynamic gesture recognition system for the Korean sign language (KSL)
A dynamic gesture recognition system for the Korean sign language (KSL)
Kim et al use a combination of Cybergloves and 3d position sensors to capture Korean sign language gestures using template matching and neural neworks. The overall gesture is match to templates. The x and y axis are divided into 8 regions and the motion of the gesture is tracked as positive or negative changes in region. Each gesture is matched to a set of template region changes to determine. Posture recognition is performed using Fuzzy Min Max Networks. Each class is determined by a max and min point defining a hyperbox and a membership function. After matching a sequence of positions to a template, the posture is used to determine which sign is represented.
Discussion
Gesture templates exist only in 2D; however it should be fairly easy to extend. Some of the templates seem to not match the motions from Figure 2. The templates also limit place a limit on the size of gestures performed.
Reference
J. S. Kim,W. Jang, and Z. Bien, "A dynamic gesture recognition system for the Korean sign language (KSL)," IEEE Trans. Syst., Man, Cybern. B, vol. 26, pp. 354–359, Apr. 1996.
Kim et al use a combination of Cybergloves and 3d position sensors to capture Korean sign language gestures using template matching and neural neworks. The overall gesture is match to templates. The x and y axis are divided into 8 regions and the motion of the gesture is tracked as positive or negative changes in region. Each gesture is matched to a set of template region changes to determine. Posture recognition is performed using Fuzzy Min Max Networks. Each class is determined by a max and min point defining a hyperbox and a membership function. After matching a sequence of positions to a template, the posture is used to determine which sign is represented.
Discussion
Gesture templates exist only in 2D; however it should be fairly easy to extend. Some of the templates seem to not match the motions from Figure 2. The templates also limit place a limit on the size of gestures performed.
Reference
J. S. Kim,W. Jang, and Z. Bien, "A dynamic gesture recognition system for the Korean sign language (KSL)," IEEE Trans. Syst., Man, Cybern. B, vol. 26, pp. 354–359, Apr. 1996.
Shape Your Imagination: Iconic Gestural-Based Interaction.
Shape Your Imagination: Iconic Gestural-Based Interaction.
Marsh and Watt present a study of naturally made gestures. By presenting a set of card with common items written on them and asking the study members to describe the items nonverbally, the researchers could determine what types of gesture the study memeber used. They found that people tend to prefer virtual to substitutive gestures and model shapes using 2 hands rather than one..
Discussion
Reference
T. Marsh, A. Watt, "Shape Your Imagination: Iconic Gestural-Based Interaction," vrais , p. 122, 1998.
Marsh and Watt present a study of naturally made gestures. By presenting a set of card with common items written on them and asking the study members to describe the items nonverbally, the researchers could determine what types of gesture the study memeber used. They found that people tend to prefer virtual to substitutive gestures and model shapes using 2 hands rather than one..
Discussion
Reference
T. Marsh, A. Watt, "Shape Your Imagination: Iconic Gestural-Based Interaction," vrais , p. 122, 1998.
A Survey of POMDP Applications
A Survey of POMDP Applications
Anthony Cassandra present the Partially Observable Markov Decision Process and summarizes several applications. This is largely a literature survey. Types of applications:
Machine Maintenance
Structural Inspection
Elevator Control Policies
Fishery Industry
Autonomous Robots
Behavioral Ecology
Machine Vision
Network Troubleshooting
Distributed Database Queries
Marketing
etc
Looking at the references for applications similar to yours could provide useful information about applying POMDPs or HMMs to your situation.
Reference
Anthony Cassandra. A Survey of Partially-Observable Markov Decision Process Applications. Presented at the AAAI Fall Symposium, 1998
Anthony Cassandra present the Partially Observable Markov Decision Process and summarizes several applications. This is largely a literature survey. Types of applications:
Machine Maintenance
Structural Inspection
Elevator Control Policies
Fishery Industry
Autonomous Robots
Behavioral Ecology
Machine Vision
Network Troubleshooting
Distributed Database Queries
Marketing
etc
Looking at the references for applications similar to yours could provide useful information about applying POMDPs or HMMs to your situation.
Reference
Anthony Cassandra. A Survey of Partially-Observable Markov Decision Process Applications. Presented at the AAAI Fall Symposium, 1998
Simultaneous Gesture Segmentation and Recognition based on Forward Spotting Accumulative HMMs
Simultaneous Gesture Segmentation and Recognition based on Forward Spotting Accumulative HMMs
Song and Kim modify the usual HMM model dividing the observation sequence into block through use of a sliding window. Each block of a gesture is used to train the corresponding HMM. Each HMM is used to recognize partial segments of the gesture over the block (train on [O1], then [o1,o2], etc), and a gesture is recognized through majority voting over the block. They determined that the optimal window size was 3. After the gesture is selected from the set of gesture HMMs it is compared either to a manually set threshold or the output of an HMM train on non-gestures. If the gesture HMM probablity exceeds the threshold or non-gesture HMM it is determined to be a gesture. Testing demonstrates that use of the non-gesture HMM spot gestures more accurately than manual thresholding.
Discussion
The gestures used in the experiment are very simple, mostly lift one arm or the other. Template matching could probably achieve similar results while being much less complex to implement.
Reference
Jinyoung Song, Daijin Kim, "Simultaneous Gesture Segmentation and Recognition based on Forward Spotting Accumulative HMMs," icpr , pp. 1231-1235, 2006.
Song and Kim modify the usual HMM model dividing the observation sequence into block through use of a sliding window. Each block of a gesture is used to train the corresponding HMM. Each HMM is used to recognize partial segments of the gesture over the block (train on [O1], then [o1,o2], etc), and a gesture is recognized through majority voting over the block. They determined that the optimal window size was 3. After the gesture is selected from the set of gesture HMMs it is compared either to a manually set threshold or the output of an HMM train on non-gestures. If the gesture HMM probablity exceeds the threshold or non-gesture HMM it is determined to be a gesture. Testing demonstrates that use of the non-gesture HMM spot gestures more accurately than manual thresholding.
Discussion
The gestures used in the experiment are very simple, mostly lift one arm or the other. Template matching could probably achieve similar results while being much less complex to implement.
Reference
Jinyoung Song, Daijin Kim, "Simultaneous Gesture Segmentation and Recognition based on Forward Spotting Accumulative HMMs," icpr , pp. 1231-1235, 2006.
Cyber Composer: Hand Gesture-Driven Intelligent Music Composition and Generation
Cyber Composer: Hand Gesture-Driven Intelligent Music Composition and Generation
Ip et al present a system that interprets hand and arm motion into music. They begin with a discussion of music theory that helps them determine what tones to play. Musical theory such as chord coherence and cadence can help determine which chord should follow previous ones. The system itself is composed of a pair of CyberGloves and a Polhemus 3D position tracker for each glove. The right hand determines the melody. New notes are generated when the user flexes his wrist, and the height of the hand determines the pitch of the note. Vertical movement of the hand can cause the pitch to shift with the motion. Finger flexion determines the dynamics and volume of the note. Lifting the left hand brings in a second instrument to play in either unison or harmony, and clenching the left hand initiates cadence and terminates the music.
Discussion
After desribing the system a usability discussion would have been useful. Waving the hand to generate notes seems like it would be very tiring. The use of chord coherence used to determine the next chord suggests that picking the chord you want may be difficult, and only possible by trial and error through pitch shifting. Also the use of Cybergloves is somewhat over kill, since all they are interested in is when the wrist bends and the general degree of finger flex.
Reference
Ip, H. H. S., K. C. K. Law, et al. (2005). Cyber Composer: Hand Gesture-Driven Intelligent Music Composition and Generation. Multimedia Modelling Conference, 2005. MMM 2005. Proceedings of the 11th International.
Ip et al present a system that interprets hand and arm motion into music. They begin with a discussion of music theory that helps them determine what tones to play. Musical theory such as chord coherence and cadence can help determine which chord should follow previous ones. The system itself is composed of a pair of CyberGloves and a Polhemus 3D position tracker for each glove. The right hand determines the melody. New notes are generated when the user flexes his wrist, and the height of the hand determines the pitch of the note. Vertical movement of the hand can cause the pitch to shift with the motion. Finger flexion determines the dynamics and volume of the note. Lifting the left hand brings in a second instrument to play in either unison or harmony, and clenching the left hand initiates cadence and terminates the music.
Discussion
After desribing the system a usability discussion would have been useful. Waving the hand to generate notes seems like it would be very tiring. The use of chord coherence used to determine the next chord suggests that picking the chord you want may be difficult, and only possible by trial and error through pitch shifting. Also the use of Cybergloves is somewhat over kill, since all they are interested in is when the wrist bends and the general degree of finger flex.
Reference
Ip, H. H. S., K. C. K. Law, et al. (2005). Cyber Composer: Hand Gesture-Driven Intelligent Music Composition and Generation. Multimedia Modelling Conference, 2005. MMM 2005. Proceedings of the 11th International.
Subscribe to:
Posts (Atom)
