Personal tools
 

Christoph Mayer

Recent research aims at enabling machines to utilize communication channels natural to human beings, such as gesture or facial expressions. Humans interpret emotion from video and audio information and heavily rely on this information during every-day communication. Therefore, knowledge about human behavior, intention, and emotion is necessary to construct convenient human-machine interaction mechanisms.

My research aims at recognizing facial expressions and the emotional state of human from camera images in order to apply this information on human-computer interaction. Traditional human-computer interaction via mouse and keyboard is often considered slow, non-intuitive and requires time-consuming manual reading or even specific training. This is mainly, because information on communication channels that are more natural to human beings, such as facial expressions, are not available to computers.

A preprocessing step often includes the segmentation of the face from the background. Skin color has been proven useful for this task. We developed an algorithm that adapts on image properties and therefore performs robust even in heavy light changes. The video below shows the original image data, the adaptive approach and a static approach that does not adapt to the specific image content.

Adaptive Skin Color Estimation

 

We rely on a model-based approach for facial expression recognition. Models represent knowledge about real-world objects, such as position, shape or texture via a model parameter and therefore provide an abstraction for the interpretation task. In case of facial expression recognition, the model reflects face related information like the position and rotation in 3D space, the rising level of the eyebrows or the opening degree of the mouth. In a sequence of images the model parameters have to be updated for every image in order to reflect the image content. For visualization, the video below shows a simple 3D tracking of a human face.

3D Face Tracking

 

In order to extract higher level information, the model parameters have to match the image content. The computational challenge of determining these model parameters is called model fitting and is tackled with a fitness measurement function between the image content and the model parameterization. These functions are either minimized to determine a good model fit or calculate a parameter update directly. The process of model fitting is visualized in the video below. In the beginning, the model parameters do not match the image content. However, by changing the model parameters the model fitness is increased. The fitness function is visualized by the color of the model, changing from red to green.

Face Model Fitting

 

From this information, machine learning techniques are applied to determine the facial expression. The image data in the movie below is taken from a standard database for facial expression recognition, the Cohn-Kanade facial expression database. Please see this reference for further information on the database: Kanade, T., Cohn, J. F., & Tian, Y. (2000). Comprehensive database for facial expression analysis. Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition (FG'00), Grenoble, France, 46-53.

Facial Expression Recognition with 3D Models



Research Topics

  • Model-Based Image Interpretation
  • Model Fitting
  • Objective Functions, Learning Objective Functions
  • Face Detection, Face Model Fitting, Head Tracking
  • Facial Expression Recognition, Emotion Recognition
  • Color Detection, Adaptive Skin Color Classification

Current Research Projects

  • Face Image Analysis : As robots emerge from their classical domain - factories - to be included in every day life, they need to gain new abilities besides those needed in manufacturing. They need not only to support humans, but also be able to socialize with their users to enhance the interactant experience and allow for social bonding. Recent progress in the field of Computer Vision allows intuitive interaction via gesture or facial expressions between humans and technical systems. Recent research aims at enabling machines to utilize communication channels natural to human beings, such as gesture or facial expressions. Humans interpret emotion from video and audio information and heavily rely on this information during every-day communication. Therefore, knowledge about human behavior, intention, and emotion is necessary to construct convenient human-machine interaction mechanisms. The human face provides much of the information that is passed between humans in every-day communication. Although most of this information is passed on a subconscious level, we still rely on the interaction partner's facial expression to determine emotional state or attention to form a prediction of his or her reaction.

Former Research Projects

  • Mudis : Since existing methods for human-machine interaction are often unintuitive, a lot of time is required for humans to adapt to the operation of a specific machine. In contrast, the MuDiS project aims at granting machines the ability to adapt to typical human behavior. The goal of the project is the development of a multimodal dialog system that considers various human communication channels such as facial expressions, spoken language and gestures for human-machine interaction. We perform experiments to determine the requirements for robots to interact with humans in an intuitive way. Insights gained from these human-human experiments are applied to the human-machine interface to grant robots the capability of participating in simple, every-day dialogs in various environments. To tackle this challenge, we unite researchers from diverse scientific areas, such as computer science, electrical engineering and psychology to reflect the interdisciplinary character of the project.

Selected Publications

Document Actions