My research aims at recognizing facial expressions and the emotional state of human from camera images in order to apply this information on human-computer interaction. Traditional human-computer interaction via mouse and keyboard is often considered slow, non-intuitive and requires time-consuming manual reading or even specific training. This is mainly, because information on communication channels that are more natural to human beings, such as facial expressions, are not available to computers.

Therefore, recent research aims at enabling machines to utilize communication channels natural to human beings, such as gesture or facial expressions. Humans interpret emotion from video and audio information and heavily rely on this information during every-day communication. Therefore, knowledge about human behavior, intention, and emotion is necessary to construct convenient human-machine interaction mechanisms.

The recognition of facial expression is conductes in several steps:

A preprocessing step often includes the segmentation of the face from the background. Skin color has been proven useful for this task. We developed an algorithm that adapts on image properties and therefore performs robust even in heavy light changes. The video below shows the original image data, the adaptive approach and a static approach that does not adapt to the specific image content.

This information is used to fit a face model onto the image. Models represent knowledge about real-world objects, such as position, shape or texture via a model parameter and therefore provide an abstraction for the interpretation task. In case of facial expression recognition, the model reflects face related information like the position and rotation in 3D space, the rising level of the eyebrows or the opening degree of the mouth. In a sequence of images the model parameters have to be updated for every image in order to reflect the image content. For visualization, the video below shows a simple 3D tracking of a human face.

Finally, in order to extract higher level information, the model parameters have to match the image content. The computational challenge of determining these model parameters is called model fitting and is tackled with a fitness measurement function between the image content and the model parameterization. These functions are either minimized to determine a good model fit or calculate a parameter update directly. The process of model fitting is visualized in the video below. In the beginning, the model parameters do not match the image content. However, by changing the model parameters the model fitness is increased. The fitness function is visualized by the color of the model, changing from red to green.

Publications