The project aims at the unification of vision-based sensing in the CoP (Cognitive Perception) in the learning and planning system. On the one hand, CoP manages the interpretation of different kinds of sensors and, on the other hand, it automatically acquires and maintains the knowledge about the world and objects in the world. CoP selects sensors and sensor interpretation algorithms based on their expected utility. To this end, CoP learns and improves inter-sensor and inter-algorithmic models for method selection from experience. Especially the vision system, the major sensor we use, provides several automatic model improving techniques. Improved models accelerate the perception process and provide more robust results.

We showed the capabilities of the cognitive perception by making pancakes with our robot Rosie, supported by the PR2. The following perception tasks were part of preparation task:

• Detection and accurate localization of a Pancake-Mixture at an approximately known location
• Detect the pancake-maker, we used therefor a planar perspective template matching
• Detect a potential pancake on the pancake-maker

• Detect and localize a Spatula on the table in order to grasp it
• Calibrate the spatula in the hand in order to get a new tool frame
• Detect Plates on which the pancake will be served (on PR2)

We are developing methods for multi-modal object detection. Using different approaches, we are searching for daily-life objects in the context of the autonomous kitchen scenario. The target of the project is to enable vision-guided grasping of such objects. Therefore, we implement an intelligent parametrization of existing algorithms as well as new approaches, if necessary. The upper image on the right shows a screenshot of CoP analyzing a complex scenario. The lower images show the results of a student's Bachelor's thesis. A textured object is detected using a GPU-based implementation of the Randomized Tree algorithm by V. Lepetit.

Please note the related software projects: http://www.ros.org/wiki/cop

We are also applying classification techniques to improve the localization. Given for example a 3D segmentation, combining 3D and feature it is possible to classify some of most usual objects appearing in table setting scenes.

The first image on the right shows such a classification published at IROS 09. The lower image shows a result of a student's Bachelor's thesis, classifying camera images of branded objects only using training data acquired by image search on the Internet.

We are working on a method for automatically accessing an Internet database of 3D models that are searchable only by their user-annotated labels, in order to use them for vision and robotic manipulation purposes. Instead of having only a local database containing already seen objects, we want to use shared databases available over the Internet.

This approach, while having the potential to dramatically increase the visual recognition capabilities of robots, also poses certain problems, like wrong annotation due to the open nature of the database, or overwhelming amounts of data (many 3D models) or the lack of relevant data (no models matching a specified label).

To solve those problems we propose the following: First, we present an outlier/inlier classification method for reducing the number of results and discarding invalid 3D models that do not match our query. Second, we utilize an approach from computer graphics, the so called ’morphing’, to this application to specialize the models, in order to describe more objects. Third, we search for 3D models using a restricted search space, as obtained from our knowledge of the environment.

Inside-out action analysis shows new aspects in observation of daily life activities. Especially, manipulation tasks can be understood easier by an observer seeing the view of the acting person. The gaze provides early information about focus of the attention and critical points during an action. Any action is prepared by observing the area of interest before manipulating it. Additionally, humans mostly keep their focus on their hands while grasping an object. This fact allows us to automatically extract hand movements out of images of a gaze directed camera in relation to the acting person and the position in the world, that can be tracked in parallel, given an adequate world model. We propose in this work a novel hybrid 2D-3D method for markerless hand tracking under the difficult conditions such a gaze directed camera imposes: gaze changes with very high speed from one scene to another, with the effect of blurred frames and and little scene overlaps. So, the major problem that we address is the case of losing track of a hand. Given such an event the image is scanned for appearing hands until a hand’s present is validated over several frames. This validation is performed using a simple 2d hand model that equally serves initializing an more exact 3d hand model that is used to track the hand till the next track loss. Based on several distance measurement, we decide for an optimal hand configuration, as well in 2D as in 3D. The distance measurements are based on skin color, edges and edge directions. Our investigations are targeting as well on action analyzing in general as for generating a handbook of hand trajectories, grasping points and gaze directions in relation to the object of interest and possibly obstacles for controlling a service robot.

This project was partly funded by CoTeSys.