Persönlicher Status und Werkzeuge

Home Research Cop: Cognitive Perception

Cop: Cognitive Perception

The project aims at the unification of vision-based sensing in the CoP (Cognitive Perception) in the learning and planning system. On the one hand, CoP manages the interpretation of different kinds of sensors and, on the other hand, it automatically acquires and maintains the knowledge about the world and objects in the world. CoP selects sensors and sensor interpretation algorithms based on their expected utility. To this end, CoP learns and improves inter-sensor and inter-algorithmic models for method selection from experience. Especially the vision system, the major sensor we use, provides several automatic model improving techniques. Improved models accelerate the perception process and provide more robust results.

Perception for everyday manipulation

We showed the capabilities of the cognitive perception by making pancakes with our robot Rosie, supported by the PR2. The following perception tasks were part of preparation task:

  • Detection and accurate localization of a Pancake-Mixture at an approximately known location
  • Detect the pancake-maker, we used therefor a planar perspective template matching
  • Detect a potential pancake on the pancake-maker

  • Detect and localize a Spatula on the table in order to grasp it
  • Calibrate the spatula in the hand in order to get a new tool frame
  • Detect Plates on which the pancake will be served (on PR2)


Object Localization

We are developing methods for multi-modal object detection. Using different approaches, we are searching for daily-life objects in the context of the autonomous kitchen scenario. The target of the project is to enable vision-guided grasping of such objects. Therefore, we implement an intelligent parametrization of existing algorithms as well as new approaches, if necessary. The upper image on the right shows a screenshot of CoP analyzing a complex scenario. The lower images show the results of a student's Bachelor's thesis. A textured object is detected using a GPU-based implementation of the Randomized Tree algorithm by V. Lepetit.

Please note the related software projects: http://www.ros.org/wiki/cop


Object Classification

We are also applying classification techniques to improve the localization. Given for example a 3D segmentation, combining 3D and feature it is possible to classify some of most usual objects appearing in table setting scenes.

The first image on the right shows such a classification published at IROS 09. The lower image shows a result of a student's Bachelor's thesis, classifying camera images of branded objects only using training data acquired by image search on the Internet.



3D Model Selection from an Internet Database

We are working on a method for automatically accessing an Internet database of 3D models that are searchable only by their user-annotated labels, in order to use them for vision and robotic manipulation purposes. Instead of having only a local database containing already seen objects, we want to use shared databases available over the Internet.

This approach, while having the potential to dramatically increase the visual recognition capabilities of robots, also poses certain problems, like wrong annotation due to the open nature of the database, or overwhelming amounts of data (many 3D models) or the lack of relevant data (no models matching a specified label).

To solve those problems we propose the following: First, we present an outlier/inlier classification method for reducing the number of results and discarding invalid 3D models that do not match our query. Second, we utilize an approach from computer graphics, the so called ’morphing’, to this application to specialize the models, in order to describe more objects. Third, we search for 3D models using a restricted search space, as obtained from our knowledge of the environment.

Eye Watch You

Inside-out action analysis shows new aspects in observation of daily life activities. Especially, manipulation tasks can be understood easier by an observer seeing the view of the acting person. The gaze provides early information about focus of the attention and critical points during an action. Any action is prepared by observing the area of interest before manipulating it. Additionally, humans mostly keep their focus on their hands while grasping an object. This fact allows us to automatically extract hand movements out of images of a gaze directed camera in relation to the acting person and the position in the world, that can be tracked in parallel, given an adequate world model. We propose in this work a novel hybrid 2D-3D method for markerless hand tracking under the difficult conditions such a gaze directed camera imposes: gaze changes with very high speed from one scene to another, with the effect of blurred frames and and little scene overlaps. So, the major problem that we address is the case of losing track of a hand. Given such an event the image is scanned for appearing hands until a hand’s present is validated over several frames. This validation is performed using a simple 2d hand model that equally serves initializing an more exact 3d hand model that is used to track the hand till the next track loss. Based on several distance measurement, we decide for an optimal hand configuration, as well in 2D as in 3D. The distance measurements are based on skin color, edges and edge directions. Our investigations are targeting as well on action analyzing in general as for generating a handbook of hand trajectories, grasping points and gaze directions in relation to the object of interest and possibly obstacles for controlling a service robot.

Acknowledgements

This project was partly funded by CoTeSys.

Publications

Journal Articles and Book Chapters

Furniture Models Learned from the WWW -- Using Web Catalogs to Locate and Categorize Unknown Furniture Pieces in 3D Laser Scans (Oscar Martinez Mozos, Zoltan Csaba Marton, Michael Beetz), In Robotics & Automation Magazine, IEEE, volume 18, 2011. [bib] [pdf]
Combined 2D-3D Categorization and Classification for Multimodal Perception Systems (Zoltan Csaba Marton, Dejan Pangercic, Nico Blodow, Michael Beetz), In The International Journal of Robotics Research, Sage Publications, volume 30, 2011. [bib] [pdf]
Importance Sampling as One Solution to the Data Association Problem in Multi-target Tracking (Nicolai v. Hoyningen-Huene, Michael Beetz), Chapter in VISIGRAPP 2009 (AlpeshKumar Ranchordas, Helder Araujo, eds.), Springer-Verlag Berlin Heidelberg, 2010. [bib]
Automatic feature generation in endoscopic images (Ulrich Klank, N. Padoy, H. Feussner, N. Navab), In International Journal of Computer Assisted Radiology and Surgery, Springer, volume 3, 2008. [bib]
The Contracting Curve Density Algorithm: Fitting Parametric Curve Models to Images Using Local Self-adapting Separation Criteria (Robert Hanek, Michael Beetz), In International Journal of Computer Vision, volume 59, 2004. [bib] [pdf]

Conference Papers

Towards Modular Spatio-temporal Perception for Task-adapting Robots (Zoltan-Csaba Marton, Florian Seidel, Michael Beetz), In Postgraduate Conference on Robotics and Development of Cognition (RobotDoC-PhD), a satellite event of the 22nd International Conference on Artificial Neural Networks (ICANN), 2012. [bib]
Object Categorization in Clutter using Additive Features and Hashing of Part-graph Descriptors (Zoltan-Csaba Marton, Ferenc Balint-Benczedi, Nico Blodow, Lucian Cosmin Goron, Michael Beetz), In Proceedings of Spatial Cognition (SC), 2012. [bib] [pdf]
Robots that Validate Learned Perceptual Models (Ulrich Klank, Lorenz Mösenlechner, Alexis Maldonado, Michael Beetz), In IEEE International Conference on Robotics and Automation (ICRA), 2012. [bib] [pdf]
Segmenting Cylindrical and Box-like Objects in Cluttered 3D Scenes (Lucian Cosmin Goron, Zoltan Csaba Marton, Gheorghe Lazea, Michael Beetz), In 7th German Conference on Robotics (ROBOTIK), 2012. [bib] [pdf]
Efficient Part-Graph Hashes for Object Categorization (Ferenc Balint-Benczedi, Zoltan-Csaba Marton, Michael Beetz), In 5th International Conference on Cognitive Systems (CogSys), 2012. [bib] [pdf]
Multimodal Autonomous Tool Analyses and Appropriate Application (Ingo Kresse, Ulrich Klank, Michael Beetz), In 11th IEEE-RAS International Conference on Humanoid Robots, 2011. [bib] [pdf]
Robotic Roommates Making Pancakes (Michael Beetz, Ulrich Klank, Ingo Kresse, Alexis Maldonado, Lorenz Mösenlechner, Dejan Pangercic, Thomas Rühr, Moritz Tenorth), In 11th IEEE-RAS International Conference on Humanoid Robots, 2011. [bib] [pdf]
Transparent Object Detection and Reconstruction on a Mobile Platform (Ulrich Klank, Daniel Carton, Michael Beetz), In IEEE International Conference on Robotics and Automation (ICRA), 2011. [bib] [pdf]
Contracting Curve Density Algorithm for Applications in Personal Robotics (Shulei Zhu, Dejan Pangercic, Michael Beetz), In 11th IEEE-RAS International Conference on Humanoid Robots, 2011. [bib] [pdf]
Combining Perception and Knowledge Processing for Everyday Manipulation (Dejan Pangercic, Moritz Tenorth, Dominik Jain, Michael Beetz), In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2010. [bib] [pdf]
Automatic Layered 3D Reconstruction of Simplified Object Models for Grasping (Lucian Cosmin Goron, Zoltan Csaba Marton, Gheorghe Lazea, Michael Beetz), In Joint 41st International Symposium on Robotics (ISR) and 6th German Conference on Robotics (ROBOTIK), 2010. [bib] [pdf]
General 3D Modelling of Novel Objects from a Single View (Zoltan-Csaba Marton, Dejan Pangercic, Nico Blodow, Jonathan Kleinehellefort, Michael Beetz), In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2010. [bib] [pdf]
Hierarchical Object Geometric Categorization and Appearance Classification for Mobile Manipulation (Zoltan-Csaba Marton, Dejan Pangercic, Radu Bogdan Rusu, Andreas Holzbach, Michael Beetz), In Proceedings of the IEEE-RAS International Conference on Humanoid Robots, 2010. [bib] [pdf]
Perception and Probabilistic Anchoring for Dynamic World State Logging (Nico Blodow, Dominik Jain, Zoltan-Csaba Marton, Michael Beetz), In 10th IEEE-RAS International Conference on Humanoid Robots, 2010. [bib] [pdf]
Acquisition of a Dense 3D Model Database for Robotic Vision (Muhammad Zeeshan Zia, Ulrich Klank, Michael Beetz), In International Conference on Advanced Robotics (ICAR), 2009. [bib] [pdf]
Real-time CAD Model Matching for Mobile Manipulation and Grasping (Ulrich Klank, Dejan Pangercic, Radu Bogdan Rusu, Michael Beetz), In 9th IEEE-RAS International Conference on Humanoid Robots, 2009. [bib] [pdf]
3D Model Selection from an Internet Database for Robotic Vision (Ulrich Klank, Muhammad Zeeshan Zia, Michael Beetz), In International Conference on Robotics and Automation (ICRA), 2009. [bib] [pdf]
Perception for Mobile Manipulation and Grasping using Active Stereo (Radu Bogdan Rusu, Andreas Holzbach, Rosen Diankov, Gary Bradski, Michael Beetz), In 9th IEEE-RAS International Conference on Humanoid Robots (Humanoids), 2009. [bib]
Partial View Modeling and Validation in 3D Laser Scans for Grasping (Nico Blodow, Radu Bogdan Rusu, Zoltan Csaba Marton, Michael Beetz), In 9th IEEE-RAS International Conference on Humanoid Robots (Humanoids), 2009. [bib] [pdf]
Visual Scene Detection and Interpretation using Encyclopedic Knowledge and Formal Description Logic (Dejan Pangercic, Rok Tavcar, Moritz Tenorth, Michael Beetz), In Proceedings of the International Conference on Advanced Robotics (ICAR)., 2009. [bib] [pdf]
An ASM Fitting Method Based on Machine Learning that Provides a Robust Parameter Initialization for AAM Fitting (Matthias Wimmer, Shinya Fujie, Freek Stulp, Tetsunori Kobayashi, Bernd Radig), In Proc. of the International Conference on Automatic Face and Gesture Recognition (FGR08), 2008. [bib] [pdf]
Persistent Point Feature Histograms for 3D Point Clouds (Radu Bogdan Rusu, Zoltan Csaba Marton, Nico Blodow, Michael Beetz), In Proceedings of the 10th International Conference on Intelligent Autonomous Systems (IAS-10), Baden-Baden, Germany, 2008. [bib] [pdf]
3D-Based Monocular SLAM for Mobile Agents Navigating in Indoor Environments (Dejan Pangercic, Radu Bogdan Rusu, Michael Beetz), In Proceedings of the 13th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), Hamburg, Germany, September 15-18, 2008. [bib] [pdf]
Towards 3D Object Maps for Autonomous Household Robots (Radu Bogdan Rusu, Nico Blodow, Zoltan-Csaba Marton, Alina Soos, Michael Beetz), In Proceedings of the 20th IEEE International Conference on Intelligent Robots and Systems (IROS), 2007. [bib] [pdf]

Workshop Papers

Modeling Cognitive Frames for Situations with Markov Logic Networks (William R. Murray, Dominik Jain), In Proceedings of the 8th International NLPCS Workshop: Human-Machine Interaction in Translation, Copenhagen Studies in Language 41, Samfundslitteratur, 2011. [bib]
Efficient Surface and Feature Estimation in RGBD (Zoltan-Csaba Marton, Dejan Pangercic, Michael Beetz), In RGB-D Workshop on 3D Perception in Robotics at the European Robotics (euRobotics) Forum, 2011. [bib] [pdf]
Voxelized Shape and Color Histograms for RGB-D (Asako Kanezaki, Zoltan-Csaba Marton, Dejan Pangercic, Tatsuya Harada, Yasuo Kuniyoshi, Michael Beetz), In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Workshop on Active Semantic Perception and Object Search in the Real World, 2011. [bib] [pdf]
Fast and Robust Object Detection in Household Environments Using Vocabulary Trees with SIFT Descriptors (Dejan Pangercic, Vladimir Haltakov, Michael Beetz), In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Workshop on Active Semantic Perception and Object Search in the Real World, 2011. [bib] [pdf]
Robotic Roommates Making Pancakes - Look Into Perception-Manipulation Loop (Michael Beetz, Ulrich Klank, Alexis Maldonado, Dejan Pangercic, Thomas Rühr), In IEEE International Conference on Robotics and Automation (ICRA), Workshop on Mobile Manipulation: Integrating Perception and Manipulation, 2011. [bib] [pdf]
EYEWATCHME - 3D Hand and object tracking for inside out activity analysis (Li Sun, Ulrich Klank, Michael Beetz), In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2009. CVPR 2009., volume , 2009. [bib] [doi]
Detecting and Segmenting Objects for Mobile Manipulation (Radu Bogdan Rusu, Andreas Holzbach, Gary Bradski, Michael Beetz), In Proceedings of IEEE Workshop on Search in 3D and Video (S3DV), held in conjunction with the 12th IEEE International Conference on Computer Vision (ICCV), 2009. [bib]
Reconstruction and Verification of 3D Object Models for Grasping (Zoltan Csaba Marton, Lucian Cosmin Goron, Radu Bogdan Rusu, Michael Beetz), In Proceedings of the 14th International Symposium on Robotics Research (ISRR09), 2009. [bib] [pdf]
CoP-Man -- Perception for Mobile Pick-and-Place in Human Living Environments (Michael Beetz, Nico Blodow, Ulrich Klank, Zoltan Csaba Marton, Dejan Pangercic, Radu Bogdan Rusu), In Proceedings of the 22nd IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) Workshop on Semantic Perception for Mobile Manipulation, 2009. (Invited paper.) [bib] [pdf]
Towards a Plan Library for Household Robots (Armin Müller, Michael Beetz), In Proceedings of the ICAPS'07 Workshop on Planning and Plan Execution for Real-World Systems: Principles and Practices for Planning in Execution, 2007. [bib] [pdf]

Other Publications

Creating and using RoboEarth object models (Daniel Di Marco, Andreas Koch, Oliver Zweigle, Kai Häussermann, Björn Schieß le, Paul Levi, Dorian Galvez Lopez, Luis Riazuelo, Javier Civera, J.M.M Montiel, Moritz Tenorth, Alexander Clifford Perzylo, Markus Waibel, Marinus Jacobus Gerardus van de Molengraft), IEEE International Conference on Robotics and Automation (ICRA), 2012. [bib]
Everyday Perception for Mobile Manipulation in Human Environments (Ulrich Klank), PhD thesis, Technische Universität München, 2012. [bib]
A Robot that Shops for and Stores Groceries (Dejan Pangercic, Koppany Mathe, Zoltan-Csaba Marton, Lucian Cosmin Goron, Monica-Simona Opris, Martin Schuster, Moritz Tenorth, Dominik Jain, Thomas Ruehr, Michael Beetz), AAAI Video Competition (AIVC 2011), 2011. [bib]
Transformational Planning for Autonomous Household Robots using Libraries of Robust and Flexible Plans (Armin Müller), PhD thesis, Technische Universität München, 2008. [bib] [pdf]
Powered by bibtexbrowser
Export as PDF or BIB

Last edited 01.08.2011 15:27 by tenorth