The project aims at the unification of vision-based sensing in the CoP (Cognitive Perception) in the learning and planning system. On the one hand, CoP manages the interpretation of different kinds of sensors and, on the other hand, it automatically acquires and maintains the knowledge about the world and objects in the world. CoP selects sensors and sensor interpretation algorithms based on their expected utility. To this end, CoP learns and improves inter-sensor and inter-algorithmic models for method selection from experience. Especially the vision system, the major sensor we use, provides several automatic model improving techniques. Improved models accelerate the perception process and provide more robust results.
We showed the capabilities of the cognitive perception by making pancakes with our robot Rosie, supported by the PR2. The following perception tasks were part of preparation task:
We are developing methods for multi-modal object detection. Using different approaches, we are searching for daily-life objects in the context of the autonomous kitchen scenario. The target of the project is to enable vision-guided grasping of such objects. Therefore, we implement an intelligent parametrization of existing algorithms as well as new approaches, if necessary. The upper image on the right shows a screenshot of CoP analyzing a complex scenario. The lower images show the results of a student's Bachelor's thesis. A textured object is detected using a GPU-based implementation of the Randomized Tree algorithm by V. Lepetit.
Please note the related software projects: http://www.ros.org/wiki/cop
We are also applying classification techniques to improve the localization. Given for example a 3D segmentation, combining 3D and feature it is possible to classify some of most usual objects appearing in table setting scenes.
The first image on the right shows such a classification published at IROS 09. The lower image shows a result of a student's Bachelor's thesis, classifying camera images of branded objects only using training data acquired by image search on the Internet.
We are working on a method for automatically accessing an Internet database of 3D models that are searchable only by their user-annotated labels, in order to use them for vision and robotic manipulation purposes. Instead of having only a local database containing already seen objects, we want to use shared databases available over the Internet.
This approach, while having the potential to dramatically increase the visual recognition capabilities of robots, also poses certain problems, like wrong annotation due to the open nature of the database, or overwhelming amounts of data (many 3D models) or the lack of relevant data (no models matching a specified label).
To solve those problems we propose the following: First, we present an outlier/inlier classification method for reducing the number of results and discarding invalid 3D models that do not match our query. Second, we utilize an approach from computer graphics, the so called ’morphing’, to this application to specialize the models, in order to describe more objects. Third, we search for 3D models using a restricted search space, as obtained from our knowledge of the environment.
Inside-out action analysis shows new aspects in observation of daily life activities. Especially, manipulation tasks can be understood easier by an observer seeing the view of the acting person. The gaze provides early information about focus of the attention and critical points during an action. Any action is prepared by observing the area of interest before manipulating it. Additionally, humans mostly keep their focus on their hands while grasping an object. This fact allows us to automatically extract hand movements out of images of a gaze directed camera in relation to the acting person and the position in the world, that can be tracked in parallel, given an adequate world model. We propose in this work a novel hybrid 2D-3D method for markerless hand tracking under the difficult conditions such a gaze directed camera imposes: gaze changes with very high speed from one scene to another, with the effect of blurred frames and and little scene overlaps. So, the major problem that we address is the case of losing track of a hand. Given such an event the image is scanned for appearing hands until a hand’s present is validated over several frames. This validation is performed using a simple 2d hand model that equally serves initializing an more exact 3d hand model that is used to track the hand till the next track loss. Based on several distance measurement, we decide for an optimal hand configuration, as well in 2D as in 3D. The distance measurements are based on skin color, edges and edge directions. Our investigations are targeting as well on action analyzing in general as for generating a handbook of hand trajectories, grasping points and gaze directions in relation to the object of interest and possibly obstacles for controlling a service robot.
This project was partly funded by CoTeSys.
|Furniture Models Learned from the WWW -- Using Web Catalogs to Locate and Categorize Unknown Furniture Pieces in 3D Laser Scans , In Robotics & Automation Magazine, IEEE, volume 18, 2011. [bib] [pdf]|
|Combined 2D-3D Categorization and Classification for Multimodal Perception Systems , In The International Journal of Robotics Research, Sage Publications, volume 30, 2011. [bib] [pdf]|
|Importance Sampling as One Solution to the Data Association Problem in Multi-target Tracking , Chapter in VISIGRAPP 2009 (AlpeshKumar Ranchordas, Helder Araujo, eds.), Springer-Verlag Berlin Heidelberg, 2010. [bib]|
|Automatic feature generation in endoscopic images , In International Journal of Computer Assisted Radiology and Surgery, Springer, volume 3, 2008. [bib]|
|The Contracting Curve Density Algorithm: Fitting Parametric Curve Models to Images Using Local Self-adapting Separation Criteria , In International Journal of Computer Vision, volume 59, 2004. [bib] [pdf]|
|Towards Modular Spatio-temporal Perception for Task-adapting Robots , In Postgraduate Conference on Robotics and Development of Cognition (RobotDoC-PhD), a satellite event of the 22nd International Conference on Artificial Neural Networks (ICANN), 2012. [bib]|
|Object Categorization in Clutter using Additive Features and Hashing of Part-graph Descriptors , In Proceedings of Spatial Cognition (SC), 2012. [bib] [pdf]|
|Robots that Validate Learned Perceptual Models , In IEEE International Conference on Robotics and Automation (ICRA), 2012. [bib] [pdf]|
|Segmenting Cylindrical and Box-like Objects in Cluttered 3D Scenes , In 7th German Conference on Robotics (ROBOTIK), 2012. [bib] [pdf]|
|Efficient Part-Graph Hashes for Object Categorization , In 5th International Conference on Cognitive Systems (CogSys), 2012. [bib] [pdf]|
|Multimodal Autonomous Tool Analyses and Appropriate Application , In 11th IEEE-RAS International Conference on Humanoid Robots, 2011. [bib] [pdf]|
|Robotic Roommates Making Pancakes , In 11th IEEE-RAS International Conference on Humanoid Robots, 2011. [bib] [pdf]|
|Transparent Object Detection and Reconstruction on a Mobile Platform , In IEEE International Conference on Robotics and Automation (ICRA), 2011. [bib] [pdf]|
|Contracting Curve Density Algorithm for Applications in Personal Robotics , In 11th IEEE-RAS International Conference on Humanoid Robots, 2011. [bib] [pdf]|
|Combining Perception and Knowledge Processing for Everyday Manipulation , In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2010. [bib] [pdf]|
|Automatic Layered 3D Reconstruction of Simplified Object Models for Grasping , In Joint 41st International Symposium on Robotics (ISR) and 6th German Conference on Robotics (ROBOTIK), 2010. [bib] [pdf]|
|General 3D Modelling of Novel Objects from a Single View , In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2010. [bib] [pdf]|
|Hierarchical Object Geometric Categorization and Appearance Classification for Mobile Manipulation , In Proceedings of the IEEE-RAS International Conference on Humanoid Robots, 2010. [bib] [pdf]|
|Perception and Probabilistic Anchoring for Dynamic World State Logging , In 10th IEEE-RAS International Conference on Humanoid Robots, 2010. [bib] [pdf]|
|Acquisition of a Dense 3D Model Database for Robotic Vision , In International Conference on Advanced Robotics (ICAR), 2009. [bib] [pdf]|
|Real-time CAD Model Matching for Mobile Manipulation and Grasping , In 9th IEEE-RAS International Conference on Humanoid Robots, 2009. [bib] [pdf]|
|3D Model Selection from an Internet Database for Robotic Vision , In International Conference on Robotics and Automation (ICRA), 2009. [bib] [pdf]|
|Perception for Mobile Manipulation and Grasping using Active Stereo , In 9th IEEE-RAS International Conference on Humanoid Robots (Humanoids), 2009. [bib]|
|Partial View Modeling and Validation in 3D Laser Scans for Grasping , In 9th IEEE-RAS International Conference on Humanoid Robots (Humanoids), 2009. [bib] [pdf]|
|Visual Scene Detection and Interpretation using Encyclopedic Knowledge and Formal Description Logic , In Proceedings of the International Conference on Advanced Robotics (ICAR)., 2009. [bib] [pdf]|
|An ASM Fitting Method Based on Machine Learning that Provides a Robust Parameter Initialization for AAM Fitting , In Proc. of the International Conference on Automatic Face and Gesture Recognition (FGR08), 2008. [bib] [pdf]|
|Persistent Point Feature Histograms for 3D Point Clouds , In Proceedings of the 10th International Conference on Intelligent Autonomous Systems (IAS-10), Baden-Baden, Germany, 2008. [bib] [pdf]|
|3D-Based Monocular SLAM for Mobile Agents Navigating in Indoor Environments , In Proceedings of the 13th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), Hamburg, Germany, September 15-18, 2008. [bib] [pdf]|
|Towards 3D Object Maps for Autonomous Household Robots , In Proceedings of the 20th IEEE International Conference on Intelligent Robots and Systems (IROS), 2007. [bib] [pdf]|
|Modeling Cognitive Frames for Situations with Markov Logic Networks , In Proceedings of the 8th International NLPCS Workshop: Human-Machine Interaction in Translation, Copenhagen Studies in Language 41, Samfundslitteratur, 2011. [bib]|
|Efficient Surface and Feature Estimation in RGBD , In RGB-D Workshop on 3D Perception in Robotics at the European Robotics (euRobotics) Forum, 2011. [bib] [pdf]|
|Voxelized Shape and Color Histograms for RGB-D , In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Workshop on Active Semantic Perception and Object Search in the Real World, 2011. [bib] [pdf]|
|Fast and Robust Object Detection in Household Environments Using Vocabulary Trees with SIFT Descriptors , In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Workshop on Active Semantic Perception and Object Search in the Real World, 2011. [bib] [pdf]|
|Robotic Roommates Making Pancakes - Look Into Perception-Manipulation Loop , In IEEE International Conference on Robotics and Automation (ICRA), Workshop on Mobile Manipulation: Integrating Perception and Manipulation, 2011. [bib] [pdf]|
|EYEWATCHME - 3D Hand and object tracking for inside out activity analysis , In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2009. CVPR 2009., volume , 2009. [bib] [doi]|
|Detecting and Segmenting Objects for Mobile Manipulation , In Proceedings of IEEE Workshop on Search in 3D and Video (S3DV), held in conjunction with the 12th IEEE International Conference on Computer Vision (ICCV), 2009. [bib]|
|Reconstruction and Verification of 3D Object Models for Grasping , In Proceedings of the 14th International Symposium on Robotics Research (ISRR09), 2009. [bib] [pdf]|
|CoP-Man -- Perception for Mobile Pick-and-Place in Human Living Environments , In Proceedings of the 22nd IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) Workshop on Semantic Perception for Mobile Manipulation, 2009.(Invited paper.) [bib] [pdf]|
|Towards a Plan Library for Household Robots , In Proceedings of the ICAPS'07 Workshop on Planning and Plan Execution for Real-World Systems: Principles and Practices for Planning in Execution, 2007. [bib] [pdf]|
|Creating and using RoboEarth object models , IEEE International Conference on Robotics and Automation (ICRA), 2012. [bib]|
|Everyday Perception for Mobile Manipulation in Human Environments , PhD thesis, Technische Universität München, 2012. [bib]|
|A Robot that Shops for and Stores Groceries , AAAI Video Competition (AIVC 2011), 2011. [bib]|
|Transformational Planning for Autonomous Household Robots using Libraries of Robust and Flexible Plans , PhD thesis, Technische Universität München, 2008. [bib] [pdf]|