User Tools

Site Tools


RoboHow.Cog: Web-enabled and Experience-based Cognitive Robots that Learn Complex Everyday Manipulation Tasks

The full potential of robots cannot be reached without the ability to physically interact with its environment and humans. Without this ability, robots will never go beyond tasks such as being museum guides or transporting goods. In the real world, a service robot will have to autonomously and skillfully perform a large and steadily expanding set of human-scale everyday tasks for which it cannot be pre-programmed, as they are not known beforehand. This is in contrast to the traditional industrial setting, where the robot repeats the same tasks many times, but even in this domain, there is a demand for simpler and more cost-efficient ways of making the robot perform new tasks.

RoboHow: A glimpse into the future of autonomous robot manipulation. Consider a robot that has to perform a task it has not been programmed for – let us say making a pancake. To make a pancake, the robot needs a set of instructions that can be found on web pages such as Being made for human rather than robot use, these instructions are typically incomplete, vague, ambiguous and require appropriate interpretation. To resolve the vagueness and ambiguities, the robot could also watch instructional videos demonstrating how humans make pancakes. However, acquiring procedural knowledge is not enough, since it also requires detection and recognition of tools and ingredients needed for making pancakes. Making pancakes also requires advanced actions such as pouring pancake mix in the frying pan and detect if spillage or similar problems have occurred. The robot must also push the spatula under the pancake in order to flip it, an action which must be performed with the appropriate force. Although rather visionary, this example demonstrates several open challenges.

The Challenges. Robots that are to perform human-scale activities will get vague instructions such as ``stir the pancake mix until the texture is smooth'' without stating the exact trajectory that one may have to follow or the speed and force to apply while stirring. The robot must decide on how to perform the task by performing the appropriate actions on the appropriate objects in the appropriate ways. System developers are required to take the "open world" challenge seriously, i.e. they are to endow the robot system with mechanisms to acquire new skills for known and novel tasks. Thus, enabling robots to competently perform everyday manipulation activities such as household chores exceeds, in terms of task, activity, behavior and context complexity, anything that we have so far investigated or successfully implemented in motion planning, cognitive robotics, autonomous robot control and artificial intelligence at large.

The Vision. The vision of Robohow is that of a cognitive robot that autonomously performs complex everyday manipulation tasks and extends its repertoire of such by acquiring new skills using web-enabled and experience-based learning as well as by observing humans.

The Goal. Robohow aims at the realization of a programming framework that enables programmers to semi-automatically expand autonomous service robot applications to perform advanced human-scale manipulation and meal preparation tasks with little effort.

The Approach. The Robohow programming framework will represent control programs as concurrent, percept-guided manipulation plans that can be generated semi-automatically by extracting and processing formal knowledge from large existing information sources. Robohow will use websites, visual instructions, and haptic demonstration as primary sources of information. The heterogeneous pieces of information will be integrated into the robot control system and combined with each other through an interface layer that provides an abstract machine for programming high-level robot manipulation plans. The interpreter for this abstract machine includes novel mechanisms for the constraint- and optimization-based movement specification and execution and for the realization of powerful perception mechanisms for perception-guided manipulation.

The Expected Results and Impacts. Robohow's main scientific result will be a methodology for how robots can be improved and improve themselves under "open world" conditions. The software with which this scientific result has been verified will be packaged and documented together with a development process tool chain and made available to the community. In addition, existing open source robotics software will be refactored and extended such that they support this and makes the result easily accessible to the research community. The technologies will result in a new generation of manipulation robots that will be applicable to a wide range of application domains that require knowledge-intensive and open-world robot systems.

research/robohow.txt · Last modified: 2011/11/14 12:32 by jain