knowledge processing for autonomous robots · 2012. 6. 13. · roboearth: concept, interface to...

Knowledge Processing for AutonomousRobots

Moritz TenorthIntelligent Autonomous Systems GroupTechnische Universitat Munchen

June 13, 2012

Knowing where to look for something

◮ Naive:Fixed controller for opening everything with a handle



◮ Parameterizable:Look up the positions of all containers in the map



◮ Parameterizable:Look up the positions of all containers in the map

◮ Knowledge-based:Infer the right container based on the types of the object and thecontainer

Knowing where to look for something[Humanoids 2011]

storagePlaceFor

instanceOf

instanceOf

Knowledge-based decision making

◮ Formulate control decisions as inference tasks

◮ Separate control flow from execution context

◮ Increase re-usability of robot plans

CRAM: Cognitive Robot Abstract Machine

Outline

◮ Wednesday: Knowledge representation (KnowRob, ProbCog)

◮ Thursday morning: CRAM execution framework

◮ Thursday afternoon: Semantic perception methods

◮ Friday: Group discussions about specific project ideas

Outline: KnowRob tutorial

◮ Introduction: What can CRAM, KnowRob, etc be used for?

◮ Languages: Prolog, OWL, Java, etc

◮ Basics: launching KnowRob, loading files, querying, debugging

◮ Objects: class taxonomy, instances, components, environmentmodels, units and coordinates, qualitative spatial relations,computable properties

◮ Actions: class-level descriptions, properties, projection, planning,action dependencies

◮ RoboEarth: concept, interface to KnowRob, up- and download ofinformation, capability matching

◮ ProbCog: interface to statistical relation inference (Dominik)

What I won’t talk much about...

◮ Time: Reasoning about time points and intervals, object posesthat change over time

◮ Changing objects: Creation/destruction/modification of objectsthrough actions and processes

◮ Interfaces to external components: Perceptual methods, SQLdatabases, Web information

◮ Human activity data: segmentation/classification/abstraction/reasoning about observations of human activities

◮ Machine learning: clustering and classification using Weka andMallet libraries

KnowRob components

The KnowRob knowledge processing system

KnowRob: A practical robot knowledge base[IROS 2009, RAS 2010]

Representations for robot knowledge

Semantic object and environment models[Humanoids 2010, IROS 2011, RAS 2012]

◮ Recognized objects andenvironment map represented inthe knowledge base

◮ Pieces of furniture as objectinstances that inherit propertiesof their types

◮ Articulation models for openingcontainers

◮ Spatio-temporal representation ofobject poses

Image courtesy of Ulrich Klank

Actions and processes◮ Actions move, split, destroy, create, join,

switch on, open and close objects

◮ Processes are started as indirect effectsof actions

◮ Integrated representation to jointlyreason about actions and processes

◮ Prediction of their effects usingqualitative high-level projection rules

Composing actions to plans[ICRA 2012]

Web-enabled knowledge bases

Task instructions from the WWW[ICRA 2010]

semantic

parsing

WordNet

lexical

database

OpenCyc

ontology

word

sense

resolution

ontology

mapping

robot

plan

generation

S

VP

VB NP

IN NP

DT NN

DT NN

PP

place/PuttingSthSomewherecup/DrinkingMugtable/Table-PieceOfFurniture

place/ cup/ table/

N08664443,V01494310,V02392762,...N03147509,N13619168,...N08266235,N04379243

Object models

◮ Automatically created ontologyof >7500 objects from theonline shop germandeli.com

◮ Class hierarchy from categories+ perishability, weight, price,origin, ...

◮ SIFT recognition models fromproduct pictures (work byDejan Pangercic)

[RAM 2011a]

Understanding observations of human actions

Acquiring knowledge by observing humans[T&D 2010]

Image courtesy of Jan Bandouch

◮ Gather knowledge from observing human activities

◮ Interpret the data to obtain reusable information

Segmentation and abstraction of motions[ICCV THEMIS 2009]

◮ Conditional Random Fields-based motion segmentation

◮ Pose-related features and data from external sensors

Extracting knowledge by observing humans[ICRA 2011]

Integrated experiments

Completing underspecified instructions[Humanoids 2011]

Comparing observations with WWW instructions

Inferring missing objects[IROS 2010]

◮ Given: incomplete table setup◮ Objective: infer which objects are additionally needed◮ Integrates perception with statistical relational models of human

preferences(see also comp missingobj, prolog perception, mod probcog)

Why Prolog for autonomous robots?

◮ Can be read declaratively

◮ Can be programmed (prodecurally)

◮ There are textbooks and tutorials

◮ Industrial-strength open-source implementations

◮ Foreign language interfaces

◮ Result interpretation: similar to a DB query language

KnowRob: based on SWI Prolog (http://www.swi-prolog.org)

Which language for what purpose?

OWL:

◮ Class taxonomy of objects, actions, events,...

◮ Instances of these classes (e.g. environment models, experiences)

◮ Robot capabilities/action requirements

Prolog:

◮ Internal representation (OWL parsed into Prolog triples)

◮ Query language

◮ Inference predicates (DL inference, computables,...)

Java:

◮ External interfaces (WWW, ROS,...)

◮ Library integration (ProbCog, Weka,...)

Prolog syntax

◮ Predicate names and constants start with a lowercase letter or areinside single quotes

◮ Variables start with an uppercase letter

◮ Formulas end with a full stop ’.’

◮ Stepping through all results of a query with ’;’

◮ Logical operators:

Operator Prolog equivalent Example∧ , type(A, ’Cup’), on(A, T).∨ ; type(A, ’Food’); type(A, ’Drink’).

Example

dairyproduct(milk).

perishable(Stuff) :- storedIn(Stuff, Ref), refrigerator(Ref).

Introduction to Prolog

milk.pl

cup(cup0).

dairyProduct(milk1).

meatProduct(ham2).

cupboard(cupboard3).

refrigerator(fridge4).

perishable(Prod) :- dairyProduct(Prod); meatProduct(Prod).

storagePlaceFor(Loc, Item) :- refrigerator(Loc), perishable(Item).

storagePlaceFor(Loc, Item) :- cupboard(Loc), cup(Item).

searchForIn(Item,Loc) :- storagePlaceFor(Loc, Item).

Example queries

Two main types of queries:

◮ Query with unbound variables → variable bindings

◮ All variables bound → true/false result

Example queries

$ prolog -f milk.pl

?- refrigerator(A).

A = fridge4.

?- refrigerator(fridge4).

true.

?- searchForIn(milk1, A).

A = fridge4 .

Example queries

Example queries

?- storagePlaceFor(A, milk1).

A = fridge4 .

?- storagePlaceFor(fridge4, A).

A = milk1 ;

A = ham2 ;

?- storagePlaceFor(A, B).

A = fridge4,

B = milk1 ;

A = fridge4,

B = ham2 ;

A = cupboard3,

B = cup0.

Under the hood: the tracer

Tracing a query

?- trace, searchForIn(milk1, P).

Call: (7) searchForIn(milk1, G369) ? creep

milk.pl

searchForIn(Item, Loc) :- storagePlaceFor(Loc, Item).


Tracing a query



. Call: (8) storagePlaceFor( G369, milk1) ? creep

milk.pl




Tracing a query




. . Call: (9) refrigerator( G369) ? creep

milk.pl



Tracing a query





. . Exit: (9) refrigerator(fridge4) ? creep

milk.pl



Tracing a query






. . Call: (9) perishable(milk1) ? creep

milk.pl




Tracing a query







. . . Call: (10) dairyProduct(milk1) ? creep

milk.pl




Tracing a query








. . . Exit: (10) dairyProduct(milk1) ? creep

milk.pl




Tracing a query









. . Exit: (9) perishable(milk1) ? creep

milk.pl




Tracing a query










. Exit: (8) storagePlaceFor(fridge4, milk1) ? creep

milk.pl




Tracing a query











Exit: (7) searchForIn(milk1, fridge4) ? creep

milk.pl



Tracing a query











Exit: (7) searchForIn(milk1, fridge4) ? creep

P = fridge4 .

milk.pl


Why is this useful for robotics?

◮ Separation of the inference task (what to infer) from theimplementation (how to infer it) and the data (what evidence toinfer it from)

◮ Search-based inference: easy integration of additional inferencemethods or knowledge sources as additional branches in the searchtree




◮ Basics: launching KnowRob, loading files, querying,debugging





Launching KnowRob

◮ KnowRob is very modular: Dependencies are automatically loadedby rosprolog

◮ General syntax: rosrun rosprolog rosprolog <pkg>

◮ starts a Prolog environment◮ checks for dependencies of <pkg>◮ locates them on the disk◮ loads the prolog/init.pl file for each of these packages

◮ For today we use:rosrun rosprolog rosprolog knowrob tutorial

TBOX queries

◮ The TBOX contains terminological statements: abstractdescriptions of classes and properties

◮ owl subclass of(Sub, Super). reads all classes derived from Super

◮ rdfs subproperty of(SubProp, Prop). lists sub-properties of Prop

◮ owl individual of(Restr, owl:’Restriction’), owl subclass of(Class,

Restr). reads all restrictions that are defined on Class

◮ owl satisfies restriction(Res, Restr). verifies whether Res satisfiesthe restriction Restr

◮ owl description(RestrOWL, RestrProlog). translates the OWL identifierof a restriction into a Prolog term

(see also http://www.swi-prolog.org/pldoc/package/semweb.html)

Example queries: TBOX

Example

$ rosrun rosprolog rosprolog knowrob tutorial

?- owl subclass of(A, knowrob:’FoodOrDrink’).

A = ’http://ias.cs.tum.edu/kb/knowrob.owl#FoodOrDrink’ ;

A = ’http://ias.cs.tum.edu/kb/knowrob.owl#Drink’ ;

A = ’http://ias.cs.tum.edu/kb/knowrob.owl#Coffee-Beverage’ ;

A = ’http://ias.cs.tum.edu/kb/knowrob.owl#InfusionDrink’ ;

A = ’http://ias.cs.tum.edu/kb/knowrob.owl#Tea-Beverage’ ;

A = ’http://ias.cs.tum.edu/kb/knowrob.owl#Tea-Iced’

ABOX queries

◮ The ABOX contains assertional statements: concrete instances ofthe abstract classes and relations between them

◮ Query predicates:◮ rdf(S, P, O) returns only exactly matching triples◮ rdf has(S, P, O) also takes the subPropertyOf relation into account,

returning matches for all specializations of P◮ rdf reachable(S, P, O) further considers transitivity of P◮ rdf triple(P, S, O) additionally includes computables◮ owl has(S, P, O) includes OWL inference results (e.g. inferred class

membership of an individual)

Important: Most predicates can be used with different combinations offree and bound variables:

◮ owl individual of(Ind, knowrob:’Cup’) reads all instances of class Cup

◮ owl individual of(knowrob:’cup0’, Class) reads all classes of cup0

Example queries: ABOX

Example

?- owl has(A, rdf:type, knowrob:’Drawer’).

A = ’http://ias.cs.tum.edu/kb/knowrob.owl#Drawer1’ ;


?- owl has(knowrob:’Drawer1’, P, O).

P = ’http://www.w3.org/1999/02/22-rdf-syntax-ns#type’,

O = ’http://ias.cs.tum.edu/kb/knowrob.owl#Drawer’ ;

P = ’http://ias.cs.tum.edu/kb/knowrob.owl#widthOfObject’,

O = literal(type(xsd:float, ’0.58045006’)) ;

P = ’http://ias.cs.tum.edu/kb/knowrob.owl#properPhysicalParts’,

O = ’http://ias.cs.tum.edu/kb/knowrob.owl#Door4’ ;

Querying KnowRob from your program

◮ Tell/ask interface via a ROS service◮ tell: add statements◮ ask: send a query

◮ Send queries as string like you would do from a console

◮ Result serialized using json

◮ Client libraries for C++, Python, Java, Lisp available

Example json prolog launch file

<launch>

<node name="knowrob" pkg="rosprolog" type="run with prolog env"

args="demo cotesys fall2010 $(find json prolog)/bin/json prolog" />

</launch>

(see also http://ros.org/wiki/json prolog)

Querying KnowRob from Python

knowrob/json prolog/examples/test json prolog.py

#/usr/bin/env python

import roslib; roslib.load manifest(’json prolog’)

import rospy

import json prolog

if name == ’ main ’:

rospy.init node(’test json prolog’)

prolog = json prolog.Prolog()

query = prolog.query("member(A, [1, 2, 3, 4]), B = [’x’, A]")

for sol in query.solutions():

print ’Found solution. A = %s, B = %s’ % (sol[’A’], sol[’B’])

query.finish()

Querying KnowRob from C++

knowrob/json prolog/examples/test json prolog.cpp

#include <string>

#include <iostream>

#include <ros/ros.h>

#include <json prolog/prolog.h>

using namespace std;

using namespace json prolog;

int main(int argc, char *argv[]) {

ros::init(argc, argv, "test json prolog");

Prolog pl;

PrologQueryProxy bdgs = pl.query("member(A, [1, 2, 3]), B = [’x’, A]");

for(PrologQueryProxy::iterator it=bdgs.begin(); it != bdgs.end(); it++) {

PrologBindings bdg = *it;

cout << "Found solution: " << (bool)(it == bdgs.end()) << endl;

cout << "A = "<< bdg["A"] << endl;

cout << "B = " << bdg["B"] << endl;

}return 0;

}

Workspace setup and debugging

◮ Global history: put the following .plrc file in your home

.plrc

rl write history :-

expand file name("∼/.pl-history", [File| ]),

rl write history(File).

:- ( current prolog flag(readline, true) ->

expand file name("∼/.pl-history", [File| ]),

(exists file(File) -> rl read history(File); true),

at halt(rl write history) ; true ).

◮ Debugging: graphical tracer available in SWI Prolog

Graphical tracer

?- guitracer.

?- trace.

(see also http://www.swi-prolog.org/pldoc/refman, Section 3.5)

Exercises (45min):

KnowRob ontology walkthrough: Overview of main classes andproperties in http://ias.in.tum.de/kb/knowrob.owl

Structure of a KnowRob ROS package

KnowRob basics tutorial:http://ias.in.tum.de/kb/wiki/index.php/Tutorial: KnowRob basics

Loading OWL files and ROS modules:http://ias.in.tum.de/kb/wiki/index.php/Loading files and ROS packages

Compile and run a json prolog example client in your favorite language(launch the KnowRob service with roslaunch knowrob tutorialtutorial.launch)

Further links

API documentation of KnowRob packages:http://ias.cs.tum.edu/kb/api/<pkg-name>/

Integration of common DL reasoners (e.g. for inferring class subsumption)http://ias.in.tum.de/kb/wiki/index.php/OWL reasoners via OWLAPIhttp://owlapi.sourceforge.net/documentation.html

PLDOC source code documentation:http://ias.in.tum.de/kb/wiki/index.php/Source code documentation with PLDOC

Prolog documentation (incl. semweb library and thea OWL library)http://www.swi-prolog.org/pldoc/refman/http://www.swi-prolog.org/pldoc/package/semweb.htmlhttp://www.semanticweb.gr/thea/index.html

Matlab interface (possibly outdated)http://ias.in.tum.de/kb/wiki/index.php/Using MATLAB from Prolog





◮ Objects: class taxonomy, instances, components,environment models, units and coordinates, qualitativespatial relations, computable properties




Object representation

Objectives:

◮ Class-level knowledge about object types

◮ Instance-level representations of object configurations

◮ Metric as well as qualitative spatial information (+ conversion)

◮ Explicit representation of units and coordinates (+ conversion)

◮ Nominal object locations in an environment

◮ Changes of locations over time

(see also http://ias.in.tum.de/kb/wiki/index.php/Reasoning about objects)

Visualizing objects

◮ A java-based visualization canvas can display objects and theirlocations

◮ visualization canvas(C). starts the canvas and stores a handle in thevariable C

◮ $C refers to the last binding of the variable C

◮ add object(Identifier, $C). sends an object instance to the canvas inorder to visualize it

(see also API documentation: http://ias.cs.tum.edu/kb/api/mod vis/)

Example queries: Visualization

Example

?- owl has(A, rdf:type, knowrob:’Drawer’),

add object with children(A, $C).



Semantic map representation

Abstract knowledgeabout object classes

Object instances and componenthierarchy

Poses in the environmentand their changes over time

Related: TBOX/SBOX, Galindo et al (RAS 2008)




Poses in the environment

and their changes over time

Spatio-temporal object representation

(see also

http://ias.in.tum.de/kb/wiki/index.php/Modeling perceived or inferred objects)

Integrating perception

(see also

http://ias.in.tum.de/kb/wiki/index.php/Modeling perceived or inferred objects)



Object instances andcomponent hierarchy



Representing articulated objects

◮ Poses and joint limits of hinges and prismatic joints

◮ Enable robot to open doors and drawers

Object part composition

◮ Part-of hierarchy: composition of objects from parts

◮ Hinged-to/fixed-to: kinematic properties

Integrating CAD models

◮ Visualization and computation of qualitative spatial relations

◮ Soon: segmentation and interpretation of object parts(see also http://ias.in.tum.de/kb/wiki/index.php/CAD Models)

Computing qualitative spatial relations

(see also http://ias.in.tum.de/kb/wiki/index.php/Tutorial: Computables)

Computables

◮ Procedural attachment to OWL classes and properties◮ Computable properties:

◮ Calculate if relation exists between two instances◮ Generate instances for which relation exists

◮ Computable classes:◮ Check for class subsumption◮ Generate individuals of a class

Defining computables

◮ Define OWL property (as usual incl. Domain and Range)Ob je c tP rope r t y : a f t e r

SubPropertyOf : t empo r a l l yR e l a t e dDomain : TimePointRange : TimePoint

◮ Write Prolog predicate that computes this relationcomp a f t e r ( Pre , A f t e r ) : -

ow l ha s ( Pre , type , ’ TimePoint ’ ) ,ow l ha s ( A f t e r , type , ’ TimePoint ’ ) ,t e rm to atom (P , Pre ) ,te rm to atom (A, A f t e r ) ,P<A.

◮ Attach this predicate to the OWL propertyI n d i v i d u a l : computeAfte r

Types : P r o l o gP rope r t yFac t s : t a r g e t a f t e r

command comp a f t e r(see also http://ias.in.tum.de/kb/wiki/index.php/Define computables)

Calling computables

◮ Computables can trigger complex calculations, include them only ifneeded

◮ Special predicate rdf triple(?P,?S,?O) that returns the union of◮ asserted knowledge (via owl has(?S,?P,?O))◮ results of computables (rdfs computable triple(?S,?P,?O))◮ provides a hook for custom predicates.

r d f t r i p l e ( Prop , Subj , Obj ) : -s u b p r o p e r t y o f ( SubProp , Prop ) ,( ow l ha s ( Subj , SubProp , Obj ) ;

r d f s c om p u t a b l e t r i p l e ( Subj , SubProp , Obj ) ;r d f t r i p l e h o o k ( Subj , SubProp , Obj ) ) .

(see also http://ias.in.tum.de/kb/wiki/index.php/Tutorial: Computables)

Video: Unpacking a shopping basket

Thanks to Dejan Pangercic, Mathe Koppany, Zoltan-Csaba Marton, Lucian Goron,Monica Opris and Thomas Ruhr for making this live demonstration possible.

Infer storage location based on generic class

knowledge

storagePlaceFor

instanceOf

instanceOf

Object ontology

◮ Automatically created ontologyof >7500 objects from theonline shop germandeli.com

◮ Class hierarchy from categories+ perishability, weight, price,origin, ...

◮ SIFT recognition models fromproduct pictures (work byDejan Pangercic)

[RAM 2011a]

(see also http://www.ros.org/wiki/comp germandeli)

Infer storage locations based on semantic object

similarity

SpatialThing

DrinkingVessel

Cup

Thing

Glass

CoffeeCup SodaGlass

Root

LCS

C1 C2

depth(LCS)

= 2

depth(C1)

= 4

depth(C2)

= 4

Learning Organizational Principles in Human Environments.

Martin Schuster, Dominik Jain, Moritz Tenorth and Michael Beetz. ICRA 2012

(see also http://www.ros.org/wiki/comp orgprinciples)

Infer storage locations based on semantic object

similarity

?- highlight best location dtree(

orgprinciples:’CoffeeFilter1’, Canvas).

Best location: knowrob:Drawer7

Objects at location knowrob:Drawer7:

WUP similarity: object (class)

0.87500: orgprinciples:CoffeGround1

(germandeli:Dallmayr Classic Ground Coffee 250g)

0.75000: orgprinciples:EspressoBeans1

(germandeli:illy Espresso Whole Beans 88 oz)

0.70588: orgprinciples:Sugar1

(germandeli:Nordzucker Brauner Teezucker 500g)

0.66667: orgprinciples:Tea2

(germandeli:Teekanne Rotbusch Tee Vanille 20 Bags)

Open the drawer where cups are stored

◮ Infer most likely storage location

◮ Read articulation model from semantic map

Check if objects are placed correctly[ICRA SPME 2011]

◮ Infer most likely storage location

◮ Compare with actual locations of objects

Select objects based on their purpose

◮ Combine abstract object knowledge with map information

◮ Here: “appliance that can be used for washing dishes”

Integrate with human tracking data

◮ Observations of humans are represented in the KB

◮ Reason about their interaction with objects in the map

Scaling up to large environmentsSearching Objects in Large-Scale Indoor Environments: A Decision-Theoretic Approach.

Lars Kunze, Michael Beetz, Manabu Saito, Haseru Azuma, Kei Okada, Masayuki Inaba. ICRA 2012

Exercises (45min):

Walkthrough: object pose representation

KnowRob objects tutorial:http://ias.in.tum.de/kb/wiki/index.php/Reasoning about objects

Additional information:http://ias.in.tum.de/kb/wiki/index.php/Measurement unitshttp://ias.in.tum.de/kb/wiki/index.php/Coordinate systemshttp://ias.in.tum.de/kb/wiki/index.php/Map data representationhttp://ias.in.tum.de/kb/wiki/index.php/CAD Modelshttp://ias.in.tum.de/kb/wiki/index.php/Mod semantic map






◮ Actions: class-level descriptions, properties, projection,planning, action dependencies



Action representation

Objectives:

◮ Different hierarchies: temporal composition, class taxonomy

◮ Action properties: toLocation, objectActedOn, bodyPartUsed,...

◮ Partial ordering of actions in a task

◮ Reason about effects of actions on objects

◮ Support projection and (simple) planning

◮ Generate executable robot plan from specification

◮ Specify requirements on robot capabilities or components

(see also http://ias.in.tum.de/kb/wiki/index.php/Reasoning about actions)

Reasoning about action sequences

◮ Generate initial task description from Web instructions


◮ Project action effects to infer interaction with objects

◮ Specific properties exactly describe object-action relation

◮ Reason about changes induced on objects


◮ Complete action sequences using background knowledge

◮ Infer storage location, opening trajectory, etc


◮ Detect knowledge gaps by comparing projected world state withrequired action inputs

◮ Apply planning to add required actions (see also knowrob actions package)


◮ Include physics-based reasoning to infer action parameterizations

◮ Work by Lars Kunze, Mihai Dolha

Action class representation

◮ Generic action class definition, incl. super-classes and sub-actions

C l a s s : Putt ingSomethingSomewhereSubClassOf :

Movement - T r a n s l a t i o nEv en tT r an spo r t a t i o nEven tsubAct i on some Pick ingUpAnObjectsubAct i on some Car ry ingWhi l eLocomot ingsubAct i on some PuttingDownAnObjecto r d e r i n gC o n s t r a i n t s v a l u e SubEventOrder ing1o r d e r i n gC o n s t r a i n t s v a l u e SubEventOrder ing2

◮ Specification of sub-actions and ordering constraints

I n d i v i d u a l : SubEventOrder ing1Types :

P a r t i a lO r d e r i n g - S t r i c tFac t s :

o c c u r sB e f o r e I nO r d e r i n g Pick ingUpAnObjecto c c u r sA f t e r I nO r d e r i n g Car ry ingWhi l eLocomot ing

Composing actions to tasks

C l a s s : SetATableAnnota t i on s : l a b e l ” s e t a t a b l e ”SubClassOf : Ac t i onEqu i va l en tTo :

subAct i on some PutP laceMat InFrontOfCha i rsubAct i on some PutP la te InCente rOfP laceMatsubAct i on some PutKn i f eR i gh tO fP l a t esubAct i on some [ . . . ]o r d e r i n gC o n s t r a i n t s v a l u e [ . . . ]

C l a s s : PutP laceMat InFrontOfCha i rEqu i va l en tTo :

Putt ingSomethingSomewhereob jectActedOn va l u e PlaceMat1t oLoca t i o n some Place1

C l a s s : P lace1Equ i va l en tTo :

i nFrontOf - G e n e r a l l y some Chai r - P i e c eO fFu r n i t u r e

I n d i v i d u a l : PlaceMat1Types : PlaceMat

Task specification

CRAM plan generation

( d e f - t o p - l e v e l - p l a n s e r v e - a - d r i n k ( )( w i t h - d e s i g n a t o r s (

( b o t t l e 1 ( ob j e c t ‘ ( ( name b o t t l e 1 )( type d r i n k i n g - b o t t l e ) ) ) )

( bed1 ( ob j e c t ‘ ( ( name bed1 )( type b e d p i e c e - o f - f u r n i t u r e ) ) ) )

( hand -pose -handove r1 ( l o c a t i o n ‘ ( ( on , bed1 ) ) ) )( r o bo t - p o s e - h andov e r 1 ( l o c a t i o n ‘ ( ( to r each )

( s i d e : r i g h t )( l o c , hand -pose -handove r1 ) ) ) )

( a rms-a t101 ( a c t i o n ‘ ( ( t ype t r a j e c t o r y )( pose , hand -pose -handove r1 )( s i d e : r i g h t ) ) ) )

( unhand -ac t i on102 ( a c t i o n ‘ ( ( t ype o p e n - g r i p p e r )( s i d e : r i g h t ) ) ) )

)

( a ch i e v e ‘ ( o b j e c t - i n - h a n d , b o t t l e 1 : r i g h t ) )( a t - l o c a t i o n ( r o bo t - p o s e - h andov e r 1 ) )( a ch i e v e ‘ ( a rms -a t , a rms-a t101 ) )( a ch i e v e ‘ ( a rms -a t , unhand -ac t i on102 ) ) ) )

Requirements of actions on robot capabilities

Objectives:

◮ Find out if robot can do a job or, if not, what is missing

◮ Technically: flexible matching between robot capabilities andaction requirements

Complex problem because of

◮ hierarchical structure of robot components

◮ temporal action hierarchy (task composed of sub-actions)

◮ inheritance hierarchy (requirements defined on higher levels)

◮ inter-dependencies (capabilities depend on other components orcapabilities)

◮ tasks do not usually come with requirement specs, they need to beinferred using prior knowledge

SRDL: Robot description

◮ Kinematic robot model imported from URDF◮ Links and joints, incl. coordinate transformations◮ linked by sub-properties of transitive subComponent property

◮ Manual semantic annotations: ’these links form a gripper’

◮ Robot capabilities, e.g. 2D navigation(see also http://ias.in.tum.de/kb/wiki/index.php/SRDL2 Tutorial)

SRDL: Query for available robot components

sub component ( Super , Sub ) : -ow l ha s ( Super , s rd l2comp : ’ subComponent ’ , Sub ) .

sub component ( Super , Sub ) : -ow l ha s ( Sub , s rd l2comp : ’ baseL inkOfCompos i t i on ’ , Base ) ,

sub component ( Super , Base ) ,ow l ha s ( Sub , s rd l2comp : ’ endL inkOfCompos i t i on ’ , End ) ,

sub component ( Super , End ) .

c omp t y p e a v a i l a b l e ( Super , SubT) : -sub component ( Super , Sub ) ,o w l i n d i v i d u a l o f ( Sub , SubT ) .

◮ Very simple queries for reading components and checkingavailability

◮ Query exploits transitivity of subComponent property

SRDL: Query for available capabilities

% c a p a b i l i t y a s s e r t e d f o r r obo t i n s t a n c ec a p a v a i l a b l e o n r o b o t (Cap , Robot ) : -ow l ha s ( Robot , s r d l 2 c a p : ’ h a s C a p a b i l i t y ’ , SubCap ) ,o w l s u b c l a s s o f ( SubCap , Cap ) .

% c a p a b i l i t y a s s e r t e d f o r r obo t c l a s sc a p a v a i l a b l e o n r o b o t (Cap , Robot ) : -r d f s i n d i v i d u a l o f ( Robot , RobotC la s s ) ,c l a s s p r o p e r t i e s ( RobotClass , s r d l 2 c a p : ’ h a s C a p a b i l i t y ’ , SubCap ) ,o w l s u b c l a s s o f ( SubCap , Cap ) .

% c a p a b i l i t y depends on l y on a v a i l a b l e components or c a p a b i l i t i e sc a p a v a i l a b l e o n r o b o t (Cap , Robot ) : -

r d f s s u b c l a s s o f (Cap , s r d l 2 c a p : ’ C a p a b i l i t y ’ ) ,

f o r a l l ( c l a s s p r o p e r t i e s (Cap , s rd l2comp : ’ dependsOnComponent ’ ,CompT) ,c omp t y p e a v a i l a b l e ( Robot ,CompT) ) ,

f o r a l l ( c l a s s p r o p e r t i e s (Cap , s r d l 2 c a p : ’ d epend sOnCapab i l i t y ’ , SubCap ) ,c a p a v a i l a b l e o n r o b o t ( SubCap , Robot ) ) .

SRDL: Query for required components

◮ Next step: read required components from action specification

◮ Recursively read dependencies for sub-actions

◮ Inherit dependencies from super-classes

◮ Equivalent for capability dependencies

% d i r e c t l y r e q u i r e d by an a c t i o n and i t s sub - a c t i o n sr e q u i r e d c omp f o r a c t i o n ( Act ion ,Comp) : -

c l a s s p r o p e r t i e s ( Act ion , s rd l2comp : ’ dependsOnComponent ’ ,Comp ) .

% r e q u i r e d by any o f i t s sub - a c t i o n sr e q u i r e d c omp f o r a c t i o n ( Act ion ,Comp) : -

p l a n s u b e v e n t s r e c u r s i v e ( Act ion , SubAct ion ) ,c l a s s p r o p e r t i e s ( SubAction , s rd l2comp : ’ dependsOnComponent ’ ,Comp ) .

% i n d i r e c t l y r e q u i r e d by r e q u i r e d c a p a b i l i t i e sr e q u i r e d c omp f o r a c t i o n ( Act ion ,Comp) : -

r e q u i r e d c a p f o r a c t i o n ( Act ion , Cap ) ,c l a s s p r o p e r t i e s (Cap , s rd l2comp : ’ dependsOnComponent ’ ,Comp ) .

SRDL: Determine missing components

◮ Next: determine missing components (required but not available)

◮ Availability is recursively evaluated: available capabilities aredefined as ’depend only on available ones’

◮ Empty result: nothing missing → action feasible

m i s s i n g c omp f o r a c t i o n ( Act ion , Robot , Comp) : -r e q u i r e d c omp f o r a c t i o n ( Act ion , Comp) ,\+ comp t y p e a v a i l a b l e ( Robot , Comp ) .

m i s s i n g c a p f o r a c t i o n ( Act ion , Robot , Cap ) : -r e q u i r e d c a p f o r a c t i o n ( Act ion , Cap ) ,\+ c a p a v a i l a b l e o n r o b o t (Cap , Robot ) .

a c t i o n f e a s i b l e o n r o b o t ( Act ion , Robot ) : -\+ m i s s i n g c a p f o r a c t i o n ( Act ion , Robot , ) ,\+ m i s s i n g c omp f o r a c t i o n ( Act ion , Robot , ) .

SRDL: Lessons learned

◮ Inference is supported by modeling:◮ Transitivity: read whole branches of kinematic tree at once◮ Property hierarchy: describe specifically, query generically◮ Class taxonomy: simply inherit dependencies from superclasses

◮ Task specifications do not need any special annotation◮ Dependencies inherited from generic action classes◮ Action dependencies are stored locally and are thus reusable

◮ Prolog facilitates definition of additional solutions:◮ E.g. determine available capabilities by looking at the ROS network

◮ Easy implementation of complex behavior◮ Few simple Prolog rules and a bit of OWL modeling◮ Robot model largely imported automatically

Exercises (30min):

KnowRob actions tutorial:http://ias.in.tum.de/kb/wiki/index.php/Reasoning about actions

Additional information:http://ias.in.tum.de/kb/wiki/index.php/Modeling tasks and actionshttp://ias.in.tum.de/kb/wiki/index.php/Modeling changing objects







◮ RoboEarth: concept, interface to KnowRob, up- anddownload of information, capability matching


Information exchange among humans

Information exchange among robots

RoboEarth

(see also

http://ias.in.tum.de/kb/wiki/index.php/Exchanging information via RoboEarth)

Application of SRDL in RoboEarth

Video: Downloading recipes, objects, and maps

Object models

Finding maps for an environment

Exercises (45min):

KnowRob RoboEarth tutorial:http://ias.in.tum.de/kb/wiki/index.php/Exchanging information via RoboEarth

Additional information:http://www.ros.org/wiki/roboearthhttp://www.ros.org/wiki/re comm

Interface to statistical relational learning and

inference

Objectives:

◮ Integrate probabilistic logical models to account for uncertaininformation

◮ Query for results generated by statistical relational models

◮ Provide evidence for statistical inference to ProbCog

(see also http://ias.in.tum.de/research/probcog)

ProbCog interface

Thank you for your attention

knowledge processing for autonomous robots · 2012. 6. 13. · roboearth: concept, interface to...

Documents