Learning to Order Objects using Haptic and Proprioceptive Exploratory Behaviors
Jivko Sinapov, Priyanka Khante, Maxwell Svetlik, and Peter Stone
Department of Computer ScienceUniversity of Texas at Austin, Austin TX 78712, USA{jsinapov,pkhante,maxwell, pstone}@cs.utexas.edu
2
Building-Wide Intelligence Project:http://www.cs.utexas.edu/~larg/bwi_web/
3
Building-Wide Intelligence Project:http://www.cs.utexas.edu/~larg/bwi_web/
4
Building-Wide Intelligence Project:http://www.cs.utexas.edu/~larg/bwi_web/
5
Motivation: Grounded Language Learning
Robot, fetch me the green empty bottle
6
Object Category Recognition in Robotics
Sridharan et al. 2008
Lai et al. 2011Rusu et al. 2009
Collet et al. 2009
7
Object Category Learning in Robotics
Thomason, J., Sinapov, J., Svetlik, M., Stone, P., and Mooney, R. (2016).Learning Multi-Modal Grounded Linguistic Semantics by Playing I, Spy
Robotics and Vision 3 Session
8
Now, when and where does this fail...
Consider the word, “weight” - how should it be grounded?
9
How do humans ground such words?
Sample Montessori toys designed to teach children about the ordinal properties of object weight, height, and size
10
Object Ordering in Psychology
11
Object Orderings in Human Environments
12
Problem Formulation
● Order Recognition: what property is a given series of objects ordered by?
“height”
“width”
13
Problem Formulation (2)
● Order Insertion: given an object series, insert a new object into the correct position
series test object
14
Three-Stage Approach
Stage 1: Object Exploration Stage 2: Unsupervised Order Discovery
. . . .
Stage 3: Semantic Grounding
weight width height
. . . .
16
Stage 1: Object Exploration
32 common household and office items
The objects vary along three ordinal properties:
1) Weight2) Width3) Height
17
Exploratory Behaviors
18
Video
19
Video
20
Video
21
Haptic and Proprioceptive Feature Extraction
Time
Join
t P
ositi
ons
(Pro
rioce
ptio
n)Jo
int
Effo
rts
(Hap
tics)
. . . . . .
22
Haptic and Proprioceptive Feature Extraction
Time
Join
t P
ositi
ons
(Pro
rioce
ptio
n)Jo
int
Effo
rts
(Hap
tics)
. . . . . .
23
Haptic and Proprioceptive Feature Extraction
Time
Join
t P
ositi
ons
(Pro
rioce
ptio
n)Jo
int
Effo
rts
(Hap
tics)
. . . . . .
24
Stage 2: Unsupervised Order Discovery
grasp
lift
hold
lower
drop
proprioception
Beh
avio
rsSensory Modalities
push
press
haptics
26
Unsupervised Order Discovery Example with Synthetic Data
Object order with highest likelihood using the method of [Kemp and Tennenbam, 2008]
Input Relational Count Matrix
27
Example Relational Count Matrix with the Press action and Haptic features
Similarity between objects i and j in the press-haptic sensorimotor context
28
Resulting Order (Press behavior and Haptic modality)
The number corresponds to the object's height in millimeters
29
Stage 2: Unsupervised Order Discovery
grasp
lift
hold
lower
drop
proprioception
Beh
avi
ors
Sensory Modalities
push
press
haptics
30
Stage 3: Order Grounding Stage
31
Order Grounding Example: “height”
Positive Examples: Negative Examples:
. . .
. . .
32
Object Order Representation
Training Example:
Object Orders Discovered During Stage 2
. . . .
33
Object Order Representation
Training Example:
. . . .
Object Orders Discovered During Stage 2
34
Object Order Representation
Training Example:
. . . .
Object Orders Discovered During Stage 2
35
Object Order Representation
Training Example:
x1
. . . .
Object Orders Discovered During Stage 2
36
Object Order Representation
Training Example:
x1
. . . .
Object Orders Discovered During Stage 2
37
Object Order Representation
. . . .
Training Example:
x1
x2
Object Orders Discovered During Stage 2
38
Object Order Representation
. . . .
Training Example:
x1
x2 . . . . x
n
Object Orders Discovered During Stage 2
39
Results: Order Recognition
40
Sample Learned Decision Trees
Hold Haptics
Lower Haptics
Lift Haptics
Press Proprioception
Press Haptics
Grasp Proprioception
weight width height
41
When does the robot make mistakes?
42
When does the robot make mistakes?
difficult easy
43
When does the robot make mistakes?
difficult easy
44
Object Order Insertion Results
45
Object Order Insertion Results
46
Object Order Insertion Results
47
Object Order Insertion Results
48
Conclusion
● A behavior-grounded framework for learning object ordering concepts
● The robot grounded three ordering concepts, “weight”, “height”, and “width”
● Future Work:– Active action selection
– Learn object ordering concepts in conjunction with object categories, pairwise object relations, etc.
– Learn from humans (for a preview, see our next talk at Robotics and Vision III)
49
Thank you!
Jivko Sinapov Maxwell Svetlik Peter Stone
http://www.cs.utexas.edu/~larg/bwi_web/
Priyanka Khante