icub: the open source humanoid robot… · with alessandra sciutti, francesco nori, thierry pozzo....
TRANSCRIPT
iCub: the open‐source humanoid robot…
Giorgio MettaUniversity of Genoa and Italian Institute of Technology
The tale of the Wright brothersThe tale of the Wright brothers(and three messages)
1 Reverse engineering:1. Reverse engineering:– Looking and copying bird flight → aircraft design– Reverse of reverse engineering → aerodynamics led to
better understanding of bird flight (forward engineering)2. Models:
– Wright started from a previous model of Lilienthal (which was wrong) → but then they had (after 2 years) to produce their own models (and test them) they built a wind tunnel their own models (and test them), they built a wind tunnel (very modern)!
3. Stability and control:• Separate models didn’t work well (either stability or Separate models didn t work well (either stability or
control)• Discovered that the key to stability and control is by
rolling → turn by rolling! Separate models don’t work, system wide approach is requiredsystem-wide approach is required.
• Understanding at the systems level
and the story goes…and the story goes
• Late 1903 first powered flight (35m • Late 1903, first powered flight (35m, 10km/hour)5 l t 2 h fli ht• 5 yrs later, 2 hours flight
• 8 yrs later, across North America• 24 yrs later, New York to Paris• 65 yrs later three people to the moon65 yrs later, three people to the moon• Now, small seats and screaming
infantsinfants
Three messagesThree messages
• Reverse engineering:Reverse engineering:– Study and be inspired by the brainModels and experiments “wind tunnels”:• Models and experiments wind tunnels :– Mathematical models and robotsGl b l h• Global approach:– Sanity check by implementing everything
on a real physical platform, complete systems, real feedback
Infants in 10‐17m range are very challanged when you put a toy in a drawer and
l it If th th illclose it. If they can, they will immediately open the
drawer and take out the toy. Our drawer resisted openingOur drawer resisted opening
by a weight attached to it.
ll h h ll h f h bPulling something that resists pulling has to start from the baseof support. Adults will start the pull by activating the gastrocnemious muscles 50 ms before the arm starts pulling.
von Hofsten C., Rosander K., et al.
10-month-old infant10-month-old infant 16-month-old infant16-month-old infant
Drawer starts movingDrawer starts moving Drawer starts movingDrawer starts moving
From: Lackner JR Dizio P Gravitoinertial force background level affects adaptation toFrom: Lackner JR, Dizio P. Gravitoinertial force background level affects adaptation to Coriolis force perturbations of reaching movements. Journal of Neurophysiology 1998, 80:546‐553.
Forwardmodel
Efferentcopy
Gripforce
controller
RealizedDesired
trajectory Inverse
Fingers Gripforce
controller
Arm Realizedtrajectory
j y Inversemodel
From: Flanagan JR, Wing AM. The role of internal models in motion planning andFrom: Flanagan JR, Wing AM. The role of internal models in motion planning and control: evidence from grip force adjustments during movements of hand‐held loads.Journal of Neuroscience 1997, 17:1519‐1528.
Results (2): Slopes0.070.07
Results (2): Slopes
0.05
0.06
0.05
0.06
p = 0 217
0.03
0.04
Slo
pes
0.03
0.04
Slo
pes
p 0.217
No significant difference among slopes in all
0.01
0.02
0.01
0.02g p
conditions.
Fixed down Fixed up Variable Variable only down Variable only up0
ConditionsFixed down Fixed up Variable Variable only down Variable only up
0
Conditions
Is it kinematics or dynamics h ?that matter?
Scheidt, Robert A., David J. Reinkensmeyer, Michael A. Conditt, W. Zev Rymer, and Ferdinando A. Mussa-Ivaldi Persistence of motor adaptation during constrained multi-joint arm movements JMussa-Ivaldi. Persistence of motor adaptation during constrained, multi-joint, arm movements. J Neurophysiol 84: 853–862, 2000.
Looking at othersLooking at others
H-reflex deviation from the mean(conductance of the spinal-muscle nerves)
Muscle anticipate the kinematics
kinematics
Borroni P Montagna M Cerri G & Baldissera F (2005) Cyclic time course
Prone position, wrist ext/flexion (ECR, FCR muscles)
Borroni, P., Montagna, M., Cerri, G., & Baldissera, F. (2005). Cyclic time course of motor excitability modulation during the observation of a cyclic hand movement. Brain Research, 1065, 115-124.
Supine positionSupine position
As before, but differentphase difference btw thephase difference btw thekinematics and the muscularactivation
Then compare with actual actionThen compare with actual action
Which was 54° on average EMG signals fromthe FCR and ECRmuscles
Which was 112° on average
The movement doesn’t change
Objects come to existence jbecause they are manipulated
Fixate target Track visual motion…
(…including cast shadows)
Detect moment of impact
Separate arm, object motion
Segment object
Which edge should beWhich edge should be considered? Maybe some cruel
grad-studentglued the cube to the table
Color of cube and table are poorly separated
table
With Paul Fitzpatrick
Cube has misleading surface pattern
Into object affordances…j
0.4
0.5
0.4
0.5
0.2
0.3
0.2
0.3Bottle, “pointiness”=0.13 Car, “pointiness”=0.07
ccur
renc
e
Rolls at right Rolls
0 10 20 30 40 50 60 70 80 900
0.1
0 5
0 10 20 30 40 50 60 70 80 900
0.1
0 5babi
lity
of o
c angles toprincipal axis
along principal axis
0.3
0.4
0.5
0.3
0.4
0.5
Ball, “pointiness”=0.02Cube, “pointiness”=0.03
mat
ed p
rob
0
0.1
0.2
0
0.1
0.2estim
0 10 20 30 40 50 60 70 80 900 0 10 20 30 40 50 60 70 80 900
difference between angle of motion and principal axis of object [degrees]With Paul Fitzpatrick
The qualitative “geometry” of pokingq g y p g
0.350.35
backslap pull in
0 1
0.15
0.2
0.25
0.3
0 1
0.15
0.2
0.25
0.3 backslap
abili
ty
pull in
-200 -150 -100 -50 0 50 100 150 2000
0.05
0.1
-200 -150 -100 -50 0 50 100 150 2000
0.05
0.1
0 250 25ated
pro
b
side tap
0 1
0.15
0.2
0.25
0 1
0.15
0.2
0.25
side tap
Est
ima
-200 -150 -100 -50 0 50 100 150 2000
0.05
0.1
-200 -150 -100 -50 0 50 100 150 2000
0.05
0.1
back slappush awayDirection of movement [deg] back slappush away
I t ti b tiInterpreting observationsInvoking the object’s natural Going against the object’s Invoking the object’s natural Going against the object’s
Demonstration by human
rolling affordance natural rolling affordance
Demonstration by human
rolling affordance natural rolling affordance
human
Mimicry in similar
human
Mimicry in similar situation
Mimicry when
situation
Mimicry whenMimicry when object is rotated Mimicry when object is rotated
Data from human graspingData from human grasping2 cameras
FrameTo disk
ImagesFrame
grabbers
RS232
Cyber-glove
Other sensors
40 msec
Tracker
To disk
Tactile RS232
sensors
Bayesian classifierBayesian classifier{Gi}: set of gestures
168 sequences per subject10 subjects
6 complete setsF: observed features{Ok}: set of objects
z
6 complete sets
p(Gi|Ok): priors (affordances)p(F|Gi Ok): likelihood to observe
a
~ 76
cm
x
yz
p(F|Gi,Ok): likelihood to observe F
( ) ( ) ( ) ( )( ) ( ) ( ) ( )| , | , | / |i k i k i k kp G O p G O p G O p O=F F F
( )ˆ arg max | ,i
MAP i kG
G G O= F -45° (b)+90° (a) b
° ( )
+45° (b)+180° (a)
0° (b)+135° (a)
f Two types of experiments
Vision Classifier
Fv, OkGi
Fv Ok Fm Ok
Vision ClassifierVMM
Fv, Ok Fm, OkGi
Learned by backpropagation ANN
EstimationEstimation
p(Gi|Ok): affordances by counting estimated • p(Gi|Ok): affordances, by counting, estimated on the whole database
• p(F|Gi Ok): EM algorithm on the parameters p(F|Gi,Ok): EM algorithm on the parameters of a mixture of Gaussians (from Matlab implementation)
• VMM: Neural network, sigmoidal activation units, linear output, trained on the whole databasedatabase
Role of motor information in action understanding
Object affordances (priors)
Visual space Motor space
Classification(recognition)Grasping actions
U d st di i s: bi b ti h G M tt G S di i L Understanding mirror neurons: a bio-robotic approach. G. Metta, G. Sandini, L. Natale, L. Craighero, L. Fadiga. Interaction Studies. Volume 7 Issue 2. 2006
Some results…Exp. I(visual)
Exp. II(visual)
Exp. III(visual)
Exp. IV(motor)
Training# Sequences 16 24 64 24
# of view points 1 1 4 1
Cl ifi ti 100% 100% 97% 98%Classification rate
100% 100% 97% 98%
# Features 5 5 5 15# Modes 5-7 5-7 5-7 1-2
Test
# Sequences 8 96 32 96
# of view points 1 4 4 4p
Classification rate
100% 30% 80% 97%
Learning how to graspLearning how to grasp
Question: How well do we guess the subject’s grasping style? show the system the initial part of the hand trajectory (22 joints + 6 position/orientation) and figure out the final
fi ticonfiguration
Learning: SVM-regression With C. Castellini, F. Orabona
Learning how to graspLearning how to grasp• Allows for a precise prediction ofAllows for a precise prediction of
– hand position: 1.5cm– hand orientation: 2 5°hand orientation: 2.5– hand posture: 7.5°
• Reasonably predictive:• Reasonably predictive:– hand position: 200ms
h d i t ti 120– hand orientation: 120ms– hand posture: 90-200ms (depending on
th bj t)the object)
Learning how to graspLearning how to grasp
Affordance: knowing the object will uniformly improve the error, at no additional computational overhead (it requires clustering of the data which can be supervised by object recognition)supervised by object recognition)
Future developmentFuture development…Speech
he h
ead
ueem
ent o
f th
lips,
tong
u
Lary
nx
Mov
e
Facial expression Shape of the tongue
The mirror system and speech: with L. Fadiga, L. Craighero
The iCub: quick summaryq yThe iCub is the humanoid baby-robot
designed as part of the RobotCub project
– The iCub is a full humanoid robot sized as a three and half year-old child.f y
– The total height is 104cm.– It has 54 degrees of freedom, including
articulated hands to be used for manipulation articulated hands to be used for manipulation and gesturing.
– The robot will be able to crawl and sit and autonomously transition from crawling to autonomously trans t on from crawl ng to sitting and vice-versa.
– The robot is GPL/FDL: software, hardware, drawings, documentation, etc.g , ,
Degrees of freedomDegrees of freedom
• Head: vergence, common tilt + 3 dof neck• Arms: 7 dof each
Sh ld (3) lb (1) i t (3)– Shoulder (3), elbow (1), wrist (3)• Hands: 9 dof each ► 19 joints
– 5 fingers ► underactuated• Legs: 6 dof each
– Hip (3), knee (1), ankle (2)• Waist: 3 dofWaist: 3 dof
Σ = 53 dof (not counting the facial expressions)
SensorizationSensorization• Cameras
– Pointgrey Dragonfly firewires
80x30mm
cameras• Force/torque sensors
– Custom development: 6 axialMi h k
58x42mm
• Microphones, speaker– Standard condenser electrect
miniature microphones– Pinnae– Pinnae
• Gyroscopes, linear accelerometers– Xsense: Mtx
• Joint level sensing• Joint level sensing– Position sensors, current consumption,
temperature• Custom electronicsCustom electronics
– DSP based, programmable embeddedcontrollers
More examples…
With Peter Ford-Dominey (U. Lyon 2) With Auke Ijspeert, Ludovic Righetti, Sarah Degallier (EPFL)
With a lot of students @ RobotCub summer school 2008 With VisLab (IST Lisbon)
Promoting the iCubPromoting the iCub• RobotCub Open Call
31 ti i t 6 i ill i f th – 31 participants, 6 winners will receive a copy of the iCub free of charge
• Further developmentFurther development– FP7 project ITALK: 4 iCub’s will be built, language
• CollaborationsCollaborations– Univ. of Karlsruhe: new and longer legs
• Simulator:Simulator– Both Open Source and as a model in Webots
FingertipFingertip
OutlineOutline
Silicon cover Electronics
3D CAD Electrode fabrication Complete prototype3D CAD Electrode fabrication
PeoplePeople• Giulio Sandini: Mentor & guidance• Lorenzo Natale, Francesco Nori: Software, testing, calibration• Marco Maggiali Marco Randazzo: firmware DSP libraries tactile sensingMarco Maggiali, Marco Randazzo: firmware, DSP libraries, tactile sensing• Francesco Becchi, Paolo Pino, Giulio Maggiolo, Gabriele Careddu: design and
integration• Roberto Puddu, Gabriele Tabbita, Walter Fancellu: assembly• Nikos Tsagarakis, William Hinojosa: legs and spine, force/torque sensorsN g , W m H n j g n p n , f / qu n• Bruno Bonino, Fabrizio Larosa, Claudio Lorini: electronics and wiring• Luciano Pittera, Davide Dellepiane: wiring• Mattia Salvi: CAD maintenance• Alberto Zolezzi: managing quotes, orders and spare partsm g g q , p p• Giovanni Stellin: hand• Ricardo Beira, Luis Vargas, Miguel Praca: design of the head and face• Paul Fitzpatrick & Alessandro Scalzo: software middleware• Alberto Parmiggiani: joint level sensinggg j g• Alexander Schmitz: fingertips• Nestor Nava: small Harmonic Drive integration• Ravinder Dahiya: FET-PVDF tactile senors• Lorenzo Jamone: fingertipsg p• Jean-Baptiste Keller, Daniel Roussy: construction• Ludovic Righetti: simulation and initial torque specification