robotics, brain and cognitive sciences department giulio sandini humanoids, brain and cognitive...

Robotics, Brain and Cognitive Sciences DepartmentGiulio Sandini

Humanoids, Brain and Cognitive

SciencesGiulio SandiniItalian Institute of TechnologyRobotics, Brain and Cognitive Sciences Dept.andLIRA-Lab, University of Genoa

Information and Communication Technology Research Forum 2009Khalifa University of Science, Technology and ResearchApril 5, 2009 – Abu Dhabi


Humanoids may have to do with “automation”, service robotics, rescue and intervention but what do they have to do with “communication technologies”?


I am going to argue that communication is at the very core of humanoid robotics and that research on humanoids can help advancing communication technologies


Communication not in the sense of “transmission” but in the sense of “mutual understanding” and team work.


Today’s humans are operating artificial systems by pushing, pulling, sliding, turning, dragging, and, to a limited extent, speaking and gesturing

Artificial systems do not understand (anticipate) the intention of the human operator.


Conversely communication in humans is a process of mutual understanding.

At both ends of a communication channel there is always a human being


Why there is still no communication channel involving at least one artificial system at one terminal?


Because to understand each other two agents need to share not only the communication channel but also knowledge and experience (and to some extent also the body shape)

The body is human’s communication channel


…humanoid robots could be the equivalent of today’s visual displays in the sense of “conveying information” but with “mutual understanding”

…in the future…

humanoids as communication terminals (with the added value that humans and humanoids will be able to act together)


..future of Humanoid Robots

Copyright: Toyota Motor Corporation – Robot Technologies, 2007

Close interactions with humans supported by natural multimodal communication channels such as gesture, vision, body-language, haptics, speech and natural language

…and its value chain


Ubiquitous Robot Companion


For this we have to build systems that are able to communicate with humans in a natural way

We have to build systems that “understand” what a human is doing and, to some extent, are able to predict what a human will be doing in the immediate future


This brings about the issue of “Brain Sciences”: to implement systems that understand what humans are doing requires a lot of “brain research” to go along with “humanoid research”


How does a human being understands what another human being is doing?

How does a humanoid understands what a human being is doing?

Brain Science

Robotics Science

Cognitive

Science


Humanoids Research

Humanoid robotics research today can be seen as a human centered discipline advancing science and developing new technologies along three main streams


Human Centered Technologies

1. Build state-of-the-art humanoids2. Study humans (i.e. perceptual,

motor, learning, communication abilities)

3. Exploit human-machine interaction

Build Humanoids

Study Humans Interaction

Italian Institute of Technology


Why a body?


The chess player…

Up to a few years ago the prevalent opinion was that it was possible to study the “human system” by investigating its “sub-systems” separately (visual, auditory, motor etc.) and that it was possible to separate perception and reasoning from the physics and mechanics of the body the chess player


The if-then-else approach

The “mind” as a logical machine:

If there is an obstacle in front then turn else go ahead.

Is it possible to arrive always of a logical decision?Is it worth to look always for a logical decision?

In the real world the possible alternatives may be so numerous that it is not possible to find a “logical” solution in time…


Obstacle

Cow-catcher

What is an obstacle? Depends on the bodyWhat does it means “avoid”? Depends on the body


Our understanding and our decisions depend in many different ways on the body and the actions that our body is able to perform


A point in space is represented by:

• Where it is• In the visual space• In the somatosensory (body) space• In the auditory space

• How it can be reached– With hands– With eyes– …

• …


Object properties can be coded in terms of motor

primitives• Position of objects can be coded in terms of the

action required to reach it• Trajectory of objects can be coded in terms of

“collision trajectory” (which body part is going to hit)• Size can be coded in terms of “grasp type” (small is

whatever can be grasped with a pinch grasp)

One can express size with gestures: how big it is?


Recognize an object without the object

• How relevant is “action” in recognizing objects?

• Can the visual system exploit the view of haptic exploration to extract shape information of the explored object?

Study done with Francesco Campanella


Vision as an embodied process

Understanding as a perceptual process

Understanding with your eyes

Understanding as a motor process (goal-driven)

Understanding with your bottom

A chair is whatever gives a goal to my sitting action


F5 Mirror NeuronsActive when another individual is seen performing manipulative gestures

From: Fadiga, L., L. Fogassi, V. Gallese, and G. Rizzolatti, Visuomotor Neurons: ambiguity of the discharge or "motor“ Perception? Internation Journal of Psychophysiology, 2000. 35: p. 165-177.

SII7b

ITSTs

F1

F4AIPVIP

Reaching

Grasping

Object/hand recognition

Visualcortex


Understanding actions performed by others means to activate the areas of the brain that are activated when we execute the same actions


Being able (learning) to act is essential to understand actions

done by others.

Understanding mirror neurons: a bio-robotic approach. G. Metta, G. Sandini, L. Natale, L. Craighero, L. Fadiga. Interaction Studies. Volume 7 Issue 2. 2006


Perception provides the essential information about

the context of the action

Which are the actions compatible with the current context?


Through the context it is possible to predict the future course of events…

Object’s representation also contains “actions”


These multimodal motor representations need to be

learned and adapted continuously…


…by active exploration and interaction


Our Guiding Philosophy

It is essential to study not only how the system works but also how it is constructed.

Cognition is necessarily the product of a process of embodied development (cannot be hand-coded )


Developmental approach


The “process of building” a human being is based on learning thought interaction with the environment and with other

human beingsShaping the self

Interacting with others


Learning to control gaze

1998


RobotCub

U. Genoa IIT

HerthfordshireIST - Lisbon U. Salford

Telerobot U. Ferrara

EPFL - Lausanne U. UppsalaU. Zurich

Scuola S. Anna

Robotcub is a collaborative 5-years project in the area of “cognitive systems”

We are now in the middle of the 4th year


People• Giorgio Metta, David Vernon• Lorenzo Natale, Francesco Nori: Software, testing, calibration• Marco Maggiali, Marco Randazzo: firmware, DSP libraries, tactile sensing• Francesco Becchi, Paolo Pino, Giulio Maggiolo, Gabriele Careddu: design

and integration• Roberto Puddu, Gabriele Tabbita, Walter Fancellu: assembly• Darwin Caldwell Nikos Tsagarakis, William Hinojosa: legs and spine,

force/torque sensors• Bruno Bonino, Fabrizio Larosa, Claudio Lorini: electronics• Davide Dellepiane: wiring• Mattia Salvi: CAD maintenance• Alberto Zolezzi: managing quotes, orders and spare parts• Ricardo Beira, Luis Vargas, Miguel Praca: design of the head and face• Paul Fitzpatrick & Alessandro Scalzo: software middleware• Alberto Parmiggiani: joint level sensing• Alexander Schmitz: fingertips• Ravinder Dahiya: FET-PVDF tactile senors• Lorenzo Jamone: fingertips• Auke Ijspeert and Ludovic Righetti: simulation and initial torque

specification


The RobotCub Approach

• Scientific Stance:Cognition emerges through interactions (a body is essential).

• Research Methodology:Cognition is best studied through a programme of progressive development (gradual acquisition of a prospective capability)

• Research Strategy:Global scientific community is best served by creating an open platform and exploiting synergies


….RobotCub two goals…

Advance our understanding of several key issues in cognition

Realize a physical platform for embodied cognitive research


The iCub platform

•The iCub is a full humanoid robot sized as a three and half year-old child.•The total height is 104cm.•It has 53 degrees of freedom, including articulated hands to be used for manipulation and gesturing.•The robot will be able to crawl and sit and autonomously transition from crawling to sitting and vice-versa.•The robot is GPL/FDL: software, hardware, drawings, documentation, etc.


Degrees of freedom

• Head: vergence required + 3 dof neck• Arms: 7 dof each

– 3 shoulder, elbow, 3 wrist

• Hands: 9 dof each ► 17 joints– 5 fingers ► underactuated

• Legs: 6 dof each– 3 hip, knee, 2 ankle

• Waist: 3 dof

Σ = 53 dof


Body cover: concept


It is small…


Sensors…

• Cameras

• Skin/tactile

• Gyros/inertial

• Microphones

640x480 color - Remote headVery small but otherwise standard

Small (0.67mm)Designing force/torque sensorFingertip 3-axis sensorJoint torque measurementConductive paint…QTC: rubber-like materialTension sensors?Organic FETs?

Standard, condenser electret


Artificial sensorized skinPrinciple of operation

Multiple elements

Skin structure

By Marco Maggiali: IITand Giorgio Cannata: UNIGE

Patented


Miniaturized electronics

Hand sensor sampling PIC-based card

Force/torque sensor fitted into the sensor

Wired with 25micron coated wires

Design and documentation

Control cards 4 times smaller than the commercial equivalent

Embedded electronics in a 6-axis sensor

With: Alexander Schmitz, Lorenzo NataleElectronics: Bruno Bonino, Fabrizio Larosa


Fingertip

Outline

3D CAD Electrode fabrication

Silicon cover

Complete prototype

Electronics


Hands are key!


WikiCVS

Part lists

Drawings


iCubs in Europe

13 iCubs are on the make


.....and we are planning for more....


Yoga…


…ICub Drummer

Ludovic Righetti, Auke Jan Ijspeert: Programmable Central Pattern Generators: an Application to Biped Locomotion Control. ICRA 2006: 1585-1590


ICub helper

Peter Ford Dominey Laboratoire d'Etude des Mécanismes Cognitifs (EMC) Equipe Neuroscience Cognitive et Representations Multimodales (NCRM)Université Lumière Lyon 2


ICub Observer

Work by: Jonas Ruesch, Manuel Lopes, Alexandre Bernardino, Jonas Hornstein, José Santos Victor, Rolf Pfeifer – IST Lisbon and University of Zurich AI Lab.


…still a long way to go…


The challenges• Mindware:

– Machine learning with little human intervention (irrespective of the application but tailored to the AI application domain)

– To tackle problems like perception and control (and possibly the two together) …vision…(still not solved..)

– Robust sensorimotor coordination, manipulation, full-body control

• Bodyware:– Materials: stronger and lighter bones– Actuators: flexible, compliant, impact absorbing– Sensors: e.g. skin, but also more resolution, range, etc.– Computation: the brain of our robot is still bulky, wiring is

also a complex problem– Energy: for autonomy– Safety


Conclusions

• Humanoids and Human research may be mutually supportive

• Perception is an “action” process• Critical mass and international collaboration

(exploit body similarity)• Communication is a key aspect of humanoid

research• Body materials will be a key aspect of humanoid

research


Thanks!


Form and function are correlated in biological systems

Physical growth and morphological change is an essential ingredient of nature’s smart solutions

…are we missing something?


“Natural” minds are shaped by interactions


In nature also the bodies are shaped by interactions

Lucia Galli, TRENDS in Neuroscience Vol. 25 No. 12 – December 2002

Micromechanical hypothesis of mosaic assembly


Development and learning are associated to morphological changes at the musculoskeletal and at the neuronal level

In biological systems body and mind co-

develop

Is this unique of biological systems?

Can we think about “artificial growth”?


Solutions in biological systems are always multi-technological

Intelligent manipulation depends as much on how the controller works (mind) as on the physical properties of the muscles

Rizzolatti discovery of Mirror Neurons: exploit motor representation for visual recognition

Bizzi e Mussa-Ivaldi theory of force fieldsExploit visco-elastic properties of muscles

SII

7b

ITSTs

F1

F4AIP

VIP

Reaching

Grasping

Object/hand recognition

Visualcortex


Exploit co-development of function and physical growth

Living materials: living in the sense that their physical characteristics are changed by functions (e.g. they become harder or softer, they change shape, they “grow” differently).


…the missing dimension…?

Life science “Robotics”

Material Science

…so far we have attempted to study the adaptation of the mind but the body is fixed…

robotics, brain and cognitive sciences department giulio sandini humanoids, brain and cognitive...

Documents

service robotics

core of humanoid robotics

communication terminals

value chain slide

abu dhabi slide

future humanoids

artificial systems

sense of mutual understanding