france-mexico summer school on image & robotics...

1Christian LAUGIERSSIR 2007 – National Polytechnic, Mexico city

FranceFrance--Mexico Summer SchoolMexico Summer Schoolon Image & Robotics on Image & Robotics (SSIR(SSIR’’07)07)

Tutorial on Tutorial on ““Robotics TechnologiesRobotics Technologies””Christian LAUGIERChristian LAUGIER

Research Director at INRIA, FranceResearch Director at INRIA, Francehttp://http://emotion.inrialpes.fr/laugieremotion.inrialpes.fr/laugier

[email protected]@inrialpes.fr

1- Grand challenges & Technological evolution

2- Basic technologies for motion autonomy

3- Two recent application domains and related technologies : Future cars & Medical robots

4- Basic Robot models

5- Basic concepts of CAD Robotics


-- Section I Section I --Grand challenges & Technological Grand challenges & Technological

evolutionevolution

Future cars ?

Future medical robots ?Future companion robots ?

Vaucanson’s automatons (1738)


Today grand challenge :Today grand challenge :Robots Robots «« sharing sharing »» our living space !our living space !

•• Why such a grand challenge ?Why such a grand challenge ?Instead of promises & impressive advances in robotics in the last decade, almost no advanced robots are currently evolving around us !

•• A large spectrum of potential applications !A large spectrum of potential applications !

•• A A favofavouurablerable technological situationtechnological situation !!⇒ Continuous & fast growing of computational power⇒ Fast development of micro & nano technologies (mechatronics)⇒ Increasing impact of information & telecom technologies on our everyday life⇒ Decreasing costs & Societal acceptance ?

Services, Entertainment, Transport, Rehabilitation & Medical care …


Where technological breakthroughs Where technological breakthroughs are required ?are required ?

• Motion & action autonomy v/s Shared control …in a dynamic world

• Increased robustness & safety (sensing & control)=> how to deal with incompleteness & uncertainty ?

• Easy programming & System adaptativity & Intuitive HMI=> self-learning capabilities, behaviors + Natural language, haptics & gestual interaction

• Smart mechatronic systems=> micro / nano robots, intelligent sensors … towards ubiquitous robotics ?

Strong human-robot interactions=> “Companion robots”

Highly dynamic world=> “Future cars”

Accuracy & Safety=> “Medical robots”

Robots inside human bodies !!


Main difficulties and approachesMain difficulties and approaches

• Previous approaches on AI & Robotics have shown their limitations=> Logics (70’s), Geometry (80’s), Random search (90’s), purely Reactive Architectures (90’s)

• The real world is too complex for being fully modeled using classical tools (in particular: incompleteness & uncertainty !!!!)=> Additional methods are required (e.g. probabilistic programming)=> Biologic inspiration could bring some help (sensori-motors systems, internal representations for motions… which seems mostly based on probabilistic laws)


Two complementary reasoning processes are Two complementary reasoning processes are requiredrequired

Mastering the complexity by using the right reasoning level & incremental approaches

=>Mainly Geometry

Taking explicitly into account the hidden variables at the reasoning level

=> Mainly Probability

Dynamic worldDynamic world

Space & MotionSpace & MotionModelsModels

Analytical & Statistical dataSensing data

Geometric & Kinematic reconstruction,SLAM

Motion prediction

Motion planMotion plan& Navigation controls& Navigation controls

Constrained Motion PlanningDifferential Flatness,

Velocity Space

IncompletnessIncompletness

UncertaintyUncertainty

Preliminary Knowledge+

Experimental Data=

Probabilistic Representation

Maximum EntropyPrinciple

( )∑− ii PP log

Queries &Queries &Decision ProcessDecision Process

Bayesian Inference(NP-Hard [Cooper 90]Heuristics & Optimization)

P(AB|C)=P(A|C)P(B|AC)=P(B|C)P(A|BC)P(A|C)+P(¬A|C) = 1


Technological evolutionTechnological evolutionfrom Automaton to Autonomous Robotsfrom Automaton to Autonomous Robots

1150 av.JC 1738 1921

1968-72 Years 60-70 1979

2001 …

1990 1995 1997

Vaucanson ‘sduck automaton

trick

Hilare (LAAS)

Ghenghis (MIT) Cog (MIT) Cycab (Inria)

2000

Aibo (Sony)

Automaton period

PoorAutonomy & Reactivity

Reactivity & Increasing interaction with humans

Industrial robots


First mechanical MS-SystemGoertz, Argone Nat. Lab.

First electrical MS-SystemGoertz, Argone Nat.Lab.

Typical control station

Exoskeleton (1972)

Advanced control stationwith force feedback

1948 1954

90’sIntroducing

CAD systems & VR

Introducing varioussensing devicesFirst Master-Slave

systems

2001 Increasing use of sensors

& Automated subtasks

Some relevant applications:Civil & Military intervention

Space, UnderwaterSurgery

Tele-surgery

70’s

Technological evolutionTechnological evolutionTeleTele--operated Robotsoperated Robots

Drones


Computer Assisted TeleComputer Assisted Tele--operationoperation

CATSystemUSER

Slave Robot

MasterSystem

controls

TVmonitors Cameras

CAD model

perceptionReflex actions

Force feedback

Forces&

Proximeters

⇒ CAD system & VR (immersive devices)⇒ Increasing use of sensing technologies⇒ Some automated sub-tasks

Human in the loop !!!


Programmed RobotsProgrammed Robots

•• Programming approachesProgramming approaches– Programming by showing (recording the joint values at a given frequency)– Predefined trajectories from CAD (sequence of key positions & velocities)– Programming languages & systems (difficult to use)– CAD-Robotics systems & Task-level systems

•• DifficultiesDifficulties– Link between the “joint space” (control) and the “workspace” (task)

=> Classical Direct & Inverse Kinematics techniques– Interpretation of sensory data & Decisional mechanisms

=> Still an open problem !!!=> Still an open problem !!!

Physical World

Articulated MechanicalSystem Control System

Programming System&

Execution Control

Interactions Interpretation

Signal processingLegend

A : Proprioceptive dataB : Execution infosD : Exteroceptive dataC1 : Numerical controlsC2 : Analogical controls

C1

A

C2

D

BUser


Accessibility issue & Reasoning spaces Accessibility issue & Reasoning spaces

•• Degrees of freedomDegrees of freedom

Redondancy for increasing the accuracy(pipe-line assembly)

Coupled d.o.f for increasing the accessibility(painting ACMA robot)

Vehicle (3 d.o.f)

Arm (6m, 3 d.o.f) End effector3 d.o.f “orientation”

2 d.o.f “fine positioning”

θ1

θ2

θ3 Pipe-line

θ1

θ2θ3

θ5θ4

θ6

θ8

θ7

θ9θ10

End effector

10 joints, but only 6 d.o.f(θ4=θ6=θ8 & θ5=θ7=θ9)

AccessibilityJoints & Redondancy

•• Reasoning spacesReasoning spaces•• Workspace (Cartesian space)Workspace (Cartesian space)

=> Position (x,y,z) + Orientation (α,β,χ) •• Joint space (nJoint space (n--dimensional dimensional ““Curved Curved spacespace””))

=> (q1, q2 .. . qn )

q1

q2 q2

q1

Motion controls : Non-holonomy


•• Sensor without contactSensor without contact–– ProximetersProximeters (infrared, ultrasound, inductive)

=> Nothing really new in the last decade !–– Passive & Active Vision, TelemetersPassive & Active Vision, Telemeters (laser, radar)

=> Improved hardware & Software technologies=> Specific sensors for automotive applications

•• Touch sensorsTouch sensors–– Force sensors Force sensors (wrist, fingers, table, legs …)–– Tactile sensors Tactile sensors (fingers, endoscopes …)

=>Miniaturized devices, Advanced integrated H/M interfaces (haptic)

•• Combining sensor modalitiesCombining sensor modalities– No sensor can give a sufficiently robust information– “Sensor fusion” is necessary for robust perception

=> Current evolution in various application domains

Sensing TechnologiesSensing Technologies

LR & SR RadarsLidar


Force & Tactile sensing technologiesForce & Tactile sensing technologies

Traditional Strain Gauges &Piezo-electric Force sensing devices

Deformable plate

Thin wire

Electrodes

Miniaturized wrist sensor (DLR, 2002)

Optoelectronic Optic fibersPiezo-resistive elements

Capacitive 8x8 (1mm)=> sensing

Pneumatic 5x5 (3mm)=> feedback

• Force sensors

• Tactile sensors

• Coupling Sensing & Human Feedback (e.g. endoscope)


Some equipped devicesSome equipped devices

Gripper (Force + Tactile) Finger (Force + Tactile + Proximetry)e.g. breast palpation for cancer

Articulated hand (Force / finger)

Sensitive glove(Position + Tactile)

Sensitive mouse(Position + Force)

Haptic device(Position + Force)


HapticHaptic interfaceinterfacefor advanced H/M interactionfor advanced H/M interaction=> Virtual Reality, Tele-operation, Simulators

Haptic systems

“Touching” virtual objects

Sensitive gloves (CyberTouch - VTI)=> position fingers & wrist (18 flexible sensors)=> vibrotactile stimulators on each finger


HapticHaptic Interaction with a virtual worldInteraction with a virtual world

“Touching” and “acting” in a virtual sceneusing a haptic interface (phantom)

A surprising “touching illusion”

© 1999 Stanford (O. Khatib & D. Ruspini)


-- Section II Section II --Technologies for motion autonomyTechnologies for motion autonomy

Motion planning & Reactive Navigation


Motion of artificial systemsMotion of artificial systems

Geometric world modeling + A priori determination of the motions that will take a robotic system from its current « configuration » (i.e. position & orientation of each individual components of the robot) to a given goal « configuration »

e.g. walking, moving around, grasping and mating objects …⇒ Motion planning is a fundamental problem in robotics

(largely addressed since the late 60’s))

Manipulator & mobile robots, Articulated hands, Drones, Automated vehicles … Virtual characters

1.1. Motion planning Motion planning (mainly geometry)(mainly geometry)

2.2. Reactive navigation Reactive navigation (mainly probability & control architectures)(mainly probability & control architectures)On-line adaptation of a motion plan according to execution conditions (hazard perception in particular)

=> Making use of some additional models (sensori-motors, behaviors …)

See tutorial on“Motion Planning”


Something to learn from biological systems ?Something to learn from biological systems ?[Berthoz 01]

Brain areas involved in:- Egocentric tasks => parietofrontal regions- Allocentric tasks => parietotemporal regions

Spatial orientation & Memory of routes :• Egocentric coding & Allocentric coding• Survey of map like strategy (planning) : Mental map => Topo-kinetic memory• Route like strategy (navigation): Memory of motions & perception (eyes, vestibular, muscles …)

=> Topo-kinesthesic memory

Network of structures contributing to saccadic eye movements (brain areas,

vestibular system, muscles) [Berthoz 97]

=> Several models & planning / navigation strategies are combinedEuropean BIBA project : Making a bridge between neurophysiological and robotics models ?

New European project : BACS (Bayesian Approaches to Cognitive Systems)


How to construct computational models ?How to construct computational models ?•• ““Mental MapMental Map”” for Motion Planning for Motion Planning (MP)(MP)

Mainly geometrical & kinematic modelsAppropriates representations (Configuration space, Velocity space …)Appropriates algorithms (collision checking, graph search, random search …)

•• ““Route like strategiesRoute like strategies”” for Reactive Navigationfor Reactive NavigationMainly reactive architectures & probabilistic modelsAppropriate representations (sensori-motor schemes, behaviors …)Appropriate approaches (reactive architectures, learning …)

q1

q2 q2

q1[Lumelski 86]

B1

CB1

Configuration Space Velocity Space (dynamic world)“Flat” Space (car-like robots)

SLAM Bayesian behaviors


A key conceptA key concept for MP for MP :: ““Configuration spaceConfiguration space””

CW ≡

A

x

y

)(qA

[Udupa 77; Lozano-Perez 83]

•• Concept of Concept of «« configurationconfiguration »»Objective: Finding a space where the mobile can be represented by a pointConfiguration: Minimal set of independent parameters uniquely specifying the position & orientation of every component of a mobile system

q )(qAiB

CW

iCB

Start

Goal

qstart

qgoal

W C

•• Path planning principlePath planning principle

Computing « C-obstacles » Searching for a path (for a point in C-space){ }

ii

free

ii

CBCCBqAqCB

U−=

≠∩= φ)(

The initial MP formulation in the workspace is not tractable !!!(i.e. continuous sequence of collision-free “position-orientations” of all the robot components)=> Configuration Space is a fundamental tool to address motion planning

e.g. C = R2 = W for the disc in the plane


How to apply this idea in practice ?How to apply this idea in practice ?

2-links planar manipulator arm amidst 2D obstacles : W = R2, q = (θ1, θ2) ⇒ C = S2

q1

q2q2

q1[Lumelski 86]

B1

CB1

New problem (Cspace)

qa

qb

Initial problem (workspace)

=> No tractable general method for computing an exact representation of the C-obstacles (and of Cfree) !!!

Approximated C-obstacle (θ−slices)

Geometric approaches(exact or slices)

Physically based approaches(gradient descent methods) Probabilistic Roadmaps



How to extend these techniques toHow to extend these techniques toreal world applications ?real world applications ?

How to deal with How to deal with uncertainty & hazardsuncertainty & hazardsof the physical world ?of the physical world ?

How to take into accountHow to take into accountNonNon--holonomicholonomic kinematickinematic constraints ?constraints ?

How to process the How to process the dynamics dynamics of bothof boththe robot and its environment ?the robot and its environment ?

Christian LAUGIERSSIR 2007 – National Polytechnic, Mexico city

Motion Planner : Motion Planner : Geometric Planner (Collision free path) + Steering Method (Feasible path)

Configuration space Equivalent “flat” space

θ

φ

φk−

L v),,,( φθyx

)(ty(y1 … ym) differentiallyindependent

any smooth curve is an admissible path

φθ

θθθ

φφθ

φ ⇒⇒

⇒⇒

===

L

RR

RRR

derivativend

Yderivativest

yxYVuandyxX

tan:2

)sin,(cos:1

),(),(),,,(

&

&

&

21

1000

0

)tan()sin()cos(

uuL

yx

X R

R

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

+

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

=

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

= φθθ

φθ&

&

&

&

&

[,])tan(22ππφφκ −∈∀=

L

Kinematic model Differential flatness

=> Linearizing outputs

Standard car

NonNon--holonomicholonomic kinematickinematic constraintsconstraints

e.g. using “shortest paths” (Dubins 57, Reeds &Shepp 90) or “CC-paths” (Scheuer 97)


•• Reactive trajectory deformationReactive trajectory deformation (Elastic strips [Khatib & Brock 99 ])

No task constraint Task constraints to satisfy

Weakly dynamic workspace: Weakly dynamic workspace: Deforming trajectoriesDeforming trajectories

“Zero order” deformation(path level)

Deform a nominal trajectory under the effects of a “repulsive potential field”

•• Dealing with NH Dealing with NH kinematickinematic constraintsconstraints [Lamiraux & Bonnafous 03 ]

“First order” deformation(perturbation of the control)


q(s)

obstacles

q(0)

q(S)

Sketch of the Sketch of the «« first order method first order method »»

q(s)

obstacles

q(0)η(s)

q(S)

q(s)

(S)q

(s)

obstacles

q(0)η

(s)+q (s)ητ

11-- Define a repulsive potential fieldDefine a repulsive potential field

22-- Integrate this potential field along the trajectoryIntegrate this potential field along the trajectory33-- Compute a direction of deformation Compute a direction of deformation ηη(s(s)) that that satisfies NH constraints & decrease satisfies NH constraints & decrease U(pathU(path))44-- Apply this deformation scaled by a small real number Apply this deformation scaled by a small real number ττ , until the collision desappears, until the collision desappears

)(1)(

min qdqu =

dssqupathUs

.))(()(0∫=

=> Trajectory deformation is obtained by => Trajectory deformation is obtained by perturbatingperturbating the input functions (controls) the input functions (controls) u(su(s))

q

up(s)

u1 (s)(s)

(s)

q’ (s)

q

))()...(),(()( 21: susususu pszingcharacteriControls =

))()...(),(()( 21

:svsvsvsv p

onsperturbatiInput=

−

(s)q

(s,τ)q

u (s)p

1(s)u1(s)+ τ

τ

v

p(s)+u p(s)

u1(s)

vτ (s) + o(η τ)

(s)

q’ (s)

q (s,τ)

q

(s)η

)(.)(),( svsusu ττ +=⇒

[Lamiraux & Bonnafous 03 ]

∑=

=p

iii sqXsusq

1)(.).()(&


Dynamic workspace : Dynamic workspace : Selecting safe controlsSelecting safe controls=> On-line avoidance of obstacles moving along arbitrary trajectories (known or sensed)=> The traditional state-time approach (zero order search) is not tractable (complexity &

real-time) … Instead, reason at the “velocity level” (first order search) !!!

•• Global Dynamic WindowsGlobal Dynamic Windows [Khatib & Brock 00]

• Generating on the fly goal-directed motions=> Sequence of safe controls

• Alternating reconstruction & planning phases=> Low dynamicity is allowed

•• Velocity ObstaclesVelocity Obstacles [Fiorini 95][Large & Shiller 00][Large et al. 03]

Future collision

Safe motion

A

• Real-time computation of V-Obstacles (velocity space)=> Instantaneous colliding velocities

• Strategies for navigating among any moving obstacles=> Obstacle avoidance & Iterative trajectory planning


[Large et al. 03]Safe navigation using VSafe navigation using V--ObstaclesObstacles

Admissible Velocities

RobotSelected Velocity

Moving obstacle Goal

Collision at time tCollision at time t+dt

No collisionbefore Th

WallsVelocity selection strategy

A single velocity outside NLVO avoids the obstacle during the time interval for which the v-obstacle was generated

•• Instantaneous escaping trajectoriesInstantaneous escaping trajectories

•• Iterative trajectory planningIterative trajectory planning Graph search based on an alternate sequence of :1- Vspace evaluation at time ti (using NLVO)2- Selection (using a cost function) of safe velocities


•• Combining onCombining on--line planning & navigation functions line planning & navigation functions

Reactive navigation=> => Path tracking & Obstacle avoidance[Raulo & Ahuactzin &Laugier 00]

Dynamic path planning=> => Adriane’s Clew Algorithm[Ahuactzin 94]

Motion planning + Reactive navigationDealing with hazards at execution timeDealing with hazards at execution time

•• Obstacle avoidance using learned Obstacle avoidance using learned «« behaviorsbehaviors »» ((bayesianbayesian programming)programming)

∏=

⊗⊗=⊗⊗⊗⊗8

181 )/()()...(

iii VDPVPDDVP φφφ

Uniform)( =⊗ φVP

∑=⊗

iDiiiiii

iiiiiiii DPDVPDP

DPDVPDPVDP)/()/()(

)/()/()()/(φ

φφwhere :

Joint distribution for the fusion :

VφArea 1

Area 2Area 8

=> Probability distributions on the controls (v,φ)


Some intervention robotsSome intervention robots

Autonomous legged underwater robotdesigned for Mine counter measures

(Tactile & Vision Sensing)

De-mining mobile & light robot⇒ Autonomous navigation

+ Remote supervision + Mine detectors

© IS-Robotics © IS-Robotics

© IS-Robotics

Autonomous robot for planetary exploration(Vision + Force)

© IS-Robotics


Towards Towards ““Companion RobotsCompanion Robots”” ??(1) Personal robot for human assistance (1) Personal robot for human assistance (intuitive robot guidance)(intuitive robot guidance)

© Stanford (O. Khatib)

33Christian LAUGIERSSIR 2007 – National Polytechnic, Mexico cityDomestic Robot (Pise)

Towards Towards ““Companion RobotsCompanion Robots”” ??(2) Personal robot for human assistance(2) Personal robot for human assistance


Towards Towards ““Companion RobotsCompanion Robots”” ??(3) Entertainment (pets & humanoid robots)(3) Entertainment (pets & humanoid robots)

Aibo (Sony)


-- Section III Section III --Towards Future Cars ?Towards Future Cars ?

•• Focusing on safetyFocusing on safety=> driving assistance & automatic driving

•• Importing concepts from aeronauticsImporting concepts from aeronautics=> « drive by wire » while filtering the actions of the driver


SocioSocio--Economic & Technical contextEconomic & Technical contextMobility is one of the characteristics of our modern society (goods & peoples)

Because of various Socio-economic and Technical reasons, Transportation systems will drastically change in the next 15-20 years (driving assistance, cybercars, advanced human/system communication …)

Governments feel more and more concerned by “safety, pollution, and traffic congestion” problemsSome numbers: 31 millions vehicles & 8000 fatalities/year in France, 1 fatality every 10mn in West Europe (e.g. 140 per day, ~1 plane crash per day)

=> After having applied more and more coercive actions, they are now looking for new technologies for increasing Safety, Pushing private cars out of cities, and Reducing the nuisances

Car constructors & Car suppliers are more and more interested inintroducing ADAS technologies in cars…. Researchers are also interested in pushing ADAS technologies towards semi-autonomy or full autonomy

Safety Traffic congestion Pollution & Space


Current & future car equipmentsCurrent & future car equipmentsSteering by wireBrake by wireShift by wire

Virtual dash-boardModern “wheel”

Navigation system

Radar, Cameras, Night Visionand other technologies for detection of obstacles

Wireless Communication Speech Recognition and Synthesis ?


Steps towards Steps towards «« automated roadautomated road »»•• The automotive approach The automotive approach (Advanced Driver Assistance Systems)(Advanced Driver Assistance Systems)

•• The The ““CybercarsCybercars”” approachapproach

ACCStop&Go

Stop&Go ++

Rural Drive Ass.

Urban Drive Ass.

Full DrivingAutomation

Longitudinal ControlLongitudinal Control

+ Lateral Control+ Lateral Control

e.g AHS in Japan; Path & IVI in USA; Prometheus, Chauffeur, Carsense in Europe

Private tracksLocal tracks

Pedestrian zonesCalm zones Suburban tracks

Intercity tracksFull driving automation on :

e.g ICVS in Japan; Praxitele, Parkshuttle, Serpentine, CyberCars in Europe

High speed

Medium speed (e.g. 4m/s, 4 times average speed of classical mobile robots)


Intelligent Transport Systems in the WorldIntelligent Transport Systems in the World•• Japan situationJapan situation

– National AHS project & Industrial R&D (Honda, Toyota, Yamaha …)– Workplan for a system deployment for year 2015 (including liability and

products development aspects)– Looks towards the European market (cars, navigation syst, R&D agreements)

•• USA situationUSA situation– AHS national project (stopped in 1998)– Several Government/Public projects, e.g. California Path (AHS), IVI

program (enhancing driving safety), Minnesota DOT program (IV technologies for trucks, buses, or snowplows)

– R&D private sector: ACC, Collision & Lane departure warning, Night vision

•• Europe situationEurope situation– Several National & European projects since the middle of the 80ths– Driving assistance (ACC, Collision avoidance …) v/s Automated driving– New substitute to the private automobile for the living heart of European

cities : “CyberCars”

•• Some other initiativesSome other initiatives– ITS Australia, Singapore, Korea, China


Some European ADAS projectsSome European ADAS projectsADAS = Advanced Driver Assistance SystemADAS = Advanced Driver Assistance System

Prometheus project (86Prometheus project (86--94)94)=> Smart cars & Smart highways=> Smart cars & Smart highways

R&D program (onR&D program (on--board & offboard & off--board systems) for increasing safety & driving comfortboard systems) for increasing safety & driving comfort

Chauffeur project (94Chauffeur project (94--98,Daimler Benz / 98,Daimler Benz / IvecoIveco))=> Automated road for trucks=> Automated road for trucks

CarsenseCarsense (car manufacturers & suppliers)(car manufacturers & suppliers)=> Sensor fusion for danger estimation=> Sensor fusion for danger estimation

French Arcos project : French Arcos project : => Vehicle=> Vehicle--InfrastructureInfrastructure--Driver systems for road safetyDriver systems for road safety

(decreasing of 30% the number of accidents)(decreasing of 30% the number of accidents)

Leading truckFollowing truck

Real world

World model

Detection/ Perception

Communication


2. Preventing from collisionwith an obtacle3. Preventing dangerousvelocities when turning

4. Preventing from veering offof the road

6. Preventing from collision withpedestrians crossing the road7. Preventing from collisionwhen turning right at intersections

5. Preventing from collision atraod intersections

1. Information on roadsurface conditions

Road-VehicleCommunication

Road-Vehicle Communication

On-boardSensors

Road SurfaceCond. Sensor

Roadside Processor

Lane MarkerSensor

On-board ECU

Obstacle Sensor

Lane Marker

Actuators

HMI

(ASV)(ASV)(AHS)(AHS)7 services for increasing safety

The Japanese ADAS approachThe Japanese ADAS approachJapanese Smart Cruise SystemsJapanese Smart Cruise Systems


The The CyberCarsCyberCars approachapproachDoor to door, 24 hours a dayDoor to door, 24 hours a daySmall (urban size), silent Small (urban size), silent User friendly interface User friendly interface Automatic manoeuvresAutomatic manoeuvres=> parking, platooning

… up to fully automated

CyberCars are focusing on historical city centres

PraxitelePraxitele : Real : Real experimentexperiment in SQY (97in SQY (97--99)99)

Industrial site Train station

Shoppingcentre

TramwayMetro

TGV

Bus

Private Car

Wal

k

FarNear

Low

High

Distance

Cap

aciy

> 500 m

BikeRoller

CyCabCyCab dualdual--mode mode vehiclevehicle ((TradedTraded by Robosoft)


Some European Some European CyberCarsCyberCars projectsprojects

ParkShuttleParkShuttle((NetherlandsNetherlands, 1997), 1997)

SerpentineSerpentine((SwitzerlandSwitzerland, , midmid 9090’’s)s)

PraxitelePraxitele((France, mid 90France, mid 90’’s)s)

EC EC CybercarsCybercars project (2001project (2001--05)05)•• 10 industrial partners10 industrial partners (Fiat, Yamaha, Frog (Fiat, Yamaha, Frog ……))•• 7 research institutes7 research institutes (Inria, Inrets, Ensmp …)•• 12 cities involved12 cities involved (Rome, Lausanne, Antibes(Rome, Lausanne, Antibes……))•• 10 M 10 M €€•• Large scale experimentsLarge scale experiments

–– Amsterdam (Amsterdam (FloriadeFloriade, 2002), 2002)–– Antibes (2004)Antibes (2004)

Next: - EC Cybercars2- Shanghai demo 2006 (EC CyberC3)


Kinodynamic Motion Planning(Dynamic constraints ...)

q

t

Graph

Trajectory

[Fraichard 92 ]

Planning CC-paths(kinematic constraints ...)

ϕ

θx

y

)(tan 1 ϕρ −= w

w

[Scheuer & Laugier 98 ]

[Laugier et al. 98 ]3-layered control architecture

Decision layer

Reactive mechanism=> Control the executionof the selected skills

Real-time “Skills”=> Close-loop controls & Sensor processing

Platooning [Parent & Daviet 96] Automatic Parallel Parking [Paromtchik & Laugier 96]

Lane Changing & Obstacle avoidance[Laugier et al. 98]

Problem 1:Problem 1: Control architecture & Driving skillsControl architecture & Driving skills

continuous curvature profile + upper-boundedcurvature & curvature derivative


«« PlatooningPlatooning »»

Electronic « Tow-bar »

CCD Linear camera + Infrared target(high rate & resolution)

[Parent & Daviet 96]


Automatic parking maneuversAutomatic parking maneuvers

Start location specification

[Paromtchik & Laugier 96]

( )

( )

φ φφ

π

π

φφ

( ) ( ),( ) ( ),

, , ,

( )

,

cos ,,

, ,

( ) . cos ,

max

maxmax max

*

**

t k A t t Tv t v k B t t T

v k k

A t

t tt tT

t t T tT t t T

tT T

T T

B t t T

vv

= ≤ ≤

= ≤ ≤

⎧⎨⎩

> > = ± = ±

=

≤ < ′− ′

′ ≤ ≤ − ′

− − ′ < ≤

⎧

⎨⎪

⎩⎪

′ =−

<

= −

,

00

0 0 1 1

1 0

12

0 5 1 4 0 ≤ ≤t T

=> On-line motion planning using sinusoidal controls φ(t) and v(t)(search for control parameters T and φmax)

On-line local world reconstruction& Incremental motion planning


Automatic driving using computer visionAutomatic driving using computer vision

Marked & Unmarked Road / Vehicle localization (Lasmea)

Road & Obstacle localization and tracking (day & night, Munich Univ)

Automatic road following on highways using vision(Munich Univ & Daimler-Benz)

… But, how to deal with various weatherconditions (night, rain …), and trafficconditions (cars, trucks, pedestrians, urban environments …) ?=> Fusion of various sensory data


Problem 2:Problem 2: Dealing with dynamic environmentsDealing with dynamic environments

=> Moving safely amidst stationary & moving obstacles=> Moving safely amidst stationary & moving obstacles(vehicles, pedestrians (vehicles, pedestrians ……) in open & dynamic environments) in open & dynamic environments

…… using inusing in--board & offboard & off--board sensingboard sensing

•• Robust interpretation of complex (sensed) dynamic scenesRobust interpretation of complex (sensed) dynamic scenesOnOn--line detection, tracking & identification of moving objects line detection, tracking & identification of moving objects …… while dealing while dealing

with temporary occlusions & target with temporary occlusions & target appearanciesappearancies/ / disappearencesdisappearencesPrediction of the future behavior of the detected moving entitiePrediction of the future behavior of the detected moving entitiess

•• Safe navigation decisions for Safe navigation decisions for ““intentional motionsintentional motions””OnOn--line (realline (real--time) path planning & Obstacle avoidancetime) path planning & Obstacle avoidance…… while taking into while taking into

account some dynamic constraintsaccount some dynamic constraintsDealing with Uncertain & Continuously Changing world modelsDealing with Uncertain & Continuously Changing world models


INRIA Experimental INRIA Experimental TestbedTestbed (AVP project)(AVP project)

AVP : Automated Valet ParkingAVP : Automated Valet Parking


The Dynamic Map ServerThe Dynamic Map Server

SceneInterpretation

Safe Navigationdecisions

Map (t)

Fusion-Tracking

Camera ProjectionDetect-Track

1'X 2'X

1'Y

2'YDistorsion

Camera Detect-Track

1'X 2'X

1'Y

2'YProjectionDistorsion

State Estimation + Motion Prediction


Topic 1:Topic 1: Autonomous navigation in a learned Autonomous navigation in a learned environmentenvironment

SLAMSLAM++

NH Motion planningNH Motion planning++

ReactiveReactive nnavigationavigation

⇒ Several functionalities (learned and downloaded) have to be combinedIncremental world modeling & localization + Motion planning + Autonomous sensor-based navigation

[Pradalier & Hermosillo 03]

[Thesis Hermosillo 2003][Thesis Pradalier 2004]


Topic 2:Topic 2: Bayesian Occupation Filter (BOF)Bayesian Occupation Filter (BOF)Patented technology from INRIA Patented technology from INRIA -- RARA

PrinciplesPrinciplesDynamic environment modellingGrid approach based on Bayesian FilteringEstimates probability of occupation AND velocity of each cell in the grid

ApplicationsApplicationsDynamic scene interpretation (state estimation & evolution prediction)Target tracking, Obstacle avoidance, Collision prediction, etc…Driving assistance & Autonomous driving

(increasing safety and comfort)

Technological transferTechnological transferPatented by INRIA & ProBayes (C. Laugier, K. Mekhnacha, M. Yguel)Joint R&D with INRIA Rhône-Alpes & Probayes (Start-up)Industrial contracts : TOYOTA, DENSO, HITACHI

Prediction

Estimation


Topic 2 (Topic 2 (cc’’eded):): Pedestrian avoidancePedestrian avoidanceBOF + Onboard laser range finderBOF + Onboard laser range finder

[Coué 03 + Coué et al. IJRR’05]

Des

crip

tion

Specification• Variables :

- Vk, Vk-1 : controlled velocities

- Z0:k : sensor observations

- Gk : occupancy grid

• Décomposition :

• Formes paramétriques :

• P( Gk | Z0:k) : BOF estimation

• P( Vk | Vk-1 Gk) : « by hand » or learning ?

Que

stio

n

Utilization

2 sensors & 3 objects


Pedestriandanger assessment

Topic 2 (Topic 2 (cc’’eded) :) : Objects tracking & Danger assessmentObjects tracking & Danger assessmentBOF + Vision trackerBOF + Vision tracker

Robust Pedestrian & Vehicle Tracking(Dealing with temporary occlusions and tracker defaults)

=> “Confidential for industrial reasons”

•• External cameras (Parkview & External cameras (Parkview & PuvamePuvame))

•• Onboard camera and radar (Toyota, Denso)Onboard camera and radar (Toyota, Denso)

[Aycard et al. 06]

[Yguel, Laugier, Mecknacha 06]

Moving peoples(Caviar)


Topic 3: Incremental Learning for Motion Prediction of Pedestrians and Vehicles

• Basic ideaIn a given environment, objects do not move at random, but engage in “typical motion patterns”, which may be learned and then used to predict motion on the basis of sensor data

• Preliminary results (D. Vasquez PhD)Solution based on an incremental learning extension of

Hidden Makov Models (GHMM model)Software libraries have been developedExperiments have been performed on both real and

synthetic data (for humans and vehicles)


Topic 4:Topic 4: Motion Planning in a dynamic environmentMotion Planning in a dynamic environment

•• ObjectiveObjectivePlan Plan safe motionssafe motions towards a given goal in a towards a given goal in a Dynamic & Uncertain Dynamic & Uncertain environmentenvironment– Deal with Robot & Environment dynamics – Computation time is limited (Real-time constraints & Environment dynamicity)– Trajectory choices rely on prediction of the obstacles motions & Handle their limited duration validity

•• ApproachApproach–– Traditional hybrid motion planning approaches donTraditional hybrid motion planning approaches don’’t take into account all these t take into account all these constraintsconstraints e.g. Global DWA [Brock & Khatib 99], Elastic band [Quinlan 93, Khatib 97], NLVO [Large 03], Fast PRM [Hsu 02] …–– ““Partial Motion PlanningPartial Motion Planning”” (PMP)(PMP) => The => The ““best partial safe motion to the goalbest partial safe motion to the goal”” is is computed at each iteration stepcomputed at each iteration step–– The time The time δδtt available available to calculate a new partial motion is function of the to calculate a new partial motion is function of the dynamicity of the environmentdynamicity of the environment–– Safety issue is addressed using the concept of Safety issue is addressed using the concept of ““Inevitable Collision StatesInevitable Collision States””

[[PettiPetti & & Fraichard 04]


Topic 4:Topic 4: Motion planning in a dynamic environmentMotion planning in a dynamic environment[Thesis Petti 06]

•• Partial Motion Planning Partial Motion Planning (PMP)(PMP)

1.1. Get model of the futureGet model of the future(a priori known / observation & prediction)(a priori known / observation & prediction)

2.2. Built tree of partial motions towards the goalBuilt tree of partial motions towards the goal3. When time 3. When time δδcc is overis over, Return Return ““ bestbest partial partial

motion motion ”” (e.g.(e.g. closestclosest, , safest)safest)

•• Inevitable Collision States Inevitable Collision States (ICS)(ICS)p v Obstacled(v) ICS (p)

[Fraichard 04]

9 controls 9 controls (α , γ)(α , γ) + 3 braking controls (for ICS)+ 3 braking controls (for ICS)+ + NH constraintsNH constraints

Repeat until goal is reachedRepeat until goal is reached


No matter the control applied to the system is, there is no trajNo matter the control applied to the system is, there is no trajectory ectory for which the system can avoid a collision in the futurefor which the system can avoid a collision in the future


-- Section IV Section IV --Medical RobotsMedical Robots

A technical and cultural revolution !A technical and cultural revolution !

•• Robot assisted surgery & MinimalRobot assisted surgery & Minimallly invasive surgeryy invasive surgery=> Navigation systems, Telesurgery, Endoscopic tools, Virtual reality

•• Rehabilitation robotsRehabilitation robots=> Assistance robots, Bionic prostheses ?


First revolution :First revolution :““Virtual information sharingVirtual information sharing””

Patient reconstruction & Medical simulatorsPatient reconstruction & Medical simulators


Automatic patient reconstructionAutomatic patient reconstructionfrom medical imaging (MRI, Scanner from medical imaging (MRI, Scanner ……))

Pre-operative Navigation & Simulation

Virtual endoscopyVirtual Colonoscopy

Virtual Cholangioscopy

Courtesy of IRCAD


• Traditional surgical training techniquesTraditional surgical training techniquesMechanical Endotrainer (passives models)Animals (ethic, different anatomy & physiology)Human patients (training curve)

© 1995 Universal Pictures © 1997 United States Surgical Corporation© 1997 United States Surgical Corporation

• Generic & PatientGeneric & Patient--based surgical simulatorsbased surgical simulators=> Surgeon training & Pre-operative surgical strategy validation

Medical simulators for new surgical proceduresMedical simulators for new surgical procedures

[Delingette 99]

=> Much more difficult than flight simulators


Stress-strain curve(litterature)

3D reconstructed model

+Three anatomic components:

- the Glisson capsule- the Parenchyma- the Vascular network

[Boux & Laugier 99]•• Constructing a Virtual liver Constructing a Virtual liver (AISIM project)

•• EchographicEchographic simulatorsimulator

Stress-strain curves(measured)

)(

)(

linearnonbxa

xF

linearxkF

−+∆

∆=

∆=

Geometric model

Measured data

+

Inria + Tim-c + UC-Berkeley [Daulignac & Laugier 00]

Constructing medical simulatorsConstructing medical simulatorsRealityReality--based modeling & Interactive dynamic simulation based modeling & Interactive dynamic simulation


Second revolution :Second revolution :““Surgical skill sharingSurgical skill sharing””

Robot assisted surgery & Minimally invasive surgery & TeleRobot assisted surgery & Minimally invasive surgery & Tele--surgery surgery


RRobot Assisted Surgery & Navigation systemsobot Assisted Surgery & Navigation systems•• Brain surgeryBrain surgery

Medical images & Registration system& Positioning robot (biopsies)

•• Orthopedic surgery (prosthesis placement)Orthopedic surgery (prosthesis placement)Rigid bodies & Optotrack

& Guidance System

Orthopilot(Aesculap)

Commercial robots: Robodoc (USA, about 7500 interventions), Kaspar (Germany)


Minimally Invasive SurgeryMinimally Invasive Surgery

=> Minimizes trauma and damage to healthy tissue… BUT reduced dexterity & workspace & sensory input to surgeon

2 d.o.f wrist + gripper Tendon driven multi-fingered

end-effector

=> New endoscopic and robotized tools are required

First steps in 1980 (laparoscopy)

Courtesy of IRCAD


Micro-surgery (JPL & MDS)- 6 d.o.f master-slave tele-manipulator- Force & Texture feedback- Clinical tests (e.g. simulated eye

microsurgery 1996, suturing…) at Cleveland Clinic

TeleTele--surgerysurgery

Tele-surgery :⇒ Improved accuracy & Gesture assistance⇒ Long-distance intervention

Robot-assisted remote tele-surgery- ZEUS Robot- Short or long distance- Digestive or Heart surgery

Courtesy of IRCAD


TeleTele--surgery: some commercial systemssurgery: some commercial systems

Robot-assisted remote tele-surgery(ZEUS Robot)

Robot-assisted remote tele-surgery(Da Vinci Robot, Intuitive Surgery Systems)


Lindbergh Lindbergh teletele--surgery experimentsurgery experiment

Prof. Marescaux from IRCAD Strasbourg(operating from New-York)

First transatlantic laparoscopic cholecystectomy (sept 7, 2001)

Patient & ZEUS Robot in Strasbourg

Communication: Optical fibers & ATM connexion (France Telecom)10 megabits/s, 70-80 ms transferts & 80 ms coding/decoding

=> 155 ms delay (up to 330 ms should be OK for the surgeon)

Courtesy of IRCAD


Third revolution ?Third revolution ?““Towards a full cooperation between Robots Towards a full cooperation between Robots

& Biological structures& Biological structures””

Medical microMedical micro--robots & Rehabilitation robots robots & Rehabilitation robots


Endoscope robots & Medical microEndoscope robots & Medical micro--robotsrobots

Micro-systems(local diagnosis & therapy)

Endocrawler (NTU Singapore) European project “Neurobot”


Rehabilitation robotics (elderly, disabled)Rehabilitation robotics (elderly, disabled)

Rehabilitation training system (MIT)

Electro-stimulation(Inria & Lirmm)

Future bionic hand prosthesis ?

Robotized wheelchair for handicapped people

(e.g. voice or gaze based control)European project “Neurobot”


Brain Controlled WheelchairBrain Controlled WheelchairB. Rebsamen, E. Burdet, C.L. Teo, M. Ang, C. Laugier

• Path following strategy

• Destination selection with P300 interface

• Future work on Motor-Sensory modelslearning (BACS)

Cognitive models & Control of a robotics Cognitive models & Control of a robotics plateformplateform(BACS European IP project : Bayesian Approach to Cognitive Syste(BACS European IP project : Bayesian Approach to Cognitive Systems)ms)


ConclusionConclusion•• Impressive improvement of some robotics technologies during the Impressive improvement of some robotics technologies during the last last

decade decade => Unreachable perspectives of the 90=> Unreachable perspectives of the 90’’s seems now to be possibles seems now to be possible•• Future robots will probably share our Future robots will probably share our «« living space living space »» …… on some well on some well

chosen domains such as chosen domains such as transportation, health care, or home servicetransportation, health care, or home service•• Such robots will have close & complex interactions with humans Such robots will have close & complex interactions with humans …… while while

using natural communication channels using natural communication channels (sound & voice, gaze, gesture)(sound & voice, gaze, gesture)•• This will probably lead to a cultural and technological revolutiThis will probably lead to a cultural and technological revolutionon

…… but various problems are still to be solved :but various problems are still to be solved :–– Technological Technological (dynamicity, robustness & safety, HMI, (dynamicity, robustness & safety, HMI, mechatronicmechatronic))–– Legal & Liability questions, Social acceptation & Ethics, Costs Legal & Liability questions, Social acceptation & Ethics, Costs ……

Future cars ?

New robotizedsurgical procedures ?

Service & Companion robots ?

Bionic prostheses ?

Medical micro-robots ?

Rehabilitation robots ?

france-mexico summer school on image & robotics...

Documents