[11-15 oct 2009] minimally invasive surgery maneuver recognition based on surgeon model
TRANSCRIPT
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
IROS 2009 : INTERNATIONAL CONFERENCE ON INTELLIGENT ROBOTS AND SYSTEMS
MINIMALLY INVASIVE SURGERY MANEUVER
RECOGNITION BASED ON SURGEON MODEL
B. Estebanez, G. Jimenez, V. Muñoz, I. Garcia-Morales, E. Bauzano and J.
Molina
Dpto. Ingeniería de Sistemas y Automática
http://www.isa.uma.es
Universidad de Málaga (Spain)
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
Minimally Invasive Surgery Maneuver Recognition Based On Surgeon Model
OUTLINE
I. Introduction
II. Intervention Model
III.Surgeon Behavior Model
IV.Implantation and Experiments
V. Conclusions and Future Works
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
Minimally Invasive Surgery Maneuver Recognition Based On Surgeon Model
I. INTRODUCTION
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
I. Introduction
Laparoscopic Surgery:
“Accessing to the abdominal
cavity with especial long
instruments and by means of an
endoscope with camera can be
seen the area of the operation.”
LAPAROSCOPIC SURGERY
Advantages
• Reduce recovery time
• Limit post-operative complications
• Lower scars
Constraints
• Movement limitations
•Techniques require high skill
• Loss of touch and 3D perception
• Hand-eye coordination problems
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
I. Introduction
COMMERCIAL SYSTEMS OF SURGICAL ROBOT
Zeus
Da Vinci
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
I. Introduction
INTERFACE OF THE ROBOTIC ASSISTANT OF TWO ARMS
Robotic Assistant of Two Arms
Surgeon and tools with marks
Computer with
Recognition System
Operation Room
Surgical
Tool
Surgeon
3D Tracker
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
Minimally Invasive Surgery Maneuver Recognition Based On Surgeon Model
II. INTERVENTION MODEL
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
THE WORKFLOW OF A SURGICAL PROCEDURE
II. Intervention Model
START
Dissecting tissue to expose cystic duct
Clipping cystic duct
Cutting cystic duct
Dissecting/Separating Gallbladder
Collecting Gallbladder
END
First Maneuver
Second Maneuver
Third Maneuver
Fourth Maneuver
Fifth Maneuver
Sixth Maneuver
Final Maneuver
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
SET OF BASIC ACTIONS OF A MANEUVER
II. Intervention Model
Four degrees of freedom
Maneuver: set of basic actions
that represent the instrument
movements: left-right, up-down, to
rotate on itself and insertion-
removal.
Others functions are clamp
opening and closing.
Each maneuver is identified as a
specific sequence of these basic
actions.
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
INTERVENTION MODEL
II. Intervention Model
UPPER LAYER INNER LAYER
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
Minimally Invasive Surgery Maneuver Recognition Based On Surgeon Model
III. SURGEON BEHAVIOR MODEL
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
LIBRARY OF MANEUVER MODELS
III. Surgeon Behavior Model
Off-line process
MODULE OF MANEUVER TRAINING
LIBRARY OF MANEUVER
MODELS
On-line process
MODULE OF MANEUVER
RECOGNITION
DATA ACQUISITION AND CODING
MODULE
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
DATA ACQUISITION AND CODING MODULE
III. Surgeon Behavior Model
CHARACTERISTIC EXTRACTION
FUZZY LABELING
Characteristic Vector
of a Movement
Signals of Movements of
surgical tools
Current
Movement
Label
Observable
Output
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
MODULE OF MANEUVER TRAINING
III. Surgeon Behavior Model
DATA ACQUISITION AND CODING
MODULE
MODULE OF MANEUVER TRAINING
LIBRARY OF MANEUVER
MODELS
Action
TypePosition and Orientation
of Tools
Outputs
SequenceParameters of a
Maneuver Model
Surgeon
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
MODULE OF MANEUVER RECOGNITION
III. Surgeon Behavior Model
Surgeon
DATA ACQUISITION AND CODING
MODULE
Change Condition
of Maneuver
Position and
Orientation
of Tools
MODULE OF MANEUVER
RECOGNITION
LIBRARY OF MANEUVER
MODELS
Outputs
Sequence
INTERVENTIONMODEL
Current
Maneuver
Parameters
of a Maneuver
Model
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
MODULE OF MANEUVER RECOGNITION
III. Surgeon Behavior Model
Change Condition
of Maneuver
Outputs
Sequence
INTERVENTIONMODEL
Current
Maneuver
Maneuver with
High
Probability
LIBRARY OF MANEUVER MODELS
Maneuver 1 (A1 , B1 matrixes)
Maneuver 2 (A2 , B2 matrixes)
Maneuver 3 (A3 , B3 matrixes)
Maneuver N (AN , BN matrixes)
ViITERBI ALGORITHM
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
Minimally Invasive Surgery Maneuver Recognition Based On Surgeon Model
IV. IMPLEMENTATION AND EXPERIMENTS
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
• Simplification of the reality.
• Recognition of trajectories of tool tip.
• The basic actions are lines with different angles
and directions.
• Library has geometrical figures like
circle, square and triangle.
• A protocol is a sequence of these maneuvers.
1800
1900
2000
2100
2200
2300
-250
-200
-150
-100
-50
-300
-200
-100
0
100
200
300
400
x (mm)y (mm)
z (
mm
)
EXPERIMENTAL SYSTEM SETUP
IV Implementation and Experiments
Tool Tip
Fulcrum
Hand
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
MANEUVER MODEL
IV Implementation and Experiments
RECOGNITION RESULTS
End
A 4
A 2A 1
A 3
Start
Table 1. Isolated Maneuvers
Trajectory % Real Correct
Answer (100 essays /
maneuver)
CIRCLE 98.35 %
SQUARE 100.00 %
TRIANGLE 95.87 %
Table 2. Protocols
Trajectory % Ideal Correct
Answer (50
essays /protocol)
% Real Correct
Answer (50
essays / protocol)
90.39 % 85.00 %
91.20 % 82.50 %
94.29 % 81.00 %
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
Minimally Invasive Surgery Maneuver Recognition Based On Surgeon Model
V. CONCLUSIONS AND FUTURE WORKS
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
Minimally Invasive Surgery Maneuver Recognition Based On Surgeon Model
CONCLUSIONS
A methodology has been established for modeling the surgical
behavior: acquisition and coding, off-line training and on-line
recognition .
The HMM are useful for surgeon maneuver recognition.
Limitations
The performance of the system can be affected by the shadows
during the instrument tracking.
Improvement of the maneuver ending condition detection.
The experiments are about fictitious maneuver and are dependent on
the figure orientation.
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
Minimally Invasive Surgery Maneuver Recognition Based On Surgeon Model
FUTURE WORKS
Acquisition of information inside an
operation room.
Inclusion of other useful information.
Signal reconstruction in instrument
shadows situation.
Design of new real maneuver
models, including the movements of two
instruments.
Integration of this recognition module
in the global system of two arms.
Syste
m E
ng
ineeri
ng
an
d
Au
tom
ati
on
Dep
art
men
t
M. Belén Estebanez Campos
THANKS FOR YOUR ATTENTION
IROS 2009 : INTERNATIONAL CONFERENCE ON INTELLIGENT ROBOTS AND SYSTEMS