robots in image-guided...

Copyright © Peter Kazanzides, CISST ERC, 2011 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology

Robots in Image-Guided Interventions

Peter Kazanzides

Associate Research ProfessorDept. of Computer Science

The Johns Hopkins University


My Background

• 1983-1988 Ph.D. EE (Robotics), Brown University• 1989-1990 Postdoctoral research at IBM on

ROBODOC• 1990-2002 Co-Founder of Integrated Surgical Systems

– Director of Robotics and Software– Commercial development of ROBODOC® System– Sales in Europe (CE Mark) and Asia– Clinical trials in U.S. and Japan

• 2002-present Research faculty at JHU– Leader of CISST ERC engineering infrastructure– Research in use of robotics for neurosurgery, cancer research

and therapy, telesurgery, microsurgery, …


Disclaimer

• Dr. Kazanzides currently receives research support from Curexo Technology, manufacturer of the Robodoc® System, and has served as a consultant to the company.


Laboratory for Computational Sensing and Robotics (LCSR)

Hackerman Hall

Swirnow Mock Operating Room Robotorium (shared lab)


What is Robotic Surgery?

Surgical CAD-CAM(image-guided robots)

Surgical Assistance

The integration of information processing with sensing and robotics to produce a

“super-human” man-machine team


Themes of Today’s Talk

• Origins of Robotic Surgery– Surgical CAD/CAM: Robodoc ® , Neuromate, …

– Surgical Assistance: da Vinci ®, Aesop, …

• Current and Future Research– Merger of Surgical CAD/CAM and Surgical

Assistance• Better situational awareness in Surgical CAD/CAM

• Adding image guidance to Surgical Assistance


Surgical CAD/CAM: Overview

• Preoperative imaging (e.g., CT scan)• Preoperative planning• Intraoperative registration• Computer assistance to execute plan

– (e.g., autonomous or semi-autonomous robot)


Surgical CAD/CAM: ROBODOC® System

• Initially developed to assist with Total Hip Replacement (THR) surgery– machine femur for cementless prosthesis

(femoral stem)


ROBODOC® SystemConventional procedure

(mallet and broach)Computer-assisted planning and execution


ROBODOC Pin-Based (Fiducial) Registration

2

3

1

Surgery to implant pins (bone screws) prior to CT



2

3

1Y

X T1

X

Y

Z

Planning software detects pins in CT coordinates



2

3

1

T2

Y

X T1

X

Y

Z

YZ

X

Robot finds pins in Robot coordinates



2

3

1

T2

Y

X T1

X

Y

Z

YZ

X

T2-1 * T1

Software checks pin distances (safety check) and then computes transformation between CT coordinates and robot coordinates


ROBODOC Benefits

• Intended benefits:– Increased dimensional accuracy– Increased placement accuracy– More consistent outcome

Broach Robot


ROBODOC Status• Approximately 50 systems were installed worldwide

– Europe (Germany, Austria, Switz., France, Spain)

– Asia (Japan, Korea, India)

– U.S. (Clinical trial for FDA approval)

• Over 20,000 hip and knee replacement surgeries

• ROBODOC no longer used in Europe (lawsuits still ongoing)

• Popular in Korea – one hospital claims 2,500 surgeries/year

• Curexo Technology still attempting to grow business


Lavallee, Troccaz, et al. 1989

Surgical CAD/CAM: Robotic Needle Guidance for Neurosurgery

Kwoh, et al. 1988


Surgical CAD/CAM: Robotic Needle Guidance for Neurosurgery

Courtesy: Integrated Surgical Systems


TRUS Guided Prostate Seed Placement(ultrasound for intraoperative planning)

JHU RadOnc: Song, DeWeeseJHU Engineering: KazanzidesQueen’s : FichtingerIndustry: Burdette, Acoustic Medsystems

Kronreif, ProFactor


Surgical CAD/CAM: Summary

• Works well when:– Registration can be performed accurately– Anatomy does not change

• Little or no motion, deformation• Thus, more often used for:

– Orthopaedics– Neurosurgery– Needle-based interventions with minimal

change between imaging and insertion


Surgical Assistance: Overview

• Provide information and/or mechanical assistance during procedure– improve physician’s existing sensing

and/or manipulation• e.g., reduced tremor, go where physician

cannot go– increase the number of sensors and

actuators (e.g., more eyes and hands)


Surgical Assistance: Overview

• Control paradigms:

– Teleoperation

– Cooperative control


Surgical Assistance: da Vinci® System

SRI telesurgery system, circa 1992

da Vinci S system, circa 2006


da Vinci Status

• Over 1,800 systems installed worldwide

• Principle application prostatectomy

– By 2007, over 50% of prostatectomies in US were performed by a da Vinci

• Financial success

– 2007 revenue $601 M

– 2010 revenue $1,413 M

– Intuitive Surgical market cap. > $15 B


Surgical Assistance: Robotic “Third Hand”Assistants

• Limb positioners• Retractors• Endoscope holders

– Aesop– IBM/JHU LARS– etc.

• Can incorporate sophisticated HMI, voice, vision, etc.

Credit: Yulun Wang


Surgical Assistance: Retinal Microsurgery

0.5 µm

(Left) Regular setup of ophthalmic procedure and (Right) Needle used to insert into a retinal vein in vein cannulation procedure

0.5 µm


Handle force

Kv

Steady Hand Guidance for Retinal Microsurgery

R. Taylor & R. KumarFree hand motion Steady hand motion


Steady Hand Guiding at the Cellular Level

Kumar, Kapoor, Taylor


The Future: Merger of Surgical CAD/CAM and Surgical Assistants

• Provides assistance to enable surgeon to execute a preoperative or intraoperative plan

– Why should a Surgical CAD/CAM system continue to execute a preoperative plan if the situation has changed?

– Why shouldn’t a Surgical Assistance system consider preoperative information?

• Result is a human/machine collaborative system


New Technical Challenges

• Provide more complete information to the surgeon

• Pre-operative images (preferably registered to view)

• Intra-operative images (e.g. ultrasound)

• Local sensing: force, tissue stiffness, oxygenation

• Provide physical guidance

• Improve safety through “no-fly” zones

• Improve repeatability through guidance (virtual rulers)

• Improve dexterity and reduce size (mechanism design)

• Robots for micro-surgical applications

• Go where humans cannot go


Case Studies

• Augmented reality for (da Vinci) minimally-invasive surgery

• Retinal microsurgery system• Cooperative control for skull base surgery


Integration of Preoperative Images

Better integration is possible!

Surgical Assistant Workstation (SAW)


Augmented Reality in Robot-Assisted Surgical Systems

Clockwise from upper left: daVinci surgical robot;Information overlay of force information on daVinci display (Okamura et al.); Real time overlay of ultrasound images on daVinci display (Taylor et al.)


Video to CT Registration

Stereo surface tracking

Stereo tool tracking

Information Fusion with

daVinci Display

PreoperativeImages

Vagvolgyi, Hager, Taylor, Su


Retinal Microsurgery System

20-25 gauge tools & sensors (proximity, force, ischemia, OCT, other)

Stereo video

Stereo video • Visualization & display• Real time image and

sensor processing • 3D modeling and

information fusion• Task representation• Safety monitoring• Manipulation assistance

and “virtual fixtures”

• Preoperative images• Other patient data• Procedure plans• Procedure logs

Modular control & sensing interfaces

OCT & Spectroscopy

System

Steady handmicrosurgicalrobots

Hand-heldactive tremor reduction(MICRON)

Surgical Workstation

Credit: Russell Taylor

NIH BRP EB007969-01


Retinal Microsurgery System

OCT DisplayOCT Display

3D Display with

Overlays

3D Display with

Overlays

MicrophoneMicrophoneForce ‐ FBG InterrogatorForce ‐ FBG Interrogator

Audio OutputAudio Output

EyeRobot2EyeRobot2

MicroscopeMicroscope

PhantomPhantom

Credit: Marcin Balicki


Manipulators for Microsurgery

MicronRiviere (CMU)

Steady Hand Robot (Rev 2)Iordachita, Balicki, Kazanzides, Taylor


Sensor-Based Manipulation (OCT)

Surface following using OCT visual servoBalicki, Kang, Taylor

Reference (0)

SurfaceSurgical Tip

1mm

Surface

Surgical Tip

Distance (mm)

Sig

nal In

ten

sity

Optical Coherence Tomography (OCT)


Sensor-Based Manipulation (Force)

Sensory substitution (force audio)Balicki, Iordachita, Taylor

Fiber Bragg Grating (FBG) sensor and interrogator


Credit: Rogerio Richa

Especially useful for hand-held instruments

Tool Tracking


Augmented Reality Display


Micro-force overlay


Continuous OCT (MScan) scan/review

(not yet using background tracking)


Example: Cooperatively-controlled Robot for Skull Base Surgery

• Skull base has complex 3D anatomy and traversing critical structures (nerves, vessels)

• Drilling of the skull base is often necessary to achieve access, such as for tumor removal

• Manual drilling can take hours, even when only millimeters are removed– Risk of damage to critical structures– Limits of human dexterity– Surgeon fatigue


Proposed Solution

• Use robot assistance to improve safety and efficiency of skull base drilling:– Define “safe zone” (virtual fixture) in CT– Register CT, patient, and robot– Robot holds cutting tool

• Cooperative control: responds to surgeon’s forces• Virtual fixtures: prevent excursion outside “safe zone”


System Description


• Drill bone around internal acoustic canal (IAC)– Robot provided ergonomic benefits

• Postoperative CT to assess accuracy– Average overcut ~1 mm – Maximum overcut ~3 mm

Cadaver Experiments


Mechanism Design: Snake Robot for Minimally Invasive Surgery

• Telerobotic system for throat MIS with high distal dexterity, force feedback and high redundancy for optimal suturing.

Taylor, Simaan, Kazanzides, Flint, Kapoor, Xu

DDU holder

Snake-like unit

end disk

moving platformspacer disk

base disk

central backboneinternal wire

Parallel Manipulation Unit

laryngeoscope

base link

rotating base

distal dexterity unit (DDU)

DDU for saliva suction

DDU holder

tool manipulation unit (TMU)

fast clamping device

snake drive unit

electrical supply /data lines


Krieger et al, IEEE TMBE, 2005Susil et al. J Urol,, 2006Krieger et al, MICCAI 2007

Mechanism Design: MR-Compatible “Robot” for Prostate Biopsy

A manual “robot” with real-time MR feedback


Summary

• Differing objectives means a wide variety of robot systems:– Surgical CAD/CAM: increase accuracy, precision,

repeatability – Surgical Assistance: put the eyes and hands of

surgeon in places they could not otherwise go– The Future: systems that combine both paradigms

• Situational awareness for Surgical CAD/CAM• Image-guidance for Surgical Assistants


Acknowledgements• Faculty

– Russell Taylor– Greg Hager– Allison Okamura– Gabor Fichtinger– Emad Boctor– Noah Cowan– Cam Riviere– Iulian Iordachita– Jin Kang

• Numerous Staff and Students

• Clinicians– Paul Flint– Michael Choti– Daniel Song– Ted DeWeese– Li-Ming Su– David Yuh– George Jallo– Jim Handa– Peter Gehlbach

robots in image-guided...

Documents