Non-Invasive Ultrasound Diagnosis and Non-Invasive Ultrasound Diagnosis and Treatment System Utilizing Robot Technology Treatment System Utilizing Robot Technology

Department of Engineering Synthesis,Department of Engineering Synthesis,School of Engineering, The Univ. of TokyoSchool of Engineering, The Univ. of Tokyo

(Nano Bioengineering Education Program) (Nano Bioengineering Education Program) Norihiro KOIZUMINorihiro KOIZUMI

ContentsContents

1.1. Stone Motion Tracking for Non-Invasive Stone Motion Tracking for Non-Invasive Ultrasound Diagnosis and Treatment SystemUltrasound Diagnosis and Treatment System

2. Probe Positioning Support Utilizing Shoulder 2. Probe Positioning Support Utilizing Shoulder Model for Ultrasound DiagnosisModel for Ultrasound Diagnosis

Stone Motion Tracking for Non-Stone Motion Tracking for Non-Invasive Ultrasound Diagnosis and Invasive Ultrasound Diagnosis and

Treatment SystemTreatment System

IntroductionIntroduction

Transducer

Affected Part

Body

Water

High Intensity Focused Ultrasound

Destruction of Stone

Clinical Use

JC HIFU System (in China)

High Intensity Focused Ultrasound（ HIFU）High Intensity Focused Ultrasound（ HIFU）

1997-1038 clinical trials

Ultrasound

Y. Matsumoto, et. al.

IntroductionIntroduction

Motion of the affected part by respiration

Establishment of the control method to compensate the motion of the

affected part by respiration

Compensation of the motion of the affected part should be required

ObjectiveObjective

Problem of HIFUProblem of HIFU

SolutionSolution

Transducer

Affected Part

UltrasoundBody

Water

High Intensity Focused Ultrasound

Non-Invasive Ultrasound Non-Invasive Ultrasound Diagnosis and TreatmentDiagnosis and Treatment

destroy the cancer region or the stone utilizing the focused ultrasound in the body directly not injuring the tissue (skin, muscle, etc.) of the patient by tracking and following the affected part which moves by respiration, etc.

FeatureUltrasound image is adopted for not injuring the tissue (skin, muscle, etc.) of patients.High speed CCD camera isn’t available.Propose the feed-forward control utilizing the periodical motion of the respiration.

[1] Y.Nakamura, H.Kishi, and H.Kawakami, “Heartbeat Synchronization for Robotic Cardiac Surgery,” Proc. of the 2001 IEEE International Conference on Robotics & Automation, pp.2014 － 2019, 2001.

Concept

Related work

Constructed System ConfigurationConstructed System Configuration

Aquarium

Rubber Membrane

Transducer

Water

BodyStone

Aquarium

Rubber Membrane

Transducer

Water

BodyStone

(a) Isometric View (b) Front View600

17

50

730

Irradiation Part

Water

Aquarium

Rubber Membrane

Body (Rabbit)

Aluminum Frame

Testing Bench

Ultrasound diagnostic device Im

ag

e p

roce

ssin

g

de

vice

ControllerXYZ

Stage

Trans-ducer

Pulsegenerator

Ultrasound probe

Video signal

(30fps)

Control signal(1kHz)

Position data　(30Hz)

Stone

ProbesTransducer

Transducer ProbesWater to secure path to the affected part

2 probes for 3 dimensionalposition data

Stone motion tracking

Motion Tracking System ConfigurationMotion Tracking System Configuration

XYZ stage

HIFU transducer

Center ultrasound probeSide ultrasound probe

Motor controller

Pulse generator

Ultrasound diagnosis device

Navigation Computer

Real-time tracking system

*Two probes for 3D tracking

XYZ motion control system

HIFU system

Visual Tracking for StoneVisual Tracking for Stone

100mm

５ mm

)4

2sin(10 ty

　 　 　　 　 　 　

The movement model formula of a model renal calculus

Model renal calculus

Ultrasound probe

Section image from center and side ultrasound probes

Processing Cycle

GrabImage

ProcessingPattern

MatchingBlob Analysis Reporting

[mm][sec]

Analysis of Kidney MotionAnalysis of Kidney Motion

-20-15

-10-505

1015

0 5 10 15

Time [sec]

Pos

ition

[mm

]

1010～～ 15 mm15 mmAmplitudeAmplitude

30 mm/s30 mm/sMax. speedMax. speed

33～～ 4 sec4 secPeriodPeriod

KidneyKidneyObjectObject

① Periodical motion ② Transition of period, amplitude

① Periodical motion ② Transition of period, amplitude

Kidney motion (man)

Stone

Dead Time in ServoingDead Time in Servoing

2dT

3dT

4dT

1dT

5dT

Dead time to capture the ultrasound images in the ultrasound diagnostic device (About 33ms)

Dead time to capture the ultrasound images in the image processing device (About 33ms)

Dead time to process the ultrasound images in the image processing device (About 33ms)

Time delay to transmit the position data acquired by the image processing (About 1ms)

Dead time to control the motor (About 1ms)

Total 100ms dead time!

Decrease servoing performance! 0.8mm≒

Feed-Forward ControlFeed-Forward Control

BtT

Ax el )2

sin(mod

)cos(mod_ tAxx eld

fffocus

ZnTntnT

txtxA rfocus

rfocus

　,)1(

))}(min())({max(5.0

)( mod_rfocus

relgain

dfbfocus xxkx

Presume Periodical Motion Parameter

Presume amplitude

Transition ofperiod, amplitude,disturbances

-

Time delay in Image processing system

Feed-Forward controller

robot30 Hz

sTdallegaink=

+

+

+Desiredfoucus position

Stone position

Focus position

Feedback

Feed-Forward

-20

-15

-10

-5

0

5

10

15

0 5 10 15

Time [sec]

Po

sitio

n [

mm

]

Kidney Motion

Periodical motion

Experimental System ConfigurationExperimental System Configuration

モデル結石

モデル駆動装置

- 12- 8- 4048

12

0 2 4 6 8

[sec]時間

[m

m]

位置

ヒトの呼吸動作

Ultrasound diagnosticdevice Im

ag

e p

roce

ssin

g d

evi

ceControllerXYZ

Stage

Transducer Pulsegenerator

Ultrasoundprobe

Video signal

(30fps)

Control signal

(1kHz)

Position data　(30Hz)

Respiration data

Introduce model that simulate the stone motion

Stone Motion Servoing Experiment Stone Motion Servoing Experiment

Experimental ResultExperimental Result

Avgerage errorAvgerage error

Without feed-Without feed-forwardforward 3.243.24　　 mmmm

With feed-forwardWith feed-forward 2.152.15　　 mmmm

-15-10

-505

1015

20 25 30 35 40

Without feed-forward With feed-forward

Servoing Error

-15-10-505

1015

20 25 30 35 40

Desired precision： sub mm　　 less than 1mm if possible

Enhance servoing performance 1mm !

Time s

Po

siti

on

mm

Time s

Po

siti

on

mm

ConclusionConclusion

　１． The concept of integrated system for non-invasive ultrasound diagnosis and treatment is proposed and constructed

　２． Feed-Forward control utilizing the periodical motion of the affected part by respiration is proposed to enhance servoing performance.

　３． Servoing experiment is conducted and the effectiveness of the proposed feedforward control method is confirmed.

Future WorksFuture Works

1. Enhance tracking performance1. Enhance tracking performance

2. Robustness of the visual tracking 2. Robustness of the visual tracking

3. Secure safety for the patient3. Secure safety for the patient

4. Animal experiments4. Animal experiments

ContentsContents

1. Stone Motion Tracking for Non-Invasive 1. Stone Motion Tracking for Non-Invasive Ultrasound Diagnosis and Treatment SystemUltrasound Diagnosis and Treatment System

2. Probe Positioning Support Utilizing Shoulder 2. Probe Positioning Support Utilizing Shoulder Model for Ultrasound DiagnosisModel for Ultrasound Diagnosis

Probe Positioning Support Probe Positioning Support Utilizing Shoulder Model for Ultrasound Utilizing Shoulder Model for Ultrasound

DiagnosisDiagnosis

IntroductionIntroduction

Remote Ultrasound Diagnostic System

Probe positioning support by the navigation utilizing the shoulder model

Strain for medical doctor during the probe manipulation

ProblemProblem

SolutionSolution

BackgroundBackground

Master-slave typed

Aging society

Probe positioning support utilizing the shoulder model, distance sensor, and force sensor

Related WorksRelated Works

[1] A. Knoll, et al., “The EndoPAR System for Minimally Invasive Robotic Surgery,” IEEE/RSJ International Conference on Intelligent Robots and Systems, 2004.

FeatureFeature

1. Shoulder model based on anatomy2. Distance information between probe and affected part3. Force information between probe and affected part

Integrate and utilize these 3 points as follows

Related workRelated work

ConceptConcept

Remote Ultrasound Diagnostic SystemRemote Ultrasound Diagnostic System

① Communication network between master and slave site ② master-slave system ③ [master site] medical doctor and master in multimedia cockpit ④ [slave site] patient, helper and slave in consulting room ⑤ Ultrasound image transmission for diagnosis ⑥ Image and audio transmission for communication

① Communication network between master and slave site ② master-slave system ③ [master site] medical doctor and master in multimedia cockpit ④ [slave site] patient, helper and slave in consulting room ⑤ Ultrasound image transmission for diagnosis ⑥ Image and audio transmission for communication

Master site Slave site

Diagnostic ExperimentDiagnostic Experiment （（ clinical clinical useuse））

Object of DiagnosisObject of DiagnosisShoulder pain in hemodialysis patientsShoulder pain in hemodialysis patients

[Diagnostic image ]③

A view of the

coracoacromial

ligamanet

[Diagnostic image ]②

A short axis view of

the tendon of the

supraspinatus muscle

[Diagnostic image ]①

A tendon of the long

head of the

biceps brachi muscle

① ② ③

①

②③

Diagnostic image

Required Shoulder ModelRequired Shoulder Model

Modeling of bones, muscles, and ligaments is required to support the probe positioning. Those muscles and ligaments are

components of diagnostic images and adhered to those bones.

humerus bonescapula bone

biceps brachii muscle

supraspinatus muscle

coracoacromial 　 ligamanet

Construct Standard Bone ModelConstruct Standard Bone Model Standard bone modelStandard bone model

Constructed based on Constructed based on Atlas of Atlas of Human AnatomyHuman Anatomy※※

※ G.Wolf-Heidegger and P.Kopf-Maier, ” Wolf-Heidegger Color Atlas of Human Anatomy, Karger, S. Inc , 2003.

Scapula bone humerus bone

Extract body regionExtract body region

Measure center line Measure center line of upper arm and of upper arm and curve of shouldercurve of shoulder

Generate unique Generate unique bone modelbone model

Registration Registration between model and between model and bodybody

Construct muscle Construct muscle and ligament modeland ligament model

Construct standard Construct standard bone modelbone model

Curve of shoulder

Center line of upper arm

Recognize Region of the Upper ArmRecognize Region of the Upper Arm

Extract body regionExtract body region

Measure center line Measure center line of upper arm and of upper arm and curve of shouldercurve of shoulder

Generate unique Generate unique bone modelbone model

Registration Registration between model and between model and bodybody

Construct muscle Construct muscle and ligament modeland ligament model

Construct standard Construct standard bone modelbone model

Modeling of MuscleModeling of Muscle

Extract body regionExtract body region

Measure center line Measure center line of upper arm and of upper arm and curve of shouldercurve of shoulder

Generate unique Generate unique bone modelbone model

Registration Registration between model and between model and bodybody

Construct muscle Construct muscle and ligament modeland ligament model

Construct standard Construct standard bone modelbone model

Adhesion points of supraspinatus muscle

Adhesion points of biceps brachii muscle

Construct Construct MMuscle and uscle and TTendon endon MModelodel

humerus bonescapula bone

biceps brachii muscle

Supraspinatus muscle

coracoacromial 　 ligamanet

Extract body regionExtract body region

Measure center line Measure center line of upper arm and of upper arm and curve of shouldercurve of shoulder

Generate unique Generate unique bone modelbone model

Registration Registration between model and between model and bodybody

Construct muscle Construct muscle and ligament modeland ligament model

Construct standard Construct standard bone modelbone model

Point Position to Push Probe & Point Position to Push Probe & Navigate Probe in Parallel Direction to ImageNavigate Probe in Parallel Direction to Image

Point desired pushing position by click

][: pixelPd

][ pixelPP prd

][: pixelPpr

Navigate probe positionIn parallel directionx

y

Realized Probe positionNavigate probe in Navigate probe in parallel direction to parallel direction to imageimage

Realize and Realize and maintain proper maintain proper contact forcecontact force

Point position to Point position to push probepush probe

Recognize probe Recognize probe positionposition

Navigate probe in Navigate probe in pushing directionpushing direction

Probe Navigation Probe Navigation in Pushing Directionin Pushing Direction

Distance sensor

Probe

Implement infra-red radiation distance sensor

|| affpr xx

Distance between probeand affected part:

Navigate probe in Navigate probe in parallel direction to parallel direction to imageimage

Realize and Realize and maintain proper maintain proper contact forcecontact force

Point position to Point position to push probepush probe

Recognize probe Recognize probe positionposition

Navigate probe in Navigate probe in pushing directionpushing direction

Approaching to affected part

[mm]

Realize and Maintain Proper Realize and Maintain Proper Contact ForceContact Force

　　　 20|| affpr xx

If distance is within threshold,

Control contact force to makeproper diagnostic image

Proper contact forceDiagnostic image 1：

2 N～ 8 N（ Avg. of 5 examinee)

Proper contact forceDiagnostic image 1：

2 N～ 8 N（ Avg. of 5 examinee)

Navigate probe in Navigate probe in parallel direction to parallel direction to imageimage

Realize and Realize and maintain proper maintain proper contact forcecontact force

Point position to Point position to push probepush probe

Recognize probe Recognize probe positionposition

Navigate probe in Navigate probe in pushing directionpushing direction

[mm]

Probe Navigation to Affected PartProbe Navigation to Affected Part

Display the probe position on shoulder model

Click the desiredprobe position

ConclusionsConclusions

1. Shoulder model is proposed.

2. A registration method is proposed between the probe and the affected part.

3. A navigation function to the affected part utilizing the shoulder model is implemented.

4. Effectiveness of the navigation function is confirmed by the navigation experiment.

Future WorksFuture Works

1. Enhance precision of registration.1. Enhance precision of registration.

2. 3 dimensional modeling & registration.2. 3 dimensional modeling & registration.

3. Probe orientation adjustment.3. Probe orientation adjustment.

4. Digitalization and technologizing of the ski4. Digitalization and technologizing of the skill of the medical doctor.ll of the medical doctor.

Acknowledgements

Prof. Mamoru MitsuishiProf. Mamoru MitsuishiDr. Hiroyuki HashizumeDr. Hiroyuki HashizumeDr. Mitsuru NagoshiDr. Mitsuru NagoshiMr. Deukhee LeeMr. Deukhee LeeMr. Kohei OtaMr. Kohei OtaMr. Takehiko TsurumiMr. Takehiko TsurumiAll Members of NML.All Members of NML.Nakashima Propellor Co., Ltd.Nakashima Propellor Co., Ltd.Shigei Medical Institute and HospitalShigei Medical Institute and Hospital

Prof. Yoichiro MatsumotoProf. Yoichiro Matsumoto

Dr. Shin YoshizawaDr. Shin Yoshizawa

Mr. Yukio KanekoMr. Yukio Kaneko

Mr. Akira ItoMr. Akira Ito

Hitachi Medical Corp.Hitachi Medical Corp.